Raise login errors after dbopen returns

[navidemad]

Problem

rb_tinytds_connect calls dbopen(), and dbopen() does the login. During the login, the DBPROCESS has no userdata yet. When the login fails, FreeTDS calls tinytds_err_handler or tinytds_msg_handler. These handlers find no userdata, so they raise at once.

A raise from C is a longjmp: it leaves dbopen() before FreeTDS can clean up. Thus FreeTDS never frees the half-open connection:

• the DBPROCESS,
• the socket and the connection objects (tds_alloc_socket, tds_init_connection),
• the error strings (dbperror, tds_alloc_client_sqlstate).

Each failed attempt leaks 70 KB to 150 KB. A client that retries against a server that is not available loses this memory on each retry.

A related leak is in the same function. If the server negotiates a TDS version older than 7.3, rb_tinytds_connect raises after dbopen() returns a connection. At that point cwrap->closed is still 1, so the free function does not call dbclose(). The open connection leaks.

Fix

1. During dbopen(), the handlers keep the first error in a tinytds_login_error struct. The struct is on the stack of rb_tinytds_connect. The handlers return to FreeTDS and do not raise. FreeTDS then stops the login and frees its objects.
2. After dbopen() returns, rb_tinytds_connect raises the kept error. If dbopen() returned a connection, the function closes it first.
3. Before the "older than 7.3" error, rb_tinytds_connect closes the connection.

The raised exception does not change. It is the first error that the handlers receive, with the same message, source, severity, db_error_number and os_error_number. During the login, info messages stay ignored, as before. For SYBETIME, the handler now returns INT_TIMEOUT to FreeTDS, which stops the login.

The pointer to the struct is a static variable. dbopen() runs with the GVL held, so only one login is in progress at a time. A handler reads the pointer only when its DBPROCESS has no userdata.

rb_tinytds_connect must build the exception without a DBPROCESS. Thus this PR moves the creation of the TinyTds::Error object from rb_tinytds_raise_error into a new rb_tinytds_new_error function. rb_tinytds_raise_error calls it and keeps its behavior.

Measurements

The tool is ruby_memcheck 3.0.1. It runs Ruby under Valgrind and removes the false positives that Ruby causes. A "definitely lost" block is a block with no pointer to it at exit.

Scenario	Attempts	master	This PR
Wrong username	5	443,394 bytes	32,879 bytes
Wrong username	20	1,674,939 bytes	32,879 bytes
Connection refused (port 1)	5	386,984 bytes	32,879 bytes
Connection refused (port 1)	20	1,449,299 bytes	32,879 bytes
tds_version: "7.1"	5	771,989 bytes	32,879 bytes
tds_version: "7.1"	20	2,989,319 bytes	32,879 bytes

On master, each attempt adds about 82 KB (wrong username), 71 KB (connection refused) or 148 KB (TDS 7.1). With this PR, the total stays the same for any number of attempts. The 32,879 bytes that remain come from the one-time dbinit allocations (the FreeTDS context).

Reproduction script

# MODE=login_error N=20 ruby_memcheck -Ilib leak_repro.rb
require "tiny_tds"

opts = { host: "mssql", username: "tinytds", password: "", database: "tinytdstest", login_timeout: 5 }
opts = case ENV.fetch("MODE")
  when "login_error" then opts.merge(username: "willnotwork")
  when "refused" then opts.merge(host: "127.0.0.1", port: 1)
  when "old_tds" then opts.merge(tds_version: "7.1")
  end

Integer(ENV.fetch("N", "10")).times do
  TinyTds::Client.new(**opts)
rescue TinyTds::Error
  nil
end

Validation

• bundle exec rake test: 124 tests, 538 assertions, 0 failures, 0 errors.
• The existing tests check the raised error for an unknown dataserver (error handler), a wrong username (message handler) and a login timeout (SYBETIME through toxiproxy).
• A new test checks the "older than 7.3" error. It covers the new dbclose() call, and it checks that a new connection works after the error.
• The full test suite under ruby_memcheck (binary_name: "tiny_tds") shows no invalid read or write, and no leak from dbopen(). The remaining 4,136-byte blocks come from nogvl_setup. Free the non-blocking error array before a raise #613 fixes them. With both PRs, only the one-time dbinit blocks remain.
• astyle --options=astyle.conf does not change the C files. standardrb reports no offense in test/client_test.rb.

Environment: Ruby 4.0.7 on Linux aarch64, FreeTDS 1.5.19 built from source, SQL Server 2022 CU27 (mcr.microsoft.com/mssql/server:2022-latest), Valgrind 3.24.0.

Note

The draft v4 branch (#603) keeps the same dbopen() call and the same handlers. That branch has the same leaks.