You are not logged in.
Pages: 1
Version: 2.4.13372, same code in current master (2.4.18151), src/net/mormot.net.sock.pas, TPollSockets.PollForPendingEvents, non-POLLSOCKETEPOLL branch. Seen on Windows with NO_ASYNC_WINIOCP (THttpAsyncServer, select mode).
What happens:
TPollAsyncSockets sets fRead.UnsubscribeShouldShutdownSocket := true. With that flag set, PollForPendingEvents skips the loop that cancels subscribe+unsubscribe pairs of the same socket (if not fUnsubscribeShutdownSocket then ...). If a connection is stopped before the poll thread has processed its Subscribe (e.g. the client closes right after connecting), its Subscribe and Unsubscribe arrive in the same batch:
1. The unsubscribes are processed first: fPoll[p].Unsubscribe(sock) fails because the socket is not in any fPoll[] yet. Only "Unsubscribe(%) failed" is logged, and the socket is not closed.
2. The subscribes are processed next: the socket is added with the tag of the stopped connection.
3. The connection goes to the GC and is freed after about 10 s. The socket stays in select() and is never closed (handle leak).
4. Every event on that socket returns the stale tag: MergePendingEvents calls TPollReadSockets.SetPending/EnsurePending on the freed instance, which sets fReadPending in fFlags (offset 12 on Win32) of freed memory.
Evidence: FastMM4 FullDebugMode reports "block has been modified after being freed", modified byte offset 12 (1 byte), previous class THttpAsyncServerConnection, allocated in TAsyncConnection.Create (accept thread), freed by TAsyncConnections.FreeGC <- DoGC <- ProcessIdleTix. This shows up within a few hundred requests with 7 client threads that open and close connections. In normal builds the stray 0x40 lands in whatever reuses the block (random access violations elsewhere).
Fix we use (both changes only matter when fUnsubscribeShutdownSocket is set; it cannot cancel the wrong pair because the socket is not closed before the Unsubscribe is processed, so its handle cannot be reused in between):
// ensure subscribe + unsubscribe pairs are ignored
for u := 0 to sub.UnsubscribeCount - 1 do
begin
sock := sub.Unsubscribe[u];
for s := 0 to sub.SubscribeCount - 1 do
if sub.Subscribe[s].socket = sock then
begin
if Assigned(fOnLog) then
fOnLog(sllTrace, 'PollForPendingEvents sub+unsub sock=%',
[pointer(sock)], self);
if fUnsubscribeShutdownSocket then
sock.ShutdownAndClose({rdwr=}false);
sub.Unsubscribe[u] := nil; // mark both no op
sub.Subscribe[s].socket := nil;
break;
end;
end;and when the Unsubscribe fails, close the socket anyway (the caller relies on the poll to close it):
if sock <> nil then
begin
if Assigned(fOnLog) then
fOnLog(sllTrace, 'PollForPendingEvents Unsubscribe(%) failed count=%',
[pointer(sock), fCount], self);
if fUnsubscribeShutdownSocket then
sock.ShutdownAndClose({rdwr=}false);
end;With the fix, the same load ran about 1,950 open/close requests (plain HTTP and TLS) with no FastMM report.
I can't open a pull request, so here is the change as a git diff against current master (2.4.18151); it applies with `git apply` from the repository root.
diff --git a/src/net/mormot.net.sock.pas b/src/net/mormot.net.sock.pas
index 29a6e71..0c03193 100644
--- a/src/net/mormot.net.sock.pas
+++ b/src/net/mormot.net.sock.pas
@@ -5564,22 +5564,26 @@ begin
(sub.UnsubscribeCount <> 0))then
fOnLog(sllTrace, 'PollForPendingEvents sub=% unsub=%',
[sub.SubscribeCount, sub.UnsubscribeCount], self);
- // ensure subscribe + unsubscribe pairs are ignored
- if not fUnsubscribeShutdownSocket then
- for u := 0 to sub.UnsubscribeCount - 1 do
- begin
- sock := sub.Unsubscribe[u];
- for s := 0 to sub.SubscribeCount - 1 do
- if sub.Subscribe[s].socket = sock then
- begin
- if Assigned(fOnLog) then
- fOnLog(sllTrace, 'PollForPendingEvents sub+unsub sock=%',
- [pointer(sock)], self);
- sub.Unsubscribe[u] := nil; // mark both no op
- sub.Subscribe[s].socket := nil;
- break;
- end;
- end;
+ // ensure subscribe + unsubscribe pairs are ignored - also with
+ // fUnsubscribeShutdownSocket (closing the socket here), otherwise the
+ // Unsubscribe below fails (not yet in fPoll[]) and the Subscribe then adds
+ // the socket with the tag of a connection which will be released by the GC
+ for u := 0 to sub.UnsubscribeCount - 1 do
+ begin
+ sock := sub.Unsubscribe[u];
+ for s := 0 to sub.SubscribeCount - 1 do
+ if sub.Subscribe[s].socket = sock then
+ begin
+ if Assigned(fOnLog) then
+ fOnLog(sllTrace, 'PollForPendingEvents sub+unsub sock=%',
+ [pointer(sock)], self);
+ if fUnsubscribeShutdownSocket then
+ sock.ShutdownAndClose({rdwr=}false);
+ sub.Unsubscribe[u] := nil; // mark both no op
+ sub.Subscribe[s].socket := nil;
+ break;
+ end;
+ end;
// use fPoll[] to retrieve any pending notifications
fPollLock.Lock;
try
@@ -5601,9 +5605,13 @@ begin
break;
end;
if sock <> nil then
+ begin
if Assigned(fOnLog) then
fOnLog(sllTrace, 'PollForPendingEvents Unsubscribe(%) failed count=%',
[pointer(sock), fCount], self);
+ if fUnsubscribeShutdownSocket then
+ sock.ShutdownAndClose({rdwr=}false); // caller expects it closed
+ end;
end;
// then subscribe to the new connections
for s := 0 to sub.SubscribeCount - 1 doAwesome, thanks
Thanks for the welcome!
I can't open an issue or a pull request on GitHub at the moment, so here is the change as a patch against current master (2.4.17982; TSqlDBOdbcConnection.Connect is unchanged there). It is small: Connect uses ConnectW and SqlDriverConnectW (already imported in mormot.db.raw.odbc) with the strings converted to UTF-16, as the statements already do with PrepareW. fSqlDriverFullString is converted back to UTF-8, so the property gives the same value as before. Nothing else changes.
Tested on Windows, 32-bit, Delphi 11, through SqlDriverConnect (full connection string):
MariaDB Connector/ODBC 3.1.20, MariaDB 10.3, CharSet=utf8mb4: before the patch, an INSERT with 'Ærø åøæ ÄÖÜ Łódź Ωμέγα 日本' fails with "[22018] Invalid character value for cast specification". With the patch, the INSERT, a WHERE with the same literal and reading it back all give the exact text.
SQL Server Express through {SQL Server}: the same before and after.
The unit also compiles with FreePascal 3.2.2 (i386-win32).
Not tested: the DSN path (ConnectW with aServerName), only compiled.
If some old ODBC driver turned out to need the ANSI connect, an option on TSqlDBOdbcConnectionProperties could keep the *A() calls for it, but I did not want to add one without a known case.
--- a/src/db/mormot.db.sql.odbc.pas
+++ b/src/db/mormot.db.sql.odbc.pas
@@ -366,6 +366,7 @@
var
Log: ISynLog;
Len: SqlSmallint;
+ server, user, pwd, conn, full: RawByteString; // UTF-16 for the *W() API
begin
SynDBLog.EnterLocal(Log, self, 'Connect');
Disconnect; // force fDbc=nil
@@ -382,26 +383,35 @@
Check(self, nil,
AllocHandle(SQL_HANDLE_DBC, fEnv, fDbc),
SQL_HANDLE_ENV, fEnv);
+ // use the Unicode API, as for the statements: some drivers (e.g. MariaDB
+ // Connector/ODBC) mark a connection opened with the *A() API as ANSI, and
+ // then convert the SQL text of PrepareW() to the system code page
with fOdbcProperties do
if fServerName <> '' then
+ begin
+ server := Utf8DecodeToUnicodeRawByteString(fServerName);
+ user := Utf8DecodeToUnicodeRawByteString(fUserID);
+ pwd := Utf8DecodeToUnicodeRawByteString(fPassWord);
Check(self, nil,
- ConnectA(fDbc, pointer(fServerName), length(fServerName),
- pointer(fUserID), length(fUserID), pointer(fPassWord), length(fPassWord)),
- SQL_HANDLE_DBC, fDbc)
+ ConnectW(fDbc, pointer(server), length(server) shr 1,
+ pointer(user), length(user) shr 1, pointer(pwd), length(pwd) shr 1),
+ SQL_HANDLE_DBC, fDbc);
+ end
else if fDatabaseName = '' then
EOdbcException.RaiseU(
'Missing ServerName=DataSourceName or DataBaseName=FullConnectString')
else
begin
- FastSetString(fSqlDriverFullString, 1024);
- fSqlDriverFullString[1] := #0;
+ conn := Utf8DecodeToUnicodeRawByteString(fDatabaseName);
+ SetLength(full, 2048); // 1024 WideChar
+ FillCharFast(pointer(full)^, length(full), 0);
Len := 0;
Check(self, nil,
- SqlDriverConnectA(fDbc, GetDesktopWindow, pointer(fDatabaseName),
- length(fDatabaseName), pointer(fSqlDriverFullString), length(fSqlDriverFullString),
+ SqlDriverConnectW(fDbc, GetDesktopWindow, pointer(conn),
+ length(conn) shr 1, pointer(full), length(full) shr 1,
Len, DRIVERCOMPLETION[fOdbcProperties.fSqlDriverConnectPrompt]),
SQL_HANDLE_DBC, fDbc);
- SetLength(fSqlDriverFullString, Len);
+ fSqlDriverFullString := RawUnicodeToUtf8(pointer(full), Len);
end;
// retrieve information of the just created connection
GetInfoString(fDbc, SQL_DRIVER_NAME, fDriverName);TSqlDBOdbcConnection.Connect (mormot.db.sql.odbc.pas) connects with the ANSI functions ConnectA and SqlDriverConnectA, while statements are then prepared with PrepareW.
Some ODBC drivers decide the character handling of the whole connection from the connect call. The MariaDB Connector/ODBC (tested with 3.1.20, 32-bit, Windows, MariaDB 10.3) marks a connection made with an ANSI connect function as an "ANSI connection". It then converts the UTF-16 text of SQLPrepareW to the Windows ANSI code page (cp1252), not to the connection's character set.
Result with CharSet=utf8mb4 in the connection string (server side character_set_client = utf8mb4):
INSERT INTO t (v) VALUES ('Ærø åøæ ÄÖÜ')
-> [22007] [ma-3.1.20][10.3.14-MariaDB]Incorrect string value: '\xC6r\xF8 \xE5\xF8...' for column `t`.`v` at row 1 (1366)
The bytes are cp1252, labelled as utf8mb4. With CharSet=latin1 the statement works, but then results lose every character outside Latin-1, so neither setting gives full Unicode.
The same applies on Linux through unixODBC, where the driver manager and driver also see an ANSI connect.
Suggestion: connect with SqlDriverConnectW / ConnectW (the connection string converted to UTF-16), as the statements already use the W functions. If some driver needs the ANSI connect, an option on TSqlDBOdbcConnectionProperties (e.g. UseAnsiConnect) could keep the old behaviour.
SQL Server ({SQL Server}, ODBC Driver 17/18) works either way, so the change should not affect it.
Thanks for mORMot 2. We are moving a Delphi BI server to mORMot's database layer for FreePascal and Linux, and the ODBC and libpq layers otherwise work very well for us.
Pages: 1