You are not logged in.
Version: 2.4.13372, same code in current master (2.4.18151), src/net/mormot.net.sock.pas, TPollSockets.PollForPendingEvents, non-POLLSOCKETEPOLL branch. Seen on Windows with NO_ASYNC_WINIOCP (THttpAsyncServer, select mode).
What happens:
TPollAsyncSockets sets fRead.UnsubscribeShouldShutdownSocket := true. With that flag set, PollForPendingEvents skips the loop that cancels subscribe+unsubscribe pairs of the same socket (if not fUnsubscribeShutdownSocket then ...). If a connection is stopped before the poll thread has processed its Subscribe (e.g. the client closes right after connecting), its Subscribe and Unsubscribe arrive in the same batch:
1. The unsubscribes are processed first: fPoll[p].Unsubscribe(sock) fails because the socket is not in any fPoll[] yet. Only "Unsubscribe(%) failed" is logged, and the socket is not closed.
2. The subscribes are processed next: the socket is added with the tag of the stopped connection.
3. The connection goes to the GC and is freed after about 10 s. The socket stays in select() and is never closed (handle leak).
4. Every event on that socket returns the stale tag: MergePendingEvents calls TPollReadSockets.SetPending/EnsurePending on the freed instance, which sets fReadPending in fFlags (offset 12 on Win32) of freed memory.
Evidence: FastMM4 FullDebugMode reports "block has been modified after being freed", modified byte offset 12 (1 byte), previous class THttpAsyncServerConnection, allocated in TAsyncConnection.Create (accept thread), freed by TAsyncConnections.FreeGC <- DoGC <- ProcessIdleTix. This shows up within a few hundred requests with 7 client threads that open and close connections. In normal builds the stray 0x40 lands in whatever reuses the block (random access violations elsewhere).
Fix we use (both changes only matter when fUnsubscribeShutdownSocket is set; it cannot cancel the wrong pair because the socket is not closed before the Unsubscribe is processed, so its handle cannot be reused in between):
// ensure subscribe + unsubscribe pairs are ignored
for u := 0 to sub.UnsubscribeCount - 1 do
begin
sock := sub.Unsubscribe[u];
for s := 0 to sub.SubscribeCount - 1 do
if sub.Subscribe[s].socket = sock then
begin
if Assigned(fOnLog) then
fOnLog(sllTrace, 'PollForPendingEvents sub+unsub sock=%',
[pointer(sock)], self);
if fUnsubscribeShutdownSocket then
sock.ShutdownAndClose({rdwr=}false);
sub.Unsubscribe[u] := nil; // mark both no op
sub.Subscribe[s].socket := nil;
break;
end;
end;and when the Unsubscribe fails, close the socket anyway (the caller relies on the poll to close it):
if sock <> nil then
begin
if Assigned(fOnLog) then
fOnLog(sllTrace, 'PollForPendingEvents Unsubscribe(%) failed count=%',
[pointer(sock), fCount], self);
if fUnsubscribeShutdownSocket then
sock.ShutdownAndClose({rdwr=}false);
end;With the fix, the same load ran about 1,950 open/close requests (plain HTTP and TLS) with no FastMM report.
I can't open a pull request, so here is the change as a git diff against current master (2.4.18151); it applies with `git apply` from the repository root.
diff --git a/src/net/mormot.net.sock.pas b/src/net/mormot.net.sock.pas
index 29a6e71..0c03193 100644
--- a/src/net/mormot.net.sock.pas
+++ b/src/net/mormot.net.sock.pas
@@ -5564,22 +5564,26 @@ begin
(sub.UnsubscribeCount <> 0))then
fOnLog(sllTrace, 'PollForPendingEvents sub=% unsub=%',
[sub.SubscribeCount, sub.UnsubscribeCount], self);
- // ensure subscribe + unsubscribe pairs are ignored
- if not fUnsubscribeShutdownSocket then
- for u := 0 to sub.UnsubscribeCount - 1 do
- begin
- sock := sub.Unsubscribe[u];
- for s := 0 to sub.SubscribeCount - 1 do
- if sub.Subscribe[s].socket = sock then
- begin
- if Assigned(fOnLog) then
- fOnLog(sllTrace, 'PollForPendingEvents sub+unsub sock=%',
- [pointer(sock)], self);
- sub.Unsubscribe[u] := nil; // mark both no op
- sub.Subscribe[s].socket := nil;
- break;
- end;
- end;
+ // ensure subscribe + unsubscribe pairs are ignored - also with
+ // fUnsubscribeShutdownSocket (closing the socket here), otherwise the
+ // Unsubscribe below fails (not yet in fPoll[]) and the Subscribe then adds
+ // the socket with the tag of a connection which will be released by the GC
+ for u := 0 to sub.UnsubscribeCount - 1 do
+ begin
+ sock := sub.Unsubscribe[u];
+ for s := 0 to sub.SubscribeCount - 1 do
+ if sub.Subscribe[s].socket = sock then
+ begin
+ if Assigned(fOnLog) then
+ fOnLog(sllTrace, 'PollForPendingEvents sub+unsub sock=%',
+ [pointer(sock)], self);
+ if fUnsubscribeShutdownSocket then
+ sock.ShutdownAndClose({rdwr=}false);
+ sub.Unsubscribe[u] := nil; // mark both no op
+ sub.Subscribe[s].socket := nil;
+ break;
+ end;
+ end;
// use fPoll[] to retrieve any pending notifications
fPollLock.Lock;
try
@@ -5601,9 +5605,13 @@ begin
break;
end;
if sock <> nil then
+ begin
if Assigned(fOnLog) then
fOnLog(sllTrace, 'PollForPendingEvents Unsubscribe(%) failed count=%',
[pointer(sock), fCount], self);
+ if fUnsubscribeShutdownSocket then
+ sock.ShutdownAndClose({rdwr=}false); // caller expects it closed
+ end;
end;
// then subscribe to the new connections
for s := 0 to sub.SubscribeCount - 1 doOffline