mORMot Open Source

vs

Member

Registered: 2019-10-21

Posts: 70

SChannel server: ESChannel WSAENOTSOCK on stop with keep-alive client

Hi Arnaud,

Thanks a lot for the SChannel server work - we switched our Windows servers from OpenSSL to SChannel this week, with the certificate taken from the LocalMachine\My store and passed as TNetTlsContext.CertificateRaw.
Tested live on two of our products, server side and client side (TSimpleHttpClient socket mode to an OAuth + GraphQL API, CloudFront video with Range) - everything works, and we could drop libssl/libcrypto from our Windows deployments.

One thing we noticed: when the server is stopped while a browser still holds an idle keep-alive TLS connection, each such connection logs an exception during shutdown:

EXC   ESChannel {Message:"shutdown: Fatal Error 10038 WSAENOTSOCK"} [#40 conn-1] at
  TSChannelNetTls.SendRawAll    (mormot.net.sock.windows.inc 2795)
  TSChannelNetTls.CleanShutdown (mormot.net.sock.windows.inc 3204)
  TSChannelNetTls.Destroy       (mormot.net.sock.windows.inc 3220)
  TCrtSocket.Close              (mormot.net.sock.pas 7616)
  TCrtSocket.Destroy            (mormot.net.sock.pas 7652)
  THttpSocket.Destroy           (mormot.net.http.pas 4977)
  FreeAndNilSafe
  THttpServerResp.Execute       (mormot.net.server.pas 5714 - cleanup)

Setup: 2.4.17809 (c3f4abb0d), FPC 3.2.3 Win64, THttpServer (useHttpSocket) + secTLS, SChannel server, CertificateRaw.
Repro: open the page over HTTPS (a plain login is enough), leave the tab open, stop the server.
Without any client connected, the stop is clean.
Reproduced 3/3 on a Debug build.

My reading: at shutdown the server closes the client handle to unblock the connection thread (by design), then the thread's own cleanup reaches TSChannelNetTls.Destroy - CleanShutdown - SendRawAll on the already-closed handle.
WSAENOTSOCK maps to nrInvalidParameter, which SendRawAll raises.
CleanShutdown absorbs it, so it is "only" log noise - with OpenSSL, SSL_shutdown() fails silently in the same situation, which is why we never saw it before.

Possible ways (your call, of course):
- in the shutdown path, treat nrInvalidParameter / WSAENOTSOCK like nrClosed (the peer is gone, nothing to flush);
- or skip CleanShutdown in Destroy when the socket handle is no longer valid - the "> 0" check is not enough once another thread closed it.

Once (n=1, Release build, after ~25 min of activity and a video stream) the same shutdown also left 10 unfreed blocks and heaptrc crashed with an AV at exit.
It did NOT reproduce in the three Debug runs, so please consider it unconfirmed - I mention it only in case it rings a bell.

I checked your commits up to 5df7dc80 (05.11): the client-side CertificateRaw work is great, but this shutdown path is unchanged.

Happy to test a fix on our side.

sh17

Member

From: Dresden/Germany

Registered: 2011-08-09

Posts: 73

Website

Re: SChannel server: ESChannel WSAENOTSOCK on stop with keep-alive client

Thanks for the detailed report - your analysis is right, and the root cause is a bit more serious than a logged exception.

On Windows, TCrtSocket.Abort (called by THttpServerResp.Shutdown from THttpServer.Destroy) calls closesocket() to release the blocking recv() of the connection thread - shutdown() is not enough on Windows, I checked. But fSock keeps its value, so when the connection thread then closes its socket, TCrtSocket.Close:

releases the TLS layer, and TSChannelNetTls.Destroy sends its close_notify over the already closed handle -> WSAENOTSOCK -> the ESChannel you see (absorbed, but logged);
then calls ShutdownAndClose = a second closesocket() on the same value.
Windows reuses SOCKET values immediately, so this handle may already belong to another connection of the process: the close_notify could be sent to it, and the second closesocket() closes it. The regression test reproduces exactly this: without the fix, a socket opened just after Abort() is closed by the aborted one.

Proposed fix, as a single commit:
https://github.com/synopse/mORMot2/comp … bort-close

TCrtSocket.Abort sets a new fAbortClosed flag before its closesocket() (Windows only) - before, since the woken up thread calls Close at once;
TCrtSocket.Close then skips the second closesocket() and calls a new INetTls.SocketClosed method before releasing the TLS layer, so SChannel won't send close_notify any more, and OpenSSL won't call SSL_shutdown();
new TNetworkProtocols.CrtSocketAbort test, with a fake INetTls (no certificate needed): sequential Abort + Close, and Abort releasing a recv() blocked in another thread.
Tested with FPC 3.3.1 on Win64 aarch64 and Linux aarch64 (TNetworkProtocols), compiled with Delphi 13 Win64.

@vs: could you try this branch with your keep-alive browser scenario, and see if the release-mode leak / heaptrc AV also disappears? I could not reproduce that one, so it may be unrelated.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,627

Website

Re: SChannel server: ESChannel WSAENOTSOCK on stop with keep-alive client

Thanks Vladi for the detailed report and reproducer.

Your analysis is correct. On Windows, `TCrtSocket.Abort` closes the underlying socket to unblock the connection thread, but the subsequent `TCrtSocket.Close` could still attempt TLS shutdown and close the same socket handle again.

The Sven's proposed fix tracks whether `Abort` has already closed the socket, skips the duplicate close, and notifies the TLS backend so that neither SChannel nor OpenSSL attempts to send `close_notify` over an invalid handle.
The patch also includes regression tests for the abort/close lifecycle.

I did integrate the fix into mORMot2 trunk after the usual build and test checks.
Merged as https://github.com/synopse/mORMot2/pull/770

Regarding the heaptrc AV and leaked blocks, I would keep that as a separate, unconfirmed issue for now. Please let us know if it reproduces with the fix applied.

Thanks again for testing SChannel in production and for contributing the fix!