mORMot Open Source

Hi Arnaud,

Thank you for the quick and insightful feedback.

That makes perfect sense. My use case doesn't involve thousands of simultaneous clients, so the mormot.net.async complexity would indeed be overkill.

The new TCrtSocket.RetryCount property is exactly what I needed to safely check for nrRetry / WSAEWOULDBLOCK without overhead. I have already updated my local mORMot 2 files to include your latest commit and I am adjusting my send logic to use it.

I appreciate you taking the time to add this property to the framework. It makes the flow control much cleaner for my implementation.

Thank you for the clarification. You are right; I am using TWebCrtSocketProcess.SendFrame() on the client side, and I now understand that it blocks once the OS TCP buffer is full.

Measuring the execution time with QueryPerformanceMicroSeconds() is a good suggestion for a reactive approach. However, to avoid the 'blocking' state altogether and keep the UI or other tasks fluid, would it be feasible to check the socket's write readiness before calling SendFrame()?

On the client side, could we use something like fSocket.WaitFor(1, [neWrite]) to see if the kernel is ready to accept more data without blocking?

If fSocket (TCrtSocket) could expose a way to check if send() would block, we could implement the same threshold-based logic we now have on the server, but based on 'Socket Readiness' instead of 'Pending Bytes'.

What do you think about this 'proactive' check for the client-side TCrtSocket?

Hi Arnaud,

Thank you very much for the quick fix!

This change is very helpful for monitoring connection health and latency in our infrastructure. I've already tested it and it works perfectly.

Hi Arnaud,

Thank you again for the PendingWrite implementation in the async server; it works perfectly for managing backpressure on the server side.

Following that logic, I am now tuning the Client Side (high-throughput frame injection). I noticed that when a client sends a massive amount of data in a loop, it can still hit a "blocking" state or saturate the local kernel buffers if the producer is faster than the network, sometimes leading to local freezing if not handled with manual Sleep() calls.

Is there (or could there be) a similar way to monitor the outbound buffer state on the Client classes (e.g., TWebSocketClient or the underlying TRawSocket)?

Having a Client.PendingWrite (or a way to check if the outgoing TCP window is full) would allow us to implement the same "threshold-based pause" on the client, ensuring a graceful flow control without relying on arbitrary Sleep() intervals.

What do you think about exposing this metrics-gathering for client-side sockets as well?

Best regards,
ec

Hi Arnaud,

Thank you for the insights.

Actually, dropping frames or flushing the buffers is exactly what I would like to avoid. My goal is to ensure data integrity without losing any frames, even when the RAM reaches a critical point.

Instead of discarding data, I am looking for a way to implement proper backpressure within the mORMot 2 async server architecture. My intention is to have the server 'wait' or slow down the injection of new frames until the current buffers are successfully transmitted and acknowledged at the TCP/Websocket layer.

Is there a recommended way in mORMot 2 to monitor the state of the async output buffers or the underlying socket readiness, so I can pause the data flow instead of saturating the RAM or being forced to flush pending data?

It seems more stable to me.

In Firefox, it works perfectly. The browser displays the warning: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT, as expected.

In other browsers, the exception occurs: "Project ecentric.exe raised exception class ESChannel with message '<>: HandshakeStep returned 80090327 [SEC_E_CERT_UNKNOWN], System Error 87 [ERROR_INVALID_PARAMETER]'.". The browser displays the warning: NET::ERR_CERT_AUTHORITY_INVALID. However, now it is possible to ignore this type of exception, and the execution continues normally.

Thank you!

In case anyone is interested, InitNetTlsContextSelfSignedServer works in Firefox too, with the pre-computed certificate.

But, I think ignoring this exception may not be a good idea, things may not work correctly

Ab, I did this (with and without openssl):

1. openssl genrsa -out private.key 2048
2. openssl req -new -key private.key -out request.csr
3. openssl x509 -req -days 365 -in request .csr -signkey private.key -out certificate.crt
4. I installed the certificate on Windows

When I use one of the ways:

...
FWsServer.WaitStarted(10, 'C:\Devel\eCentric-11\certificate\certificate.crt' , 'C:\Devel\eCentric-11\certificate\private.key');

Or

...
InitNetTlsContext(FTls, {server=}True, 'C:\Devel\eCentric-11\certificate\certificate.crt', 'C:\Devel\eCentric-11\certificate\private.key');
FWsServer .WaitStarted(10, @FTls);

Now the following exception is occurring:
Project ecentric.exe raised exception class ESChannel with message '<>: PFXImportCertStoreSEC_E_CERT_UNKNOWN returned 80090327 [SEC_E_CERT_UNKNOWN], System Error -2146885630 [Error during encryption operation or decoding]'.

PS C:\Windows\system32> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls13
PS C:\Windows\system32> Invoke-WebRequest -Uri https://192.168.0.128:7799
Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for the
SSL/TLS secure channel. At line:1 character:1 + Invoke-WebRequest -Uri https://192.168.0.128:7799 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo: InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc eption + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

I'm using Windows 11 on the server side. Until a few days ago, this exception was not occurring. Now the browser's websocket client causes this exception on the server!

Sorry for the inconvenience, ab. The code snippet was just intended to illustrate how I was doing it on both sides (client and server). The exception that is appearing on the server side is:
Project ecentric.exe raised exception class ESChannel with message '<>: HandshakeStep returned 80090326 [SEC_E_ILLEGAL_MESSAGE], System Error 87 [ERROR_INVALID_PARAMETER]'.

The error at the browser side is:
ERR_SSL_KEY_USAGE_INCOMPATIBLE

Hi Arnaud !

I understood.

Anyway, I noticed that with a small increase in the number of test connections, the increase in connection time is also happening with openssl, only on the server side! But, despite the increase in connection time, the server continues to work, and new connections are established normally.

Thank you very much for your attention

Hi Arnaud !

I'm not sure if this is the same problem observed by @maro.

Tested only with Default SSPI Windows API (Windows 10 and 11). I'm sure this only occurs on the server side.

Reproducing what I observed is very simple.

1) I created a websocket listener:
      FWsServer :=
        TWebSocketAsyncServer.Create('0.0.0.0:7799', Nil, Nil, 'Acceptor', 32, 30000, [
        hsoNoXPoweredHeader, hsoHeadersInterning, hsoThreadSmooting,
        hsoNoStats, hsoBan40xIP, hsoEnableTls ]);
    ...

2) The reproduction of what I observed can be done as follows:
    In Google Chrome, using two instances of the "Simple Websocket Client" extension,
    pointing to: "wss://192.168.0.109:7799/Test", it is observed that the connection of the first instance occurs normally.
    Connecting and disconnecting, with a single instance, occurs quickly.
    With the first instance connected, the second instance's connection behaves differently, the time to connect increases,
    and, after several repetitions, the server stops responding.

It only occurs when I'm using TLS.

I hope I have contributed in some way.

Thank you very much for your attention.

Good morning !

The following situation was also observed: Quickly connecting and disconnecting a single websocket client, everything works well, it is possible to connect and disconnect quickly and efficiently. From the second websocket client onwards, the behavior of new connections changes. Thereafter, there is an unusual delay in all new connections, until, finally, the new connections are rejected. In the test carried out there is no data traffic, only websocket connection and disconnection.

** This does not happen without TLS !!!

I hope I contributed in some way

Okay Ab.

I got better results as follows:

             fEvent.ResetEvent;
             fWaitForReadPending := true; // should be set before wakeup
             if new <> 0 then
               fOwner.ThreadPollingWakeup(new)
             else
           { ec...
           } begin
                new := fOwner.fClients.fRead.PollForPendingEvents(ms);
                If new = 0 Then Begin
                   TThread.Sleep(1);
                   Continue;
                End;
                fOwner.ThreadPollingWakeup(new);
             end;

Or simply ...

        atpReadPoll:
          // main thread will just fill pending events from socket polls
          // (no process because a faulty service would delay all reading)
          begin
            fWaitForReadPending := false;
          { new := fOwner.fClients.fRead.PollForPendingEvents(ms);
          } new := fOwner.fClients.fRead.PollForPendingEvents(ms + 1100); // << it works too

I hope it helps !

Thank you very much for your attention.

It didn't stabilize! It stopped working, with the same symptoms as before, after an hour of testing

Goodnight !

Ab, me, again. On the same subject!

...
procedure TAsyncConnectionsThread.Execute;
...
{$ifdef OSWINDOWS}
{ start := -1; // ensure never SleepHiRes(0)
} start := 1; // ec
  {$else}
   start := 0; // best reactivity
{$endif OSWINDOWS}
...

After two hours of testing (30 clients), with the start value = -1, the following situation occurs:

-One of the connections stops working. The other 29 are still working;
-No new connections to the server are possible;
-CPU consumption goes up to 50% or more.

Changing the value of start to 1, and adding another 70 clients, no problem occurs and, strangely, one has the impression of better performance and stability!

What do you think ?

Good afternoon !

About the high CPU consumption, I tested it a little while ago and apparently it works correctly. See if that makes any sense to you...
Excessive context switching of threads is causing high CPU consumption.
At mormot.cor.windows.inc we have:
...
procedure SleepHiRes(ms: cardinal);
begin
   if ms <> 0 then
     Windows.Sleep(ms) // follow the HW timer: typically up to 16ms on Windows
   else
     SwitchToThread; // <-- Executed millions of times per minute
end;
...

At mormot.net.async.pas
...
procedure TAsyncConnectionsThread.Execute;
...
         atpReadPoll:
           // main thread will just fill pending events from socket polls
           // (no process because a faulty service would delay all reading)
           begin
           { start := 0;
           } start := 1; // < -- My Test is this. It is the same as SleepHiRes(1)
             while not Terminated do
...

Disregarding possible side effects, it works !!!

What do you think ?

I understood. Anyway, thank you immensely for your attention. I will provide the logs as soon as possible.

Thanks for the quick feedback.

Well, the question is exactly this. When using a ServerThreadPoolCount > 1 (4, 8, or the default 32) CPU consumption increases substantially, up to 20%, for no processing no Delphi 11.3