mORMot Open Source

MacIn

Member

Registered: 2018-12-12

Posts: 4

THttpApiWebSocketServer multithreading

Hi, All!
I'm looking for a networking library to replace Indy and Synopse networking modules seem as a perfect fit. First of all, why I'm looking for other libraries is because "1 connection = 1 thread" doesn't allow some to service, say, 10000 connections at the same time, because you can't have 10000 threads in Win 32 bit app.
As I understand from the documentation, Synopse networking modules use asynchronous model over http.sys and thread pools, which should allow having multiple persistent simultaneous connections.

However, when I started experimenting with it, I've stumbled upon HTTP stage, which seem strange.
First of all, example 31 (I'm going to use http and websocket connections) has following example:

constructor TSimpleWebsocketServer.Create;
begin
  fServer := THttpApiWebSocketServer.Create(false, 8, 10000);
  fServer.AddUrl('','8888', False, 'localhost');
  fServer.AddUrlWebSocket('whatever', '8888', False, 'localhost');
  fServer.RegisterProtocol('meow', False, onAccept, onMessage, onConnect, onDisconnect);
  fServer.RegisterCompress(CompressDeflate);
  fServer.OnRequest := onHttpRequest;
  fServer.Clone(8);
end;

This example, as I understand, specifies two pools: 1) 8 server clones means there are 8 synchronous (?) http server threads processing http requests,
and the 8 here:

  fServer := THttpApiWebSocketServer.Create(false, 8, 10000);

creates thread pool of 8 worker threads for websocket connections.
Am I right regarding these two pools? In case there are, say, 100 live connections, these 8 websocket worker threads will service them in round-robin fashion?

Now with the http part. For experiment, I've used simple:

function TSimpleWebsocketServer.onHttpRequest(Ctxt: THttpServerRequest): cardinal;
begin
  Result := 200;
  sleep(60*1000);
end;

After first request "falls " into sleep mode, I'm expecting 2nd, 3rd etc to arrive, because server is cloned 8 times. But subsequent requests do not trigger the .onHttpRequest until first thread awakens. Other threads(servers) just do not return from

      Err := Http.ReceiveHttpRequest(fReqQueue,ReqID,0,Req^,length(ReqBuf),bytesRead);

before sleep of first thread is over, like it was single-threaded, like there isn't a call to the .clone; but .clone is on place.
I clearly understand, that I'm missing some part here, but don't get, what it might be.

Also, since .onHttpRequest calls are blocking, if 8 requests do require long processing, server would stall, even if 9th request is simple, say, resolves immediately with 403. As I understand, I'd need to implement some custom event-driven addition to overcome this issue, but returning response is also synchronous, so I'm not sure, if it is doable in mormot framework. Probably there are some other components of mORMot that already do something similar that I could use as reference?

Is it doable at all (due to syncrhonous nature of http processing - you can't take the thread in THttpApiServer and redirect it to other duties before request has been processed, as I understand from the code).

Thank you in advance.

alpinistbg

Member

Registered: 2014-11-12

Posts: 124

Re: THttpApiWebSocketServer multithreading

Hi MacIn,

Perhaps it will be slightly out of 'Synopse' topic, but you can not expect to handle 10K simultaneous connections with a pool of threads, Win32 have practical limit of about 2K threads per process. Hi-performance servers like http.sys, nginx actually all works with some sort of 'callbacks' to overcome this limitation. http.sys  works with IO completion ports, which is similar to a callback. The other issue is that threads also include significant overhead in memory usage and context switching. So, if you have such a tough requirements you should consider some kind of queueing, callbacks or perhaps protothreads/coroutines.

Regards,

MacIn

Member

Registered: 2018-12-12

Posts: 4

Re: THttpApiWebSocketServer multithreading

Hi MacIn,

Perhaps it will be slightly out of 'Synopse' topic, but you can not expect to handle 10K simultaneous connections with a pool of threads, Win32 have practical limit of about 2K threads per process. Hi-performance servers like http.sys, nginx actually all works with some sort of 'callbacks' to overcome this limitation.

It is not off-topic, but plainly wrong, since I have said same thing in my question. I have began my post with statement about problem with "1 connection = 1 thread" which is not suitable for handling lot of connections in win 32 app:

First of all, why I'm looking for other libraries is because "1 connection = 1 thread" doesn't allow some to service, say, 10000 connections at the same time, because you can't have 10000 threads in Win 32 bit app.

This is the second sentence of my question.
Thread pool term can be used in different contexts, including async handling, just as I stated, when few threads do serve many connections. That also would be thread pool, it all depends on how threads do interact with connection. Per description of Synopse mORMmot I concluded that it can handle many connections. Connections themselves are handled by http.sys, which is used by Synopse networking modules (SynCrtSocket) and arrived queries are handled by pool of threads (for websockets layer)/set of threads (for usual HTTP queries layer)

Last edited by MacIn (2018-12-12 14:55:59)

alpinistbg

Member

Registered: 2014-11-12

Posts: 124

Re: THttpApiWebSocketServer multithreading

It is not off-topic, but plainly wrong, since I have said same thing in my question.

Oops, you've got me here!  Thanks for pointing me that I've said something "plainly wrong" but "the same thing in (my) question"
And yes, pool of threads can handle anything, including a single thread (which can be seen as a trivial case of a pool) can do anything...

You said also:

After first request "falls " into sleep mode, I'm expecting 2nd, 3rd etc to arrive, because server is cloned 8 times. But subsequent requests do not trigger the .onHttpRequest until first thread awakens. Other threads(servers) just do not return from

      Err := Http.ReceiveHttpRequest(fReqQueue,ReqID,0,Req^,length(ReqBuf),bytesRead);

before sleep of first thread is over, like it was single-threaded...

I presume you've meant the call in the method THttpApiServer.Execute.

Actually this can happen only in the case when ReceiveHttpRequest() was called in a synchronous manner (it is) and there is no request arrived in the specified queue.
See: https://docs.microsoft.com/en-us/window … ttprequest

I suppose you should check the way you generate the requests first.

Regards,

MacIn

Member

Registered: 2018-12-12

Posts: 4

Re: THttpApiWebSocketServer multithreading

It is not off-topic, but plainly wrong, since I have said same thing in my question.

Oops, you've got me here!  Thanks for pointing me that I've said something "plainly wrong" but "the same thing in (my) question"

ROFL. I was meant "it is plainly wrong to present something I've already said as a contradiction to my statement". Linguistic barrier, you know

Actually this can happen only in the case when ReceiveHttpRequest() was called in a synchronous manner (it is) and there is no request arrived in the specified queue.
See: https://docs.microsoft.com/en-us/window … ttprequest

I suppose you should check the way you generate the requests first.

That's what I thought - it should wait only if there's no requests. Yes, I'm just opening several pages in browser, so it might be some sort of its internal caching logic comes into play. Will try something better, thank you; just for starters I wanted to be sure that I understand logic of SynCrtSocket module correctly. Am I right regarding second thread pool (for websockets)? So it has separate pool for websocket requests and set of threads (cloned servers) for http stage?
Thanks in advance.

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

Re: THttpApiWebSocketServer multithreading

Both thread pool (for web socket and for HTTP) works in the same way. And you can handle 10 000 connections even on Win32.

For real life project we use THttpApiWebSocketServer just for sending a messages to the user with one WebSocket thread (to notify a user something he is interesting on are changed in the database). After receiving a notification message from WebSocket client made a HTTP request to get a real data (HTTP server use up to 24 HTTP thread .Clone(24) )

I'm personally do not verify on production how stable webSocket part work neither with several thread nor in case of some logic inside are inside websocket worker (but suppose it's work).

In our latest project we switch to Linux. Instead of THttpApiWebSocketServer we use a RabbitMQ (it have a WebSocket interface) as a message broker.

Last edited by mpv (2018-12-12 18:39:24)

alpinistbg

Member

Registered: 2014-11-12

Posts: 124

Re: THttpApiWebSocketServer multithreading

@MacIn,

AFAIK, the http.sys is well recognized by its high performance and since THttpApiWebSocketServer(THttpApiServer) takes an advantage of it, it should be a good choice for this kind of heavy duty work.

My advice is to have a good unit tests and to continue with caution, because IMHO the authors have put the big emphasis on the rapid expansion of the functionality rather on the robustness ... Either way the Delphi/Win32 target should be on the safe side.

BTW, should the app be oriented to a certain established protocol or just need to serve many subscribers on a custom-made one?

MacIn

Member

Registered: 2018-12-12

Posts: 4

Re: THttpApiWebSocketServer multithreading

Both thread pool (for web socket and for HTTP) works in the same way. And you can handle 10 000 connections even on Win32.

I didn't say it is impossible in Win32 as such. I have said it is impossible in Win32 using Indy-alike libraries that launch 1 thread for each connection.

In our latest project we switch to Linux. Instead of THttpApiWebSocketServer we use a RabbitMQ (it have a WebSocket interface) as a message broker.

Thanks, but I have to stick to NT.

the http.sys is well recognized by its high performance and since THttpApiWebSocketServer(THttpApiServer) takes an advantage of it, it should be a good choice for this kind of heavy duty work.

That's exactly why I have chosen it. Just stumbled upon the issue described above.

BTW, should the app be oriented to a certain established protocol or just need to serve many subscribers on a custom-made one?

Standard HTTP + custom protocol over WebSockets. It is done with Indy now, but due to win32 threads limitation I'm switching to more suitable lib, which Synopse modules seem to be. That will be sort of filtering/load balancing server that would have dozen of production servers behind it.