Problem migrating THttpApiServer -> THttpServer

mpv · 2018-01-09 14:23:30

While migrating my projects from Windows/Delphi to Linux/FPC I discovered a problem with HTTP server threading model and need an advice how to solve it.
The problem:
- In case of THttpApiServer, we have a fixed size thread pool of "workers" where I can run a business logic, URI methods are passed to these threads to be executed, the number of threads in the pool is defined in a call to THttpApiServer.Clone(threadPoolSize), threads in the pool are stopped when HTTP server is stopped
- in case of THttpServer "worker" threads are created and stopped depending on client's Keep-Alive and request size

Every "worker" thread in my case creates a database connection using ThreadSafeConnection and JS engine, thus, since we have a fixed size thread pool (in case of HTTP API server) we know the number of connections to a database will not exceed threadPoolSize.
In the case of THttpServer every Keep-Alive session will create its own thread and in my case DB connection + JS engine. This thread is terminated and closes DB connection and release JS engine after keepalive timeout is expired.

Connecting/disconnecting to a database can produce a huge overhead. In case of Oracle, we have very hi-load systems and cost of connecting is very expensive, in case of Postgre even for a small load connect is expensive, because of Postgre forks on every connection. And initializing of JS engine is expensive too.

That's why we definitely need to have I fixed size thread pool of "worker" threads for THttpServer, the same as in case of THttpApiServer.

The possible solution:
I can create my own TSynThreadPoolTHttpServer descendant to implement my requirements (and make a THttpServer.ThreadPool property writable). Or can you suggest a better solution?

ab · 2018-01-09 14:52:23

Could you not just use a nginx front-end with keep-alive on this side, and non persistent HTTP connections on the mORMot loopback server?
Then it will use the thread pool of THttpServer.

mpv · 2018-01-09 15:19:05

I'll try this idea and answer about results here. Looks like the variable THREADPOOL_BIGBODYSIZE should be set to infinity for this case, but this is a solution, for sure. Thanks!

ab · 2018-01-09 16:13:47

Otherwise we could with not much problem implement an event-driven HTTP server, starting from TPollSockets for instance.
But if we can just tune a nginx front-end, it is a better solution, since nginx is very much tuned and proven.

Please forward your finding about this matter!
Thanks!

mpv · 2018-01-09 19:09:19

I agree turning nginx is a "Unix way" - mORMot/SyNode for business logic, nginx for HTTP, static, cache. I will publish a final nginx config/mORMot patch after solve all problems.

mpv · 2018-01-10 13:11:57

I'm made two small changes in SynCrtSock - one for disable KeepAlive for TWinHTTP and one to prevent reading on GET request body if Content-Length header is missing (browsers do not send a Content-Length header for GET requests). See - this commit. For second change - see this git commit I think we can remove and (KeepAliveClient or (fMethod = 'GET')) condition at all?

ab · 2018-01-10 13:59:39

AFAIR it seems to me that

    if (ContentLength<0) and (KeepAliveClient or (fMethod = 'GET')) then

was needed for some clients...

mpv · 2018-01-10 15:28:07

@ab - can you, please, accept a 69 merge request. It make a ResolveIPToName function the same for Delphi & FPC.....

ab · 2018-01-10 16:06:56

Merge request accepted/reviewed.

mpv · 2018-01-10 16:46:28

Thanks!

mpv · 2018-01-26 17:59:56

Status report: After fixing unexpected THTTPServer main loop termination - see pull request 78 and explicitly force server to use a thread pool only by introducing a DisableKeepAlive property we able to run out software on Linux behind a nginx with a good performance.
One more feature what should be implemented is sending files asynchronous like in HTTP.sys by using X-Accel-Redirect nginx feature, and we will be ready to replace Windows+HTTP.sys to Linux+nginx on hiload

ab · 2018-01-26 18:32:29

Good news!

I made some changes after this pull request, which broke some part of the framework.
See https://synopse.info/fossil/info/1174112c23826fa7
But there was indeed a nasty bug!

About X-Accel-Redirect it will be easy to add, I guess - and very helpful in respect to the current load-in-memory-then-send-on-the-loopback implementation.
Is there similar options for other web servers?

mpv · 2018-01-26 18:40:47

Yes, Lighttpd has this feature and there is a mod_xsendfile for Apache

ab · 2018-01-26 19:26:16

Check https://synopse.info/fossil/info/0d130cce62
(not tested....)

ssoftpro · 2018-01-27 10:46:53

Dear @ab!

Thanks a lot for the latest changes and sorry for broke some other parts of the framework.
I'm currently trying to wrap up all the things and found that you missed ThreadPoolContentionAbortDelay property initialization. And currently this is a read-only property, should it be writable, what do you think?

ssoftpro · 2018-01-27 11:24:58

And one more thing - it could be handy to get back ThreadPoolContentionCount property but increase it only once - just before the repeat loop starts. This would allow average contention delay calculation dividing ThreadPoolContentionTime by ThreadPoolContentionCount

ab · 2018-01-27 14:56:47

Please check https://synopse.info/fossil/info/b94adf736c

mpv · 2018-01-28 09:28:51

Yes, latest changes what enable to register several folders for Nginx is exactly what we need. Thanks a lot! (not tested yet)

mpv · 2018-01-28 10:04:23

I'm fix implementation of THttpServer.OnNginxAllowSend - see [f3080e88a]. Previous not compiled in Delphi and work with fNginxSendFileFrom as a string instead of array..

ab · 2018-01-28 11:10:38

fNginxSendFileFrom[] was changed into an array, and your fix was incorrect due to wrongfNginxSendFileFrom[f] index, and a non intialized n variable.
Please check https://synopse.info/fossil/info/c2e8707720

ssoftpro · 2018-01-28 14:50:03

On my side was a check of contention parameters calculation and I've found that it is needed to add 'tix := GetTickCount;' assignment immediately after SleepHiRes block. Otherwise, the last loop before a successful fThreadPoolPush call is not counted in fThreadPoolContentionTime.
Removing 'break' after a successful fThreadPoolPush call is not an alternative because we should not include the time spent on the fThreadPoolPush call itself.

ab · 2018-01-28 15:25:20

I guess that the time spent on the fThreadPoolPush call itself is much lower than one millisecond - so won't inflect the whole time reported.
Less than a microsecond for sure, to loop over the whole threads list.

But flow is more obvious with an explicit break, anyway.

See https://synopse.info/fossil/info/2c1e8aeb7d

mpv · 2018-02-23 14:03:08

I will merge curl http fix to the trunk manually (Sergey make pull request to the fb_synode_311 - this is wrong)
Another fix in pull request 83 - about usage of HTTP proxy under Linux. We fix it on the SyNode level by getting a default proxy server from http_proxy environment variable - this is a common way to specify a proxy under Linux. But I think will be good do this to the SynCrtSock. @ab - what do you think?

mpv · 2018-02-23 14:18:32

@ab - I commit to fossil, but can't commit to github (my company setup a app firewall ) Please, apply SynCrtSock changes to github trunk..

UPD. I push it...

Last edited by mpv (2018-02-23 14:24:11)

ab · 2018-02-23 14:44:57

Nice TCurlHTTP issue catch!

But for http_proxy, sometimes it is not needed (e.g. if server is on loopback)...
Perhaps it is up to the client application to properly set the proxy value, according to http_proxy, before calling SynCrtSock...

mpv · 2018-02-23 16:10:20

Ok. Let's it live inside SyNode. In case certain app need proxy I can run it as `http_proxy=http://bla-bla.com && myapp'. We use a pm2 or docker as a process managers, for both of them configuration using environment variables is a native way.

cybexr · 2018-06-19 07:43:04

@mpv Could you pls paste your nginx configuration ? Same keep-alive performance prob occurs, and not familiar with nginx, That will be very helpful to me !

mpv · 2018-06-25 14:42:49

Don't forgot to set a THttpServer.RemoteIPHeader, disable KeepAlive by setting a KeepAliveTimeOut parameter to 0 duruing call to THttpServer.create(). And use a THttpServer.NginxSendFileFrom to send static files (if you need it - see internal location definition below)

The simplest config:

server {
    listen       80 default_server;
    listen       [::]:80 default_server;

    #proxy all requests to the mORMot backend
    location / {
        proxy_pass          http://localhost:8881;
        proxy_set_header	Host    $host;
        proxy_set_header	X-Real-IP  $realip_remote_addr;
        proxy_redirect      off;
    }
    #BLOB stores internal locations
    location  /ubstatic-autotest/simple {
        internal;
        alias D:/app/documents;
    }
}

Last edited by mpv (2018-06-25 14:44:19)

cybexr · 2018-07-06 09:22:27

thanks a lot mpv, just have made nginx_proxy_mormot works fine on linux, and performance seems great!
Tested with Jmeter (60 thread target 6 service-url ,and test 3 round), backend mormot is SOA-service, mainly issue sql query aginst oracle-db then return JSON to Jmeter.

client-Jmeter keepalive, Post to mormot, result is: 466KB/sec, 7QPS;
client-Jmeter keepalive, Post to nginx_proxy_mormot, result is: 13MB/sec, 190QPS;

client-Jmeter shortconnect , Post to mormot, result is: 15MB/sec, 225QPS;
client-Jmeter shortconnect , Post to nginx_proxy_mormot, result is: 14MB/sec, 207QPS;

when service on windows HTTP.sys:
client-Jmeter keepalive, Post to mormot(HTTP.sys), result is: 4MB/sec, 70QPS;

good news for nginx&mormot@linux

mpv · 2018-08-28 13:08:39

I'm discover a problem while working with THttpRequest under Linux using libcurl. THttpRequest expect what library implementation will throw in case of errors, and TWinINet / TWinHTTP throw. But TCurlHTTP.InternalRetrieveAnswer just return 404 as status and a libcurl error in body. This cause a different behavior for windows/linux. I'm create a pull request #127 to fix this problem.

@ab - please, merge it to master

mpv · 2018-10-05 09:59:06

@ab - about your improvements for Linux HTTP server thread pool.

It's a big step forward, but endless length of PendingContext queue will cause several problems:
1) critical for me - we lost ability to gracefully load balance our server farms since all request will be putted to the queue of first server in the balancing pool
2) "out of memory" in case of huge amount of pending requests in queue

I consider to create a fixed length PendingContext array because:
- we already have something similar for HTTP.SYS in THttpApiServer.HTTPQueueLength. So HTTPQueueLength property can be moved to THttpServerGeneric level and used by both type of servers.
- we can allocate fixed length fPendingContext on startup and use it as a Ring-Cache - this prevent memory move inside TSynThreadPool.PopPendingContext
- in case cache is overflow server should response 503 Service Unavailable ASAP - in this case load balancer (at last nginx) will mark this server as unavailable for a while ( see Health checks section here)

The default HTTPQueueLength in HTTP.SYS is 1000 and this enough for 99% of my apps.
In my practice I increase this value only once to 5000 because we do not have a load balancing where. Usually we keep it default and relay on load balancing.

What do you think - is it possible to implement fixed length pending queue?

ab · 2018-10-05 11:03:06

Good idea.
I was waiting for feedback for you, indeed.

So I guess that if the queue is too big, the input socket should be rejected/shutdown immediately, so that load-balancing (via nginx?) will be possible?

Do you make a pull request or do you want me to do it?

mpv · 2018-10-05 12:01:33

I'm not sure for 100% socket rejection/shutdown is OK. HTTP.sys report 503 if request queue is overflow. But I think nginx is smart enough to decide socket rejection is equal to "response from a particular server fails with an error" as described in their documentation.

If you can, I would prefer you to make such a complex change, if not - I can try to write a pull request on weekend.

ab · 2018-10-06 12:24:28

503 seems fine - but it needs to parse the input before sending the output, or do you think it is fine to send the 503 error header directly then shutdown the socket?

mpv · 2018-10-07 17:40:43

I look at the problem from different points and now I quite sure your first propose abut "input socket should be rejected/shutdown immediately" is a good choice.
I found documentation about what is unsuccessful attempt for nginx, so even more - we do not need to "accept" incoming connection if our http queue is overflow.

Having this config

location / {
        proxy_pass              http://balancing_upstream;
        proxy_next_upstream     error timeout invalid_header http_500 http_503;
        proxy_connect_timeout   2;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
}

- in case we do not accept incoming connection Nginx will wait for 2 second (proxy_connect_timeout = 2) and use next server from upstream.

- in case connection as already accepted but we shutdown socket without sending anything nginx will use next server from upstream because of invalid_header parameter in proxy_next_upstream directive

- in case we have one mORMot server we just keep a default proxy_connect_timeout (60 second) timeout and nginx will wait for our server to "accept" incoming connection

ab · 2018-10-08 10:15:12

Please see https://synopse.info/fossil/info/714d2e83a8

I tested it with https://github.com/synopse/mORMot/tree/ … 0Benchmark and

ab -n 10000 -c 1000 http://localhost:8888/root/abc

command.

See https://gist.github.com/synopse/c0b2346 … 608636bf7b
with HTTPQueueLength set to 100.

Failed requests:       472

And https://gist.github.com/synopse/98c2343 … 39a027bd04
with HTTPQueueLength set to 1000 (default).

Failed requests:        0

Note that performance numbers seem pretty consistent.
80-100MB/s with a SQlite3 request returning 78,000 bytes of JSON, on both Windows (http.sys) and Linux (our socket server).
(I run abbench from the Windows WSL layer, so actual performance on a real Linux server would actually be higher)

mpv · 2018-10-08 15:07:06

Perfect! This is exactly what i need.
All my CI are passed for all environment (Linux, Windows). Today I put this version on the pre-production environments for a several days. So we can see how it work in the real life. Thanks again!

mpv · 2018-12-05 18:22:03

Since last post we setup two production on Linux with latest Socket server changes, both behind nginx as a reverse proxy. Client is browser. Real load is small - average is 10RPS, max is 600 RPS. All work as expected. And this is GOOD

But two days ago I try to setup Kerberos (and LDAP-baser custom auth) for server behind nginx and found a problem for both HTTP.SYS / Socket based server types.

The problem is in client ConnectionID used to identify NTLM/Kerberos and LDAP in my case. Situation is the same as with Real IP described above. We should "proxy" connectionID from nginx side.

I create a pull request 162 and introduce a RemoteConnIDHeader property (as we did with RemoteIPHeader).

But some changes need to be discussed:

- I use need to SynCommons for GetInt64 function. May be move functions required by SynCrtSock from SynCommons to SynStrUtils ( we have a copy of the same functions in both SynCommons and CrtSock - UpperCase, GetCardinal, Trim etc.). Or use a SynCommons in CrtSock?
- I think we should change ConnectionID type from Integer to Int64. With a load of 3000RPS we overflow Int for 8 days

mpv · 2018-12-06 19:23:35

Thanks to @Chaa & @ab for review. All issues is resolved. Actually I miss GetNextItemUInt64 in SynCrtSock.

Today I test RemoteConnIDHeader functionality on Windows with both Socket/HTTP.SYS based servers with and without reverse proxy and browser client (both FF & Chrome). NTLM auth work as expected now.
Hope tomorrow I setup a Kerberos on my domain and test a Linux part, but even without it request can be merged into master IMHO.

mpv · 2018-12-08 15:28:33

I'm still have a troubles with Active Directory based domain and server on Unix, but this is likely my infrastructure problems. @ab - please, merge #162 into master - at last on Windows it verified and works. I need this path for my productions...

mpv · 2018-12-10 16:35:27

Ups, I did it again Forgot to propagate RemoteIPHeader & RemoteConnIDHeader to HttpApi server clones - please apply pull #165

mpv · 2018-12-14 16:54:32

Just for information - we successfully verify Negotiate authentication when server side is on Linux, Domain is Active Directory and clients is a Chromium browser.
In case server side is HTTP1.1 with KeepAlive then all work without any additional step

In case of server is HTTP 1.0 (no keep alive) or behind nginx:

// in Pascal server 
HttpServer.RemoteConnIDHeader := 'X-Conn-ID'

// in nginx config
proxy_set_header        X-Conn-ID       $connection

mORMot Open Source

#1 2018-01-09 14:23:30

Problem migrating THttpApiServer -> THttpServer

#2 2018-01-09 14:52:23

Re: Problem migrating THttpApiServer -> THttpServer

#3 2018-01-09 15:19:05

Re: Problem migrating THttpApiServer -> THttpServer

#4 2018-01-09 16:13:47

Re: Problem migrating THttpApiServer -> THttpServer

#5 2018-01-09 19:09:19

Re: Problem migrating THttpApiServer -> THttpServer

#6 2018-01-10 13:11:57

Re: Problem migrating THttpApiServer -> THttpServer

#7 2018-01-10 13:59:39

Re: Problem migrating THttpApiServer -> THttpServer

#8 2018-01-10 15:28:07

Re: Problem migrating THttpApiServer -> THttpServer

#9 2018-01-10 16:06:56

Re: Problem migrating THttpApiServer -> THttpServer

#10 2018-01-10 16:46:28

Re: Problem migrating THttpApiServer -> THttpServer

#11 2018-01-26 17:59:56

Re: Problem migrating THttpApiServer -> THttpServer

#12 2018-01-26 18:32:29

Re: Problem migrating THttpApiServer -> THttpServer

#13 2018-01-26 18:40:47

Re: Problem migrating THttpApiServer -> THttpServer

#14 2018-01-26 19:26:16

Re: Problem migrating THttpApiServer -> THttpServer

#15 2018-01-27 10:46:53

Re: Problem migrating THttpApiServer -> THttpServer

#16 2018-01-27 11:24:58

Re: Problem migrating THttpApiServer -> THttpServer

#17 2018-01-27 14:56:47

Re: Problem migrating THttpApiServer -> THttpServer

#18 2018-01-28 09:28:51

Re: Problem migrating THttpApiServer -> THttpServer

#19 2018-01-28 10:04:23

Re: Problem migrating THttpApiServer -> THttpServer

#20 2018-01-28 11:10:38

Re: Problem migrating THttpApiServer -> THttpServer

#21 2018-01-28 14:50:03

Re: Problem migrating THttpApiServer -> THttpServer

#22 2018-01-28 15:25:20

Re: Problem migrating THttpApiServer -> THttpServer

#23 2018-02-23 14:03:08

Re: Problem migrating THttpApiServer -> THttpServer

#24 2018-02-23 14:18:32

Re: Problem migrating THttpApiServer -> THttpServer

#25 2018-02-23 14:44:57

Re: Problem migrating THttpApiServer -> THttpServer

#26 2018-02-23 16:10:20

Re: Problem migrating THttpApiServer -> THttpServer

#27 2018-06-19 07:43:04

Re: Problem migrating THttpApiServer -> THttpServer

#28 2018-06-25 14:42:49

Re: Problem migrating THttpApiServer -> THttpServer

#29 2018-07-06 09:22:27

Re: Problem migrating THttpApiServer -> THttpServer

#30 2018-08-28 13:08:39

Re: Problem migrating THttpApiServer -> THttpServer

#31 2018-10-05 09:59:06

Re: Problem migrating THttpApiServer -> THttpServer

#32 2018-10-05 11:03:06

Re: Problem migrating THttpApiServer -> THttpServer

#33 2018-10-05 12:01:33

Re: Problem migrating THttpApiServer -> THttpServer

#34 2018-10-06 12:24:28

Re: Problem migrating THttpApiServer -> THttpServer

#35 2018-10-07 17:40:43

Re: Problem migrating THttpApiServer -> THttpServer

#36 2018-10-08 10:15:12

Re: Problem migrating THttpApiServer -> THttpServer

#37 2018-10-08 15:07:06

Re: Problem migrating THttpApiServer -> THttpServer

#38 2018-12-05 18:22:03

Re: Problem migrating THttpApiServer -> THttpServer

#39 2018-12-06 19:23:35

Re: Problem migrating THttpApiServer -> THttpServer

#40 2018-12-08 15:28:33