#1 mORMot Framework » Stop the war! » 2022-02-24 07:40:16

mpv
Replies: 16

Today I woke up at 4 am from rocket explosions. In Kiev. In the centre of Europe. Russia started a war against my country and the entire civilized world.
Do not believe a single word of Russian propaganda, All the words of their officials are lies.
Tell that to your friends and your goverment. Help us stop the war.
See you after our victory!

Pavlo, Kyiv, Ukraine

#2 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-14 21:30:00

mpv

Test results:
- Server stable in case clients are HTTP 1.1, 1.0 and mix of HTTP1.1. and 1.0.
- I can't reproduce a delay anymore

But
HTTP1.0 mode still too slow ~1864 RPS

The strange thing is what in HTTP 1.0 mode server consume only small part of CPU (each core is about ~10%), while in HTTP 1.1 - 100%

I analyse a syscalls  - everything is OK

Try to profile via valgrind - and got another strange result - if sample profiler is active all responses are non 200, in verbose mode server start reporting

 warn  mormot.net.async.THttpAsyncConnections(04b36040) Execute: Accept(8888) failed as Too Many Connections

what is not true, because ss -s shows 3595 socket in use

PS
test result above is done before Edit2 added in post above

Edit2
with latest commit result is the same

#3 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-14 15:57:18

mpv

Delay occurs only when proxy using HTTP 1.0, if proxied using HTTP 1.1 performance is ~70000 RPS.
Updated nginx config with 3 endpoint:
  - /echo proxy using http1.0 w\o keep-alive
- /echo11 - uses HTTP1.1 with keep alive (up to 1024 connection for each nginx worker)
- and (just to compare performance) /pure endpoint what respond 200 "gangnam style!' on nginx level

upstream mormot2 {
    server localhost:8888 max_fails=2 fail_timeout=30;
    keepalive 1024;
}

server {
    listen       82;
    server_name localhost;
    server_tokens off;

    location /echo {
       proxy_pass http://127.0.0.1:8888;
    }
    location /echo11 {
        proxy_pass http://mormot2;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
    location /pure {
        return 200 'gangnam style!';
    }
}

#4 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-14 15:41:32

mpv

Now it works, if I use wrk even not hangs,  but very slow (2000 RPS).

There is 60 second random delay, it can be reproduced if i send queries using curl into nginx by hands (Up Enter; Up Eneter in console) with command `curl -v  http://localhost:82/echo`

After 15 - 100 requests (randomly) server stop responding, think one minute and continue to respond - see time in line 4 (15:34) and 5 (15:35) below:

20220114 15340646  1 trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) UnlockSlotAndCloseConnection: Write() finished on handle=75
20220114 15340646  1 trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) Stop sock=54 handle=75 r=2 w=0
20220114 15340646  1 trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) OnClose {"THttpAsyncConnection(7f12a5219bc0)":{Handle:75}}
20220114 15340704  2 trace mormot.net.async.THttpAsyncConnections(7f12a5cfe040) ConnectionNew {"THttpAsyncConnection(7f12a5219d40)":{Handle:76}} socket=55 count=0
20220114 15350710  2 trace mormot.net.async.THttpAsyncConnections(7f12a5cfe040) ConnectionNew {"THttpAsyncConnection(7f12a5219ec0)":{Handle:77}} socket=56 count=0
20220114 15350710  & trace mormot.net.async.TPollAsyncReadSockets(7f12a5d8e240) GetOnePending(R1 mormotHttpServer)=7f12a5219d40 1 #1/2
20220114 15350710  & trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) ProcessRead recv(55)=Ok len=96 in 12us {"TPollAsyncReadSockets(7f12a5d8e240)":{PendingIndex:1,PendingCount:2}}
20220114 15350710  & trace mormot.net.async.THttpAsyncConnection(7f12a5219d40) OnRead hrsGetCommand len=96 GET /echo11 HTTP/1.1$0d$0aHost: mormot2$0d$0aConnection: close$0d$0aUser-Agent: curl/7.68.0$0d$0aAccept: */*$0d$0a$0d$0a
20220114 15350710  & trace mormot.net.async.THttpAsyncConnection(7f12a5219d40) Write len=151 HTTP/1.0 200 OK$0d$0aServer: mORMot2 (Linux)$0d$0aX-Powered-By: mORMot 2 synopse.info$0d$0aContent-Length: 31$0d$0aConnection: Close$0d$0a$0d$0agot request from connection #76
20220114 15350710  & trace mormot.net.async.THttpAsyncConnection(7f12a5219d40) AfterWrite Done ContentLength=31 Wr=0 Flags=pc
20220114 15350710  & trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) Write 55 closed by AfterWrite handle=76
20220114 15350710  & trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) UnlockSlotAndCloseConnection: Write() finished on handle=76
20220114 15350710  & trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) Stop sock=55 handle=76 r=2 w=0
20220114 15350710  & trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) OnClose {"THttpAsyncConnection(7f12a5219d40)":{Handle:76}}
20220114 15350710  & trace mormot.net.async.TPollAsyncReadSockets(7f12a5d8e240) GetOnePending(R1 mormotHttpServer)=7f12a5219ec0 1 #2/2
20220114 15350710  & trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) ProcessRead recv(56)=Ok len=96 in 9us {"TPollAsyncReadSockets(7f12a5d8e240)":{}}
20220114 15350710  & trace mormot.net.async.THttpAsyncConnection(7f12a5219ec0) OnRead hrsGetCommand len=96 GET /echo11 HTTP/1.1$0d$0aHost: mormot2$0d$0aConnection: close$0d$0aUser-Agent: curl/7.68.0$0d$0aAccept: */*$0d$0a$0d$0a
20220114 15350710  & trace mormot.net.async.THttpAsyncConnection(7f12a5219ec0) Write len=151 HTTP/1.0 200 OK$0d$0aServer: mORMot2 (Linux)$0d$0aX-Powered-By: mORMot 2 synopse.info$0d$0aContent-Length: 31$0d$0aConnection: Close$0d$0a$0d$0agot request from connection #77
20220114 15350710  & trace mormot.net.async.THttpAsyncConnection(7f12a5219ec0) AfterWrite Done ContentLength=31 Wr=0 Flags=pc
20220114 15350710  & trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) Write 56 closed by AfterWrite handle=77
20220114 15350710  & trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) UnlockSlotAndCloseConnection: Write() finished on handle=77
20220114 15350710  & trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) Stop sock=56 handle=77 r=2 w=0
20220114 15350710  & trace mormot.net.async.TAsyncConnectionsSockets(7f12a5fef340) OnClose {"THttpAsyncConnection(7f12a5219ec0)":{Handle:77}}

BTW trace is perfect thing!

#5 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-14 14:10:14

mpv

I sure the same handle is twice. I repeat tests - see 3e

20220114 14062029  ' trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) Write 3b closed by AfterWrite handle=50
20220114 14062029  ' trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) UnlockSlotAndCloseConnection: Write() finished on handle=50
20220114 14062029  ' trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) Stop sock=3b handle=50 r=2 w=0
20220114 14062029  ' trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) OnClose {"THttpAsyncConnection(7f81914dbb40)":{Handle:50}}
20220114 14062029  2 trace mormot.net.async.THttpAsyncConnections(7f8191fc1040) ConnectionNew {"THttpAsyncConnection(7f81914dbe40)":{Handle:52}} socket=3d count=0
20220114 14062029  2 trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) Start sock=3d handle=52
20220114 14062029  2 trace mormot.net.async.THttpAsyncConnections(7f8191fc1040) Execute: Accept(8888)={"THttpAsyncConnection(7f81914dbe40)":{Handle:52}}
20220114 14062029  - trace mormot.net.async.TPollAsyncReadSockets(7f8192051240) GetOnePending(R7 mormotHttpServer)=7f81914dbe40 1 #1/1
20220114 14062029  ( trace mormot.net.async.THttpAsyncConnection(7f81914dbcc0) Write len=151 HTTP/1.0 200 OK$0d$0aServer: mORMot2 (Linux)$0d$0aX-Powered-By: mORMot 2 synopse.info$0d$0aContent-Length: 31$0d$0aConnection: Close$0d$0a$0d$0agot request from connection #51
20220114 14062029  2 trace mormot.net.async.THttpAsyncConnections(7f8191fc1040) ConnectionNew {"THttpAsyncConnection(7f81914dbfc0)":{Handle:53}} socket=3e count=0
20220114 14062029  - trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) ProcessRead recv(3d)=Ok len=56 in 4us {"TPollAsyncReadSockets(7f8192051240)":{PendingCount:1}}
20220114 14062029  - trace mormot.net.async.THttpAsyncConnection(7f81914dbe40) OnRead hrsGetCommand len=56 GET /echo HTTP/1.0$0d$0aHost: mormot2$0d$0aConnection: close$0d$0a$0d$0a
20220114 14062029  ( trace mormot.net.async.THttpAsyncConnection(7f81914dbcc0) AfterWrite Done ContentLength=31 Wr=0 Flags=pc
20220114 14062029  * trace mormot.net.async.TPollAsyncReadSockets(7f8192051240) GetOnePending(R8 mormotHttpServer)=7f81914dbfc0 1 #1/1
20220114 14062029  * trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) ProcessRead recv(3e)=Ok len=56 in 4us {"TPollAsyncReadSockets(7f8192051240)":{}}
20220114 14062029  * trace mormot.net.async.THttpAsyncConnection(7f81914dbfc0) OnRead hrsGetCommand len=56 GET /echo HTTP/1.0$0d$0aHost: mormot2$0d$0aConnection: close$0d$0a$0d$0a
20220114 14062029  * trace mormot.net.async.THttpAsyncConnection(7f81914dbfc0) Write len=151 HTTP/1.0 200 OK$0d$0aServer: mORMot2 (Linux)$0d$0aX-Powered-By: mORMot 2 synopse.info$0d$0aContent-Length: 31$0d$0aConnection: Close$0d$0a$0d$0agot request from connection #53
20220114 14062029  * trace mormot.net.async.THttpAsyncConnection(7f81914dbfc0) AfterWrite Done ContentLength=31 Wr=0 Flags=pc
20220114 14062029  * trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) Write 3e closed by AfterWrite handle=53
20220114 14062029  * trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) UnlockSlotAndCloseConnection: Write() finished on handle=53
20220114 14062029  * trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) Stop sock=3e handle=53 r=2 w=0
20220114 14062029  * trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) OnClose {"THttpAsyncConnection(7f81914dbfc0)":{Handle:53}}
20220114 14062029  - trace mormot.net.async.THttpAsyncConnection(7f81914dbe40) Write len=151 HTTP/1.0 200 OK$0d$0aServer: mORMot2 (Linux)$0d$0aX-Powered-By: mORMot 2 synopse.info$0d$0aContent-Length: 31$0d$0aConnection: Close$0d$0a$0d$0agot request from connection #52
20220114 14062029  ( trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) Write 3c closed by AfterWrite handle=51
20220114 14062029  ( trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) UnlockSlotAndCloseConnection: Write() finished on handle=51
20220114 14062029  ( trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) Stop sock=3c handle=51 r=2 w=0
20220114 14062029  ( trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) OnClose {"THttpAsyncConnection(7f81914dbcc0)":{Handle:51}}
20220114 14062029  - trace mormot.net.async.THttpAsyncConnection(7f81914dbe40) AfterWrite Done ContentLength=31 Wr=0 Flags=pc
20220114 14062029  - trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) Write 3d closed by AfterWrite handle=52
20220114 14062029  - trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) UnlockSlotAndCloseConnection: Write() finished on handle=52
20220114 14062029  - trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) Stop sock=3d handle=52 r=2 w=0
20220114 14062029  - trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) OnClose {"THttpAsyncConnection(7f81914dbe40)":{Handle:52}}
20220114 14062029  2 trace mormot.net.async.TAsyncConnectionsSockets(7f81922b2340) Start sock=3e handle=53
20220114 14062029  2 trace mormot.net.async.THttpAsyncConnections(7f8191fc1040) Execute: Accept(8888)=20220114 14062029  2 warn  mormot.net.async.THttpAsyncConnections(7f8191fc1040) Execute raised uncatched EAccessViolation -> terminate mormotHttpServer
20220114 14062029  2 info  mormot.net.async.THttpAsyncConnections(7f8191fc1040) Execute: done AW mormotHttpServer
20220114 14062029  2 info  mormot.net.async.THttpAsyncConnections(7f8191fc1040) Execute: State=esFinished

you do not reproduce it?

#6 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-14 13:03:21

mpv

If verbose log is disabled problem exists. I think I understand the reason - see this lines

20220114 12173421  ( trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) Stop sock=2d handle=36 r=2 w=0
20220114 12173421  ( trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) OnClose {"THttpAsyncConnection(7f211a357640)":{Handle:36}}
20220114 12173421  2 trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) Start sock=2d handle=36

The same sock 2d used twice. This may happens because nginx aggressively reuse connections..

#7 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-14 12:23:32

mpv

Additional info:
I disable x64mm and now (if behind nginx) got AV very quickly (AV not logged into file, only on console)

0220114 12173421  ' trace mormot.net.async.THttpAsyncConnection(7f211a3574c0) OnRead hrsGetCommand len=56 GET /echo HTTP/1.0$0d$0aHost: mormot2$0d$0aConnection: close$0d$0a$0d$0a
20220114 12173421  ( trace mormot.net.async.THttpAsyncConnection(7f211a357640) AfterWrite Done ContentLength=31 Wr=0 Flags=pc
20220114 12173421  ( trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) Write 2d closed by AfterWrite handle=36
20220114 12173421  ( trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) UnlockSlotAndCloseConnection: Write() finished on handle=36
20220114 12173421  ( trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) Stop sock=2d handle=36 r=2 w=0
20220114 12173421  ( trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) OnClose {"THttpAsyncConnection(7f211a357640)":{Handle:36}}
20220114 12173421  2 trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) Start sock=2d handle=36
20220114 12173421  2 trace mormot.net.async.THttpAsyncConnections(7f211ae3e040) Execute: Accept(8888)=20220114 12173421  2 warn  mormot.net.async.THttpAsyncConnections(7f211ae3e040) Execute raised uncatched EAccessViolation -> terminate mormotHttpServer
20220114 12173421  2 info  mormot.net.async.THttpAsyncConnections(7f211ae3e040) Execute: done AW mormotHttpServer
20220114 12173421  2 info  mormot.net.async.THttpAsyncConnections(7f211ae3e040) Execute: State=esFinished
20220114 12173421  ' trace mormot.net.async.THttpAsyncConnection(7f211a3574c0) Write len=151 HTTP/1.0 200 OK$0d$0aServer: mORMot2 (Linux)$0d$0aX-Powered-By: mORMot 2 synopse.info$0d$0aContent-Length: 31$0d$0aConnection: Close$0d$0a$0d$0agot request from connection #36
20220114 12173421  ' trace mormot.net.async.THttpAsyncConnection(7f211a3574c0) AfterWrite Done ContentLength=31 Wr=0 Flags=pc
20220114 12173421  ' trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) Write 2c closed by AfterWrite handle=35
20220114 12173421  ' trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) UnlockSlotAndCloseConnection: Write() finished on handle=35
20220114 12173421  ' trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) Stop sock=2c handle=35 r=2 w=0
20220114 12173421  ' trace mormot.net.async.TAsyncConnectionsSockets(7f211b12f340) OnClose {"THttpAsyncConnection(7f211a3574c0)":{Handle:35}}

#8 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-14 10:59:44

mpv

Yes, you are right about UDS and my mORMot1 socket problems...

I can confirm enabling fpcx64mm improve performance - for test case above +12% RPS what is very good..

Tries to test HTTP1.0 mode (just enable nginx as a reverse proxy) and found in both 1.0 and 1.1   mORMOot return ~2000 responses and hangs/ Will continue to investigate later (evening) to give more info
nginx config is:

upstream mormot2 {
    server localhost:8888 max_fails=2 fail_timeout=30;
    keepalive 32;
}

server {
    listen       82;
    server_name localhost;
    # prevent nginx version exposing in Server header
    server_tokens off;
    # Enable gzip compression.
    # Default: off
    gzip on;
    location /echo {
      proxy_pass http://mormot2;
      # proxy_http_version 1.1;
    }
    location /pure {
      return 200 'gangnam style!';
    }
}

Test command:

wrk -c 100 -t 16  http://localhost:82/echo

#9 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-13 13:16:59

mpv

I will verify a HTTP 1.0 today evening, but it's a little hard to do - we quickly run out of sockets (whey are remains in "closed" state) - this is my main problem currently in mORMot1 without keep-alive.
Even with such OS tuning, behind NGINX (nginx is very efficiently use connections to back-end), moving all static no nginx etc. max number of actively working users for one IP address is up to 10000. So we need to run several servers on several IPs to handle more. Lucky me I have only one client with huge user base.

#10 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-13 13:08:15

mpv

Just pull and verify HTTP1.1 kep-alive scenario - now server is stable Even with 10000 concurrent connections smile

#11 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-13 08:54:26

mpv

With latest commit situation is the same sad
I still use wrk as a stress load tool - see "Load testing tool" thread for details, there is other tools @Eugene Ilyin notes in this post (Class D)

Problem can be reproduced for example on server with 16 thread using

~/dev/wrk/wrk -c 500 -t 8  http://localhost:8888/echo

After stress we can see (using `strace curl -v http://localhost:8888/echo`) what server accept request but don`t respond

And I got a stack trace in log!

EInvalidPointer {Message:"Invalid pointer operation"} [R0mormotHttpSvr] at 41a1f3 httpServerRaw.lpr  (82) 
../../src/net/mormot.net.sock.pas   mergependingevents (2265)
../../src/net/mormot.net.sock.pas   tpollsockets.pollforpendingevents (2464)
../../src/net/mormot.net.async.pas tpollasyncreadsockets.pollforpendingevents (949)
../../src/net/mormot.net.async.pas tasyncconnectionsthread.execute (1625)

#12 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-12 17:08:29

mpv

My test results (HTTP server with 4 thread)
- in idle mode CPU consumption is 0% - GOOD
- while requesting with 100 concurrent HTTP 1.1 connections max RPS is 274 176 - !impressive!
   For comparison on the same hardware for the same test we are:
      - 6x times faster compared to nodejs what gives 41 000 RPS  (in case mORMot thread pool size is 1 I still got 145659 RPS) 
      - 2x times faster compared to nginx!! what gives 104 642  RPS (4 working thread - the same as for our server)

Now the bad part sad
Implementation is not stable - for some conditions server hangs in case ServerThreadPoolCount > 1.
For ServerThreadPoolCount=4   100 (and even 500) concurrent connection newer hang the server, but 1000 does (after some amount of requests 2 thread consumes 1 core each after test ends, server continue to accept and even read request headers, but newer answer)
For ServerThreadPoolCount=16 500  concurrent connection always hangs (after some amount of requests server consume 0 CPU, accepts, read headers but newer answer)

Verified with -O3, the same with -O1 optimization flag

#14 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-08 09:52:42

mpv

Still 3% CPU for idle server with latest sources.
I profile using valgrind - most CPU time spends on internal fpc_initilize\fpc_finalize what wraps PollForPendingEvents function

  THTTPAsyncServer.Execute 
     if fAsync.fClients.fWrite.GetOne(1000, 'W', notif) then
       
         if fPending.Count = 0 then
            PollForPendingEvents({timeoutMS=}10); <-------- this call        

I remove all try\finally inside PollForPendingEvents but fpc_initialize\finalize still exists. Is this because of records in var block?

P.S.
  a raw server sources I use for testing

#15 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-07 18:44:10

mpv

This is syscalls statistic for 521053 HTTP keep alive request (10 second duration load with 1000 concurrent connections)

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 23.46   13.710255         331     41384     12693 futex
 21.29   12.438808        8421      1477         1 accept
 21.11   12.336652         858     14370           nanosleep
 17.78   10.389063          19    522047           sendto
 15.20    8.882895          17    522076           recvfrom
  1.05    0.612528         333      1835           epoll_wait
  0.04    0.024850           8      2952           epoll_ctl

~ 45% of time we are in futex and nanosleep syscalls

#16 Re: mORMot Framework » TCP (HTTP) server improvements for Linux » 2022-01-07 18:38:00

mpv

Thanks to Christmas I found some time for a mORMot2 (still not enough to migrate my product to it, but starts to discover a codebase more closely).
The main mORMot2 feature for me is an epoll (Linux) based HTTP1.1 (keep-alive) HTTP/WebSocket server with a fixed size workers thread pool.
So I starts from dummy THttpAsyncServer instance to see how it works - results are PERFECT I got ~ 213 000 RPS  for dummy server with 1000 concurrent keep-alive connections

But there is some problems:
- first one is about broken keep-alive -it is simple and fixed in #70
- second one  is about performance and it's more complex: THttpAsyncServer in idle mode (no requests) uses ~3% of host CPU. This is because of massive futex & nanosleep   syscalls:

$ sudo strace -cf -p `pidof httpServerRaw`

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 32.20    0.329206         120      2733      1366 futex
 29.48    0.301435          46      6492           nanosleep

The sources for it  is TPollSockets.GetOne - a futex is a critical section inside GetOnePending and nanosleep is inside SleepStep call.
We definitionally should do something with this.

Currently I can`t provide a good solution. Just some ideas:
  - timeouts can be implemented using timerfd_* together with epoll
  - lock in request queue can be avoided by using ring buffer

Will continue to investigate

#17 Re: mORMot Framework » Search engine » 2022-01-07 16:50:00

mpv
htits2008 wrote:

Hi,mpv!
Can you share a Complete example ?

Please, see real life project sources here

#18 Re: mORMot Framework » Calculate SHA256 hash of large file » 2021-12-29 21:59:40

mpv

There is an experimental utf8 support for file names in Windows 10 (checkbox on the language setting in control panel). If it is checked -  Windows should see file, so mORMot1 HashFile will work.

#19 Re: mORMot Framework » mormot.db.sql.odbc is not compiled with $define PUREMORMOT2 » 2021-12-01 10:06:09

mpv

I do not know in advance which drivers will be used by my users. In my product database connection is configured,  so I need to support any driver.

By "flag" you mean  adds an ifdef directive?

#21 Re: mORMot Framework » mormot.db.sql.odbc is not compiled with $define PUREMORMOT2 » 2021-11-29 14:27:48

mpv

@ab - do we really need to throw in case ODBC driver name is not in predefined list?
I propose to remove this throw - https://github.com/synopse/mORMot2/blob … #L350-L352  (in mORMot1 also)

P.S.
I tries to use MS Access from Linux and driver name for FOSS ODBC MS Access driver is `mdbtools`, since it not contains `access` I got Exception. But the same exception will be throw for any ODBC driver we can works with, but it name is not in hard-coded constant in mormot.db.sql.odbc

#22 Re: mORMot Framework » SynZip with UTF8 file names inside archive » 2021-11-17 11:16:44

mpv

It's strange, but when I sets madeBy flag to $0A14 then zipinfo -v fileName.zip shows OS as TOPS-20 (even don't know before what such OS exists), not an NTFS as in spec, but Unicode names works in this case

$ zipinfo -v utf8.zip 
....

Central directory entry #1:
---------------------------

  папка/новыйФайл.txt

  offset of local header from start of archive:   0
  file system or operating system of origin:      TOPS-20
  version of encoding software:                   2.0
  minimum file system compatibility required:     MS-DOS, OS/2 or NT FAT

From zipinfo sources https://github.com/LuaDist/unzip/blob/m … fo.c#L1003 for NTFS 11 ($0B) is used (the same zipinfo.c is in apple repository)

If I sets madeBy flag to $0B14 zipinfo show OS as NTFS but filename is broken

Central directory entry #1:
---------------------------

  ╨┐╨░╨┐╨║╨░/╨╜╨╛╨▓╤Л╨╣╨д╨░╨╣╨╗.txt

  offset of local header from start of archive:   0
  file system or operating system of origin:      NTFS
  version of encoding software:                   2.0
  minimum file system compatibility required:     MS-DOS, OS/2 or NT FAT

So I really don't know where is a source of truth.

P.S.
$03 - Unix works as expected for both OS info and file name encoding

#23 Re: mORMot Framework » SynZip with UTF8 file names inside archive » 2021-11-16 18:50:25

mpv

I attach 2 files to PR, both with cyrillic file names inside, one before fix and one - after fix

#24 mORMot Framework » SynZip with UTF8 file names inside archive » 2021-11-16 16:06:53

mpv
Replies: 5

current SynZip implementation always sets an "operating system of origin"
- in mORMot1 to 0 (DOS)
- in mORMot2 to ZIP_OS constant what depends on OS compilation target), but for windows target - 0

If "operating system of origin" is 0, even in case UFT8 flag is sets for file name, most of clients ( verified on Win with winrar, 7zip, explorer and on Linux with zipinfo tool) displays wrong file name (ignores UTF8 bit and display as ASCII)

I made a PR in mORMot1 - https://github.com/synopse/mORMot/pull/414 what force OS to Unix, in mORMot2 I propose to set ZIP_OS to 3 for Windows

#25 Re: mORMot Framework » Can TSqlRecord primary key be changed? » 2021-10-09 12:49:31

mpv

For such capacity Postgres IMHO is not a best candidate. Even with partition indexes become too huge to fit in memory.
We uses InfluxDB for last few years to store a time-series. With proper downsampling and data retention even on midle-level server it will fit your needs.
Also Influx uses very efficient data compression internally.
We use HTTP interface to put data into Influx (app server group data into portions and bulk insert using HTTP POST), and HTTP to retrieve a data.
Also We use Grafana to visualize a data.

#26 Re: mORMot Framework » Can TSqlRecord primary key be changed? » 2021-10-09 11:40:38

mpv

Postgres (and most or RDBMS) not allows several primary keys on table. The only way I see is to have primary key what includes partition column, and another non primary unique key for ID column as required by ORM.

@wxinix - what is your goal of using partitions? From my experience it needed very rarely,  mostly for historical data (partition by years/month) in case table contains > 100 millions of rows. For data isolation Postgre RLS can be used, and it transparent for ORM

#28 Re: mORMot Framework » Send and Receive messages via WebSocket in mORMot2 » 2021-09-30 19:05:45

mpv

There is a samples in https://github.com/synopse/mORMot/tree/ … WebSockets
They for mORMot1, but the idea is the same in mORMot2.
Hope it's help

#29 Re: mORMot Framework » Disable insecure SSL3 protocol? » 2021-09-29 17:46:42

mpv

An actual documentation about windows registry for TLS settings - https://docs.microsoft.com/en-us/window … n-settings

BTW TLS1.1 supported only on Windows11 - see comparison table here

#30 Re: SyNode » Problems with TSMEngineManager inside a DLL module » 2021-09-28 20:03:20

mpv

Im still on mORMot1 and plan to migrate to mORMot2 in next year.

#31 Re: mORMot Framework » Disable insecure SSL3 protocol? » 2021-09-28 08:59:35

mpv

I confirm - for http.sys level the only way is editing a registry systemwide (also this is a way to enable TLS1.2/1/3 on win10).
+++ for @ab remark about nginx + optional Let's Encrypt

#32 Re: mORMot Framework » TCurlHTTP » 2021-09-28 08:55:08

mpv

I can confirm - curl timeouts works as expected - verified on production

#33 Re: mORMot Framework » Potential bug? mormot.db.sql.postgres does not allow to set Port » 2021-09-23 17:38:09

mpv

Ah, also missed in mORMot1. I miss it there mostly because I always use a DatabaseName (it can accept a full Postgres URI ) and keep a serverName empty.

#34 Re: mORMot Framework » Multi-threaded BatchAdd performance » 2021-09-23 17:26:34

mpv
wxinix wrote:

But,  PostgreSQL layer DROPS the single-threaded insertion rate from Zeo's 100,000 record/sec to about 50,000 records/sec

I think this depends on data you insert. ZEOS interally uses binary pg protocol, so if inserted data is mostly numbers it can be faster compared to mORMot Postgres layer, where text pg protocol is used

#35 Re: mORMot Framework » mORMot 2 does not compile » 2021-09-22 15:16:33

mpv

Git itself is not designed for binary files. We try to use a git-lfs extension, but github.com require payments in case git-lfs is used.
So yes, binaries should be downloaded either from synopse.info or from releases page on gihtub - https://github.com/synopse/mORMot2/releases

#36 Re: mORMot Framework » mORMot2 fpc linker error » 2021-09-15 09:59:56

mpv
pvn0 wrote:

Another thing, since fpc source moved to gitlab, fpcupdeluxe wont accept the svn revision number r45643 because git hashes are now the "revision number", this makes documentation for installing on Linux obsolete.

IMHO under Linux the easiest way is to install from packages. May be we shod use this in documentation :

Debian (tested on Ubuntu 20.04):

mkdir -p ~/dist
cd ~/dist
wget -O fpc-laz_3.2.0-1_amd64.deb https://sourceforge.net/projects/lazarus/files/Lazarus%20Linux%20amd64%20DEB/Lazarus%202.0.10/fpc-laz_3.2.0-1_amd64.deb/download
wget -O fpc-src_3.2.0-1_amd64.deb https://sourceforge.net/projects/lazarus/files/Lazarus%20Linux%20amd64%20DEB/Lazarus%202.0.10/fpc-src_3.2.0-1_amd64.deb/download
sudo apt install ./fpc-laz_3.2.0-1_amd64.deb ./fpc-src_3.2.0-1_amd64.deb -y

RPM (tested on OracleLinux8)

wget -O fpc-3.2.0-1.x86_64.rpm https://sourceforge.net/projects/lazarus/files/Lazarus%20Linux%20x86_64%20RPM/Lazarus%202.0.10/fpc-3.2.0-1.x86_64.rpm/download
wget -O fpc-src-3.2.0-1.x86_64.rpm https://sourceforge.net/projects/lazarus/files/Lazarus%20Linux%20x86_64%20RPM/Lazarus%202.0.10/fpc-src-3.2.0-1.x86_64.rpm/download
sudo dnf install ./fpc-3.2.0-1.x86_64.rpm ./fpc-src-3.2.0-1.x86_64.rpm -y

#37 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-09-15 08:45:46

mpv

Now problem is solved - thank you very much!
I found my CI server uses XeonE5-2640 v2 CPU which do not support AVX2 (E5v2 - only AVX, E5v3 - AVX2) - this is the reason why I caught this error so late.

#38 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-09-14 18:06:51

mpv

Unfortunately the same behavior - string become empty after call to IsValidUTF8 (with -O1 string is OK)

#39 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-09-14 15:04:04

mpv

I found the strange behavior of new ASM code - reproduced only under Windows x64 and only if compiled with -O2 and higher (FPC3.2.0)
After call to IsValidUTF8 in scenarios like

function test(const aStr: RawByteString);
begin
  if not IsValidUTF8(pointer(aStr), length(aStr)) then
   ..
end;

aStr become nil.
Not reproduced under Linux x64 with any optimization level (we already use it on prod under Linux).
@ab - may be you have some ideas why this may happens, because I cant understand yet..

P.S.
In real life string become nil after this line - https://github.com/synopse/mORMot/blob/ … te.pas#L42

#40 Re: mORMot Framework » TCurlHTTP » 2021-09-13 10:58:44

mpv

I found TCurlHTTP ignores send and receive timeouts - this cause my server to wait forever in case resource I call has a bugs (a real life situation).
I create a patch what sets CURLOPT_TIMEOUT to SendTimeout + ReceiveTimeout - this is better than hangs in any case - see https://github.com/synopse/mORMot/pull/404
Also I add usage of CURLOPT_TIMEOUT_MS (available since libcurl 7.16.2 - April 11 2007, so exists in all systems)
@ab - I think it will be good to do the same for mORMot2

#41 Re: mORMot Framework » Is it possible doing crud via curl/common httpclient on mormot orm? » 2021-09-07 18:57:17

mpv

As far as I understand,  mORMot "extended" format is compartible with json5, and json5 parsers exists in JS and Dart...

#42 Re: mORMot Framework » Support for http 2.0 » 2021-08-30 21:32:10

mpv

IMHO its very hard to implement HTTP/2 correctly. I remember many issues with HTTP/2 even in ngnx. But last 2 years it has been stable and, as @ab recommends, we use nginx as http2 reverse proxy for all our productions (some of them are very big).
And we cautiously start looking at http3 (QUIC)

#43 Re: Low level and performance » How to unzip or zip files content » 2021-08-13 08:47:46

mpv

Another small fix - https://github.com/synopse/mORMot/pull/401 for TZipWriteAbstract.AddFromZip - we must use an original file name instead of one, where we replace '/' -> '\' (see pull request description)

#44 Re: mORMot Framework » Json Escape characters \u » 2021-08-05 18:38:55

mpv

In UTF8 encoding you do not need to escape cyrilic characters...

#45 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-31 14:13:35

mpv

OK, than I merge a #400 into SyNodeCleanup brunch, which i am using to build UnityBase and deploy version on the my testing environment to confirm everything works as expected. Currently my autotests are passed, but on Monday testing team starts works with real use cases and we ensure everything is OK.  If so, I will ask you to do the necessary changes SynCommons -> SynTable.

#47 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-30 17:31:20

mpv

Great work!! Many thanks!

A adopt a new IsValidUTF8 funtcion for mORMot1 (I'm still on mORMot1) - https://github.com/synopse/mORMot/pull/400

#48 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-25 19:08:35

mpv

To be clear: i'm fiine with mormot JSON parser, it is perfect and fits all my needs in terms of memory and performance. The only function I use from simdjson is validate_utf8 as a replacement of IsUTF8Valid.

#49 Re: SyNode » Problems with TSMEngineManager inside a DLL module » 2021-07-25 19:01:59

mpv

If this is cros country restriction, I think this is is not from my side. unitybase.info is hosted on our private cloud and we do not add any restriction there.
Please, try a google drive - https://drive.google.com/drive/folders/ … sp=sharing

Board footer

Powered by FluxBB