mORMot Open Source

Smoke test on server shows +4% (+50k) RPS for /json for new HASEVENTFD. I'll check in more detail later.

In mORMot1 I upload such huge files by manually split it on chunks (in my case - in browser) and send chunks one-by-one to the server endpoint with chunk number in URL  htp://.../fileUpload?chunk=x&totalChunks=y
On the server side endpoint just append HTTP request body what contains chunk to the file.

The same can be used with mORMot2

@ttomas - thanks for investigation. At last now we know it didn't help. BTW I investigate jemalloc with the same result. So for now we can consider glibc mm is good enough

@dcoun - this round (started 2023-02-23) should be the same as previous, because our last MR with minor improvements is not merged yet.

@ab - I investigating a pipe (simple program with one writer, what write pointer into pipe and many readers concurrently read it) - performance depends very much on kernel version. For old kernel (4.18 as on my server) it's terrible. On never  (5.18 as on my desktop)  it`s faster a lot (x10 times). Still planning to try eventfd + ring buffer. Will post a gists when finish.

Thank you very match!

I will made a new PR to TFB based on today`s sources with all minor changes and clean-upped raw.pas (w\o enabling smoothing). This give us a clean picture for comparison of event vs eventfd in future.

P.S.
Ready #7944

all frameworks decrease in the same way after adding Sync() - we not first.
When PG server is local the decrease is not so significant.   

@ab - crazy idea - what if we use named pipes (mkfifo) to wake up a threads in pool instead of futex? Have you considered this option?
I mean TAsyncConnections creates a fifo pipe and write when data is ready, and all workers simply read from pipe in blocking mode? And let's kernel decide what worker wakes up by read successfully..
In this case we twice syscalls count (write + read instead of futex) but avoid loops over fThreads and simplify code a lot. Isn't it?

P.S.
Even simple pipe should work. And where is also a message queue if pipes is slow...

Results are ready - we still #16. Impressive plaintext improvements, rawqueries (with PG Sync) worse than I expected, small improvement for json and db (I think because of -O4)

Weights		1.000	1.737	21.745	4.077	68.363	0.163
Composire #	JSON	1-query	20-query Fortunes Updates Plaintext 	Weighted score
38 	mormot 	731,119	308,233	19,074	288,432	3,431	2,423,283 	3,486  2022-10-26 - 64 thread limitation
43 	mormot 	320,078	354,421	19,460	322,786	2,757	2,333,124 	3,243  2022-11-13 - 112 thread (28CPU*4)	
44 	mormot 	317,009	359,874	19,303	324,360	1,443	2,180,582 	3,138  2022-11-25 - 140 thread (28CPU*5) SQL pipelining
51 	mormot 	563,506	235,378	19,145	246,719	1,440	2,219,248 	2,854  2022-12-01 - 112 thread (28CPU*4) CPU affinity	
51 	mormot 	394,333	285,352	18,688	205,305	1,345	2,216,469 	2,586  2022-12-22 - 112 threads CPU affinity + pthread_mutex
34 	mormot 	859,539	376,786	18,542	349,999	1,434	2,611,307 	3,867  2023-01-10 - 168 threads (28 thread * 6 instances) no affinity
28 	mormot 	948,354	373,531	18,496	366,488	11,256	2,759,065 	4,712  2023-01-27 - 168 threads (28 thread * 6 instances) no hsoThreadSmooting, improved ORM batch updates
16 	mormot 	957,252	392,683	49,339	393,643	22,446	2,709,301 	6,293  2023-02-14 - 168 threads, cmem, inproved PG pipelining
15 	mormot 	963,953	394,036	33,366	393,209	18,353	6,973,762 	6,368  2023-02-21 - 168 threads, improved HTTP pipelining, PG pipelining uses Sync() as required,  -O4 optimization  

@ab - you are right. Replacing TOSLightLock into TLightLock dose not change futex calls count. So, this is thread awaking. Will play with WakeupLoad later...   

@ttomas - current round result for mormot /plaintext is ~7M - and this is 10Gb network limitation on TFB hw. I think we should focus on /json. If we speed up HTTP for json,  this automatically speed up db related  tests.

Problem is what db layer expect `?` is a parameter placeholder.
For ExecuteInlined use inlined parameters syntax  :(paramValue):

But to to prevent SQL injection and speed up excution better to execute query with parameters:

stmt := conn.NewStatementPrepared('insert into NEW_TABLE (ID, DESCRIPT) values (?, ?)', false, true)
stmt.Bind(1, 4);
stmt.Bind(2, 'I''m confused?');
stmt.ExecutePrepared;

Nice idea, I will investigate it!
Different thread counts for non-db test seams make sense. I'll try to determinate best and may be enable CPU affinity for them.
As I understand from python sources, DB container is restarted for each test(once for default and once for postgres-raw), and current test order is  db->json->queries->updates->fortunes->plaintext is OK (it's important what updates is after db/queries). So I do not see a reason to create separate containers for db and rawdb (but will check dose it make sense)

About "tuning the dockers" - we can't do this from inside a container. Docker daemon is configured on the host.

P.S.
My understanding of docker (simplified):
Docker is a slim wrapper around Linux namespaces and cgroups (+AuFS as a read only FS with layers). So everything is executed on host, but executable links to the libraries from docker image. No magic at all
BTW this is a reason we can't use io_uring - as far as i understand TFB host machine is based on Ubuntu 18 and io_uring not supported there. TFB have a planes to migrate it to 22.04 in next round.

Switching to io_uring is too radical a change, IMHO. And as I see, the top 10 frameworks use epool...

For my opinion our current problem is what with the same number of threads increasing listening sockets count improves json performance
I catch a case, where performance difference is dramatically on both server and on my PC (for other therads/sockets count numbers more sockets also always win, but the difference is not so big)

Server (28 cores)

wrk -d 15 -c 64 --timeout 8 -t 28 "http://localhost:8080/json" 

./raw 56 28 1 (1x56 thread) -  143K RPS
./raw 28 28 2 (2x28 thread) -  770K RPS

My PC (12 cores)

wrk -d 15 -c 64 --timeout 8 -t 12 "http://localhost:8080/json" 

./raw 24 12 1 - 300K
./raw 12 12 2 - 500K

perf (for server) shows what:
in case of one socket most of the load is gone into only 2 thread R18080 R28080 - see flamergath_json_56_1
in case of two sockets load if distributed between many threads - see flamergath_json_28_2

I do not understand the reason of such behavior

P.S.
I also tried
- switching the PG protocol to binary mode - performance is the same 
- sets HttpQueueLength to 0 - no changes in performance except small amount of sockets errors

Yes, unexpectedly..
I post an announcement for our work on freepascal forum. May be community generate more ideas for optimizations....

All tests passed. Changes a tiny, сlose to a statistical error. I will take them for the next PR...

@ab - one request for future mORMot2 release - please, compress static in mormot2 release using tar.gz. 7z is not available in default distribution (on both Linux and Windows), and installing p7zip package, for example inside docker build as we do in TFB bench, just to decompress  mormot2static.7z  takes some time, and worst what it require internet connection - I sometimes have problem with Internet after missile attacks

All TFB tests are passed
/plaintext results:
my PC: 1 271 496 -> 3 764 547
server:  2 621 195 -> 4 343 425 (near the same results should be on TFB hardware)

Perfect!

I need some time to solve PG Sync() and will prepare a PR

Unfortunately, as noted here  our PG pipeline optimization breaks General test requirements #7 rule:
   "If using PostgreSQL's extended query protocol, each query must be separated by a Sync message"

I do not understand why this requirement exists for synchronous case, but we have what we have. So I will rewrite PG pipelining again

About valgrid - I can't see a bottleneck there (mostly because it works slow), but now I trues to use perf + flamegraph - this techniques show more details  because of HI-frequency sampling.

What I fund for a moment is a way to optimize /plaintext in HTTP pipelining mode. Flamegraph for pipelined plaintext shows what most of the time we are in send syscall - see flamegrapj for ./raw 4 4 1 (clickable)
This is because in current HTTP pipelining implementation we do less recv syscalls (read several GET request in one syscall) but we can also buffer an output and do less send`s!

This can be done if HTTP state machine detect what for GET request with Connection: keep-alive and HTTP 1.1  where is some additional GET after `\r\n`
and when send`s buffered response either if send buffer is overflow (for example nodeJS buffers such responses in 64k buffer) or pipeline ends (no additional bytes read)

Test case for 2 pipelined HTTP query:

(echo -en "GET /plaintext HTTP/1.1\nHost: foo.com\nConnection: keep-alive\n\nGET /plaintext HTTP/1.1\nHost: foo.com\n\n"; sleep 10) | telnet localhost 8080

currently produce 2 sendto(7, "HTTP/1.1... syscall, but can produce one.

Since HTTP pipelining should not be used in production for many reasons, this optimization can be done under hsoHttpPipelining flag. @ab - can you implement this idea?

P.S. - to become clickable flamegraph should be downloaded first and when opened in browser - click do not work when svg is opened in Google Drive preview

mORMot uses global lock for SQLITE3 updates, so problem not happens where.

I use my own TubSQLite3ThreadSafeConnection = class(TSQLDBConnectionThreadSafe) without global lock (it actually exists inside sqlite) and sets

TSQLDataBase.WalMode := true;
TSQLDataBase.BusyTimeout := 10000; // 10 seconds

In case of sqlite3_busy  - this error is passed to user and on UI message "try again later" is displayed.

Using -O4 optimization level (never use it before because of "beware" notes) slightly increases performance  (+44k for json for example) and pass all tests.
Also tries Whole Program Optimization - it's decrease executable size from 5Mb to 3Mb but without visible performance changes (compared to -O4)

@ab - how do you think - can we use -O4 for TFB (I'm afraid of accidental falls)?

Memory usage statistic

//libc
Maximum resident set size (kbytes): 28896
Minor (reclaiming a frame) page faults: 12867
Voluntary context switches: 5888357
Involuntary context switches: 5049

//x64mm (NumTinyBlockArenasPO2 = 7)
Maximum resident set size (kbytes): 124380              
Minor (reclaiming a frame) page faults: 44196          
Voluntary context switches: 5220211                    
Involuntary context switches: 8087

With arenas, bind to threadID fortunes result is 313K RPS - very close to 355K with libc. Congratulations!

Flags: BOOSTER  assumulthrd smallpools perthrd erms                              
Small:  blocks=3K size=309KB (part of Medium arena)                              
Medium: 51MB/51MB  sleep=10K                                                     
Large:  0B/640KB  sleep=0                                                        
Total Sleep: count=10K                                                           
Small Getmem Sleep: count=16                                                     
288=14 80=2                                                                      
Small Blocks since beginning: 234M/28GB (as small=42/46 tiny=746/1008)           
48=89M  112=37M  80=27M  128=18M  32=14M  96=9M  64=9M  144=4M                   
160=4M  256=4M  416=3M  880=3M  1264=3M  272=2M  960=465K  1376=464K             
Small Blocks current: 3K/309KB                                                   
48=2K  64=426  352=200  32=87  128=80  112=73  80=48  96=21                      
192=14  416=8  576=7  880=7  288=6  736=5  160=4  672=4

P.S.
sleeps count is increased, but overall speed - also

BTW - glibc MM for x64 by default uses arenas count = CPUcores*8

By filtering valgrind profiling data using `mem` I found what the only valuable difference is  self timing of _getmem and _freemem
x64mm: 0.69 0.46
glibc    : 0.26  0.23

Last fpclibcmm changes tested - all OK.
Will prepare new TFB PR, hope they don't get sick of me..

Nice catch!

I still worry about valgrind memcheck (`valgrind -v` mode) - it produce too many warnings about access to uninitialized memor/conditional jump or move depends on uninitialised value.
The source of most of them is in TLecuyer and AESNI.
We discuss this problem some years ago - it's also exists in mORMot1.

Maybe this is not a problem from algorithm POW, but this is problem for valgrind DRD tool (I tries to use it to found thread safety problems, but can't do this because of too many memcheck errors). Hope we slowly fix such warnings. At last I will tries..

For comparison - results for 28 cores server what shows performance increasing. First 3 because of MM, raw* - mostly because of new PG pipelining. For other endpoints, what almost not allocate results near the same

			     x64mm	 libc
/fortunes                   181 000	361 000
/rawfortunes                367 000	424 000
/queries?queries=20          33 000	 35 000
-- raw perf increased because of new PG pipeline impl
/rawqueries?queries=20       6 000	 50 000
/rawupdates?queries=20       3 000	 26 000

if the server does not crash I expect mORMom can be in the top 10

Also catched the same pattern in libs/mORMot/src/net/mormot.net.sock.posix.inc:1165 (SelLength for dyn array what points to nil).
Crash is not stable - sometimes app crash, sometimes - not. Sometimes after 200k requests, sometimes after 400k

BTW - I uses only libc (SynFPCCMemAligned) with mORMot1 for many years without any problems. So I expect there is some error in mORMot2, not in FPC

No stack track yet. Reprodoced only on server hardware with core dump. Will the core dupmp helps?

P.S.
Sorry, my bad. After googling a little now I know how to get stack trace from core dump. Will get it in the morning