mORMot Open Source

Already verified. FPC RTL mm is much slover compared to x64mm.

I can't reproduce libc problems in TFB bench on my PC, where I can debug, but application crash is also easy reproducible  in `mormot2tests` - when I put CMem as a first unit

uses
  CMem,
  {$I ..\src\mormot.uses.inc} // may include mormot.core.fpcx64mm.pas

and compile without any defines

@ab - do we need to do something specific like in mORMot1 SynFPCCMemAligned instead of using CMem?

BTW - names for TDynArraySortCompare SortDynArray* functions family (SortDynArrayInt64, ....) are very confusing (DynArray part in name).  I understand such naming reason - it's because of FPC TListSortCompare naming, but in fact these functions not related to DynArray at all and can compare not only DynArray elements, and not only for sorting purpose, right?

May be (before mORMot2 is not released) rename all of them to Compare* (CompareInt64, ...) and family itself to TCompare. Or TfnCompare ->  fnCompareInt64, fnCompareQWord etc to prevent possible conflicts.

I do not measure a valuable performance difference between ORM with ID and without ID. But in case of ORM without ID we break the rules. When we get to the top, someone will look at the source and say that this is not a "fair play".

I propose to rollback ORM also. In fact for Postgres having primary key in select fields affects only serialization and a tiny amount of traffic - PK value is always in buffer cache...

Tested improved TOrmCache.Retrieve - it gives a small but measurement improvement about + 3000 RPS for /db performance (~ +3000 RPS).

If you don't mind, I prefer to wait for the result of the next round (without Smooting) to decide whether we need Smooting or not before making a new PR.
And during this time, I might find a way to reproduce the cached-queries problem....

Solved TFB /rawqueries and /rawupdates performance problems:
- PostgreSQL pipeline mode has been rethought - better to use Flush after each statement instead of PipelineSync after last statement; In this case server starts to execute queries ASAP;
- use ::bigint[] instead of ::NUMERIC[] typecast in /rawupdates  - NUMERIC is a floating point type, but we need Int64; added order by id to minimize locks waits (as in ORM)

See mORMot PR #140
Now I expect /rawupdates and /rawqueries to be in top 10.At last on my server hardware results are:
- 52000 RPS for /rawqueries?queries=20
- 26000 RPS for /rawupdates?queries=20
 
Will test once more after @ab merge PR#140 and prepare a merge request to TFB.....

Also a small performance tip to be fixed: TRestOrm.Retrieve call Model.GetTableIndexExisting twice - here inside Retrieve and second time - inside fCache.Retrieve
@ab - may be you add optional 3-d parameter tblIdx to Cache.Retrieve ?

Yes, updated HTTP headers parser helps for a few % for both /json and /plaintext. And with this changes where is no reason to verify disabled hsoHeadersInterning. I switch TFB PR 7860  to this mORMot version (it not merged yet, so waiting for a next round).
In PR 7860 I also remove hsoThreadSmooting - seems that it improves only plaintext, but bad for other endpoints.
I still investigating bad pipelining performance - no results yet, but at last I reproduce it on server hardware (not reproduced on my PC)

I tests new feature by assigning connection to thread CustomObject and there is almost no difference with threadvar version (at last on thread pool size = 28). So - you are right - better to keep ThreadSafeConnection  for auto-deleteion\reconnectiong etc.

Please, look at https://github.com/synopse/mORMot2/pull/135/files. It you think what removing lock is unsafe we can at last use ReadLock in function body and WriteLock in TryPrepare sub-function.

I detect a bottleneck - it's in ThreadSafeConnection implementation.
Current implementation ( on hi load) is looping almost every time to found connection for thread ID, Ideal implementation should  use array of fixed size (equal to HTTP server threads count) and get connection by thread index.

I have almost finished prototyping and got at last 10% in /db performance, but @#$%^&* russians  turned off electricity by their @#$%^&* missiles and I lost all modifications.

@ab, what do you think about fixed thread pool for DB connections?
Filled by nulls on start and indexSafeConnection(threadIndex) instead of ThreadSafeConnection?

As far as I understand every new round is starts from trunk, so will include PR with the new thread layout.
While we wait for results of new round I will try to solve the problem with pipelining mode

Looks like I fixed the update performance.
The reason is what we update table in random order, and simulations updates are lock each other (achieved after reading an epic book form postgrespro team https://postgrespro.ru/education/books/internals, unfortunately available only on russian).
So I add order by ID (Alternatively, we can sort by ID at the application server level, but at the database level it is easier) - see PR134

I setup and update test on 48core server - performance (with 512 concurrent connections and /updates?queries=20)  increased from ~4k RPS to 16k RPS

@ab, most of the open sources maintainers are not as quick as you (thanks for your perfect support!)
So yes, will wait for a next round...

The core of cloud provider business it to sell more resources they have.
For servers with 32 physical cores they usually sells at last 32*4 vCPU. We never know what tasks are running in the VM on the same host where our VM is located, and test results will vary over a wide range.
This is why I prefer dedicated servers.
But you can try.

Such king of bugs often occurs when  variable on stack is of unexpected size (see for example this topic - https://synopse.info/forum/viewtopic.ph … 798#p28798 and fix - https://github.com/synopse/mORMot/pull/299/files)
If so, it can disappear while compiling in debug/release/different optimization level or just changing a unit order. If so, it is a symptom of "dangling pointers" or incorrect var size.

In hsoThreadSmooting but WITHOUT hsoThreadCpuAffinity results are a little better

/json                     1 175 000
/db                         452 000
/fortunes                   181 000
/cached-queries?count=20    612 000
/queries?queries=20          33 000
/plaintext                1 335 000
/plaintext                2 900 000 (pipelining 1024 concurrent conns)
/plaintext                2 761 000 (pipelining 16384 concurrent conns)

so I remove "if servers = 1 then " to always enable it and made an MR to TFB

Added ability to set used cores and server instances count - see MR132
Will try to setup database tomorrow to verify /db and other endpoints
Happy New Year!

with the new pool of asynchronous threads the results are worse than with the previous one(in all cases)

~280k for 64 thread and 1 server (vs 350 with old algo),   720K in 24thread*6 server mode (vs 1050).

CPU load is strange - see this picture for 6 server mode - https://drive.google.com/file/d/1jPiYQI … share_link

@ab, please, add an option hsoReusePort for THttpAsyncServer and in function NewSocket in line 1733 before bind

      // Server-side binding/listening of the socket to the address:port
      v := 1;
      sock.SetOpt(SOL_SOCKET, SO_REUSEPORT, @v, SizeOf(v));
      if (bind(sock.Socket, @addr, addr.Size) <> NO_ERROR)

An experiment: I add a SO_REUSEPORT to the listening socket of async server - now it's possible to run several instance of server on the same port and kernel redistribute incoming connections between them
And YES!!! for /plaintext (instances are limited by first 24 CPU cores) with 256 concurrent connections
- 1 server   with 192 thread = 564k RPS
- 2 servers with   96 thread = 750k PRS  - faster than one
- 3 servers with   64 thread = 790k RPS  - the same total number of working threads, but redistributed on 3 thread pool
- 4 servers with   48 thread = 920k RPS  (~840k for /json)

Here is a picture for 4 servers - https://drive.google.com/file/d/1e3Qgwe … share_link

For 4 server mode and 512 concurrent connections /json is ~940k, so problem not in per process epool size.
It's definitionally somewhere is Async server queue implementation.

I shure this is not syscalls limitation, because on the same hardware nodejs (in cluster mode, limited to 16 cores) shows better results for json (500k RPS). And wrk remains on rhe same load.
Node create listening socket per fork using SO_REUSEPORT https://lwn.net/Articles/542629/. May be this is a way to go?

PS
Will try CMem after blackout, but I think memory manager is not a bottleneck

TFB round ends, but CPU statistic is empty
Anyway I found hardware similar to TFB (2CPU 24 cores each) and reproduce a problem with /json
 
I remove a hsoThreadCpuAffinity option and bind server to first  16 cores and wrk to last 16 cores using taskset

taskset  0xFFFF00000000 wrk...
taskset  0xFFFF raw

Two screenshots below (one for "raw 32" and one for "raw 64") shows what our server do not consume a full CPU power. RPS is ~200k for 32 thread vs ~370k for 64 thread. But CPU consumption is near the same.
Problem is:
not in memory manager - I tries all possible combinations ( default MM, FPC_X64MM, FPCMM_SERVER)
not in pthread/spin lock - I tries with current and old mORMot2

It seems that the gap is in "waits" inside workers...

Default MM 32 thread - https://drive.google.com/file/d/1JMBYxp … share_link
Default MM 64 thread - https://drive.google.com/file/d/1df9njW … share_link

P.S.
Setting fHttpServer.HttpQueueLength := 0; does not change performance, only small amount ~300 of read errors occurs in wrk

P.S.2
Increasing the number of available cores

taskset  0xFFFFFF raw 64

decrease an RPS (~10%)

P.S.3
binding of thread affinity to first 16 cores using hack

SystemInfo.dwNumberOfProcessors := 16;
// + enabling hsoThreadCpuAffinity

does not change anything (a small perf incr)

P.S.5
Using poll instead of epoll - near the same results (for 256 connections)

MR is ready. Unfortunately last two Citrine rounds are stuck - see this issue. Reason is "power problem". Hope their servers is not in Ukraine

I will test it first on my evn. Hope today evening