High-performance frameworks

jaclas · 2020-08-11 14:40:54

Hi Arnaud,

a small challenge for the mORMot ;-)

https://www.techempower.com/benchmarks/ … ph&test=db

and:

https://github.com/TechEmpower/Framewor … s-Overview

quote:

"We invite fans of frameworks, and especially authors or maintainers of frameworks, to join us in expanding the coverage of this project by implementing tests and contributing to the GitHub repository. The following are specifications for each of the test types we have included to-date in this project. Do not be alarmed; the specifications read quite verbose, but that's because they are specifications. The implementations tend to be quite easy in practice."

Last edited by jaclas (2020-08-11 14:41:30)

ab · 2020-08-11 15:27:28

Yes, we know these benchmarks.

A quick search: https://synopse.info/forum/search.php?s … =395664325

Check e.g. https://synopse.info/forum/viewtopic.php?id=3677
In a nutshell, pavel (aka mpv) wanted to include a mORMot sample since years.
I don't know what is the status on his side.

mpv · 2020-08-17 10:48:37

We did some optimizations in mORMot especially to participate in TechEmpower benchmark, for example topic about hi-performace Postgres access and HTTP server optimizations

For a long time we can't participate because we can't compile a mORMot in cloud environment. Now that FPC 3.2 is released it is possible.

Currently the main problem is one thread per Keep-Alive connection in THTTPServer implementation. TechEmpower benchmark scenarios executed using up to 512 concurrent keep-alive connections and 512 thread is too much. 512 Postgres connections is also too much. We need a separated "workers pool" of a fixed size (as in HTTP.SYS based server) and epool based reactor for http keep-alive. Hope @ab implements it in mORMot2

Another issue is a luck of Date HTTP response header (#5 here) but it's easy to solve.

Last edited by mpv (2020-08-17 10:50:52)

ab · 2022-06-16 08:47:39

This may be time to make this old thread awake again!

I included most needed features in mORMot 2.
Especially:
- HTTP/1.1 async/epoll server - so that we can use e.g. 8 threads and maintain only 8 connections to Postgres;
- Enhanced FPC + Linux x86_64 support;
- Date HTTP header as part of the server option.

My guess is that we could have some nice numbers to show.

mpv · 2022-07-18 08:10:16

I start implementing Techempower tests - a "raw" (without using ORM) approach, but ORM based implementation can be added later.
Infrastructural questions like Techempower development environment and docker based build is solved.

I will finish implementation today and publish for review.... (and numbers is GOOD!)

mpv · 2022-07-18 11:39:06

Initial implementation is pushed as Pascal/mormot, so will appears on TFB site in Pascal language category with framwork = mormot.

@ab - all endpoint except /fortunes pass TFB tests.
I'm not works with variants in mORMot before, please, look here - how to implement adding strings into variant array correctly?

May be someone have some other suggestions - fill free to post it here or do a PR

Last edited by mpv (2022-07-18 11:39:24)

Chaa · 2022-07-18 12:10:23

Something like this:

var
  doc: variant;
begin
  doc := _Arr([]);
  // or TDocVariantData(doc).Init(mDefault, dvArray)

  with _Safe(doc)^ do
  begin
    // add new item
    AddItem(MESSAGE_FORTUNES);
  end;

  // or 
  P := _Safe(doc);
  P^.AddItem(MESSAGE_FORTUNES);
end;

ab · 2022-07-18 13:35:49

Great work!
Do you have any numbers to share?

Perhaps we could publish both RAW (i.e. direct DB) and ORM (using mORMot ORM) for both /db and /fortunes endpoints.
I have seen other frameworks do this.

I would not use TDocVariant anywhere, because they are not the ORM.

I will propose something with the ORM later.

mpv · 2022-07-18 14:39:33

Implemented Fortunes as proposed by @Chaa.
Problem now with UTF8 compare - do we have any true Utf8 comparison function in mORMot2? Benchmark requires Japanise `フレームワークのベンチマーク` to be > 'fortune' but default SortDynArrayAnsiString sort returns <.
I need to use it here

Complete test over all frameworks runs 5 days, so for far I measure a `queries` endpoint over `jist` framework (a fastest in round 20)
Some results so far on my hardware:

8 threads and 8 connections for http://tfb-server:8080/query?q=2
44022 RPS in just-js vs 38783 RPS in mORMot

12 threads and 512 connections for http://tfb-server:8080/query?q=15
16244 RPS in just-js vs 6093 RPS in mORMot

Last edited by mpv (2022-07-18 15:16:48)

mpv · 2022-07-18 15:16:16

Seems my hardware is similar to TFB "Cloud" hardware. At last according to Round 20 Cloud results just-js results is the same as on my desktop, so mORMot with 6093 RPS will be #5

ab · 2022-07-18 16:58:09

There was a problem with SortDynArrayRawByteString Intel assembly.
Should be fixed https://github.com/synopse/mORMot2/commit/5f87b8b8

In the meantime, we could use SortCmp().

Please try https://gist.github.com/synopse/eb38cde … f6df14504f
- use the ORM for /db and /fortunes
- kept the direct DB access to /dbraw
- an optional SQLite3 in-memory storage to validate without PostgreSQL
- some minor fixes/changes about performance (more threads e.g.) and memory leaks

ab · 2022-07-18 20:13:28

I have made some changes.
Now you have /queries and /queriesraw - the first using the ORM.
Please try https://gist.github.com/synopse/f4b2c2e … 8aadff6c88

[Edit]
My guess is that raw performance would be similar than the ORM with PostgreSQL, since the ORM perform each external request without lock.
So I have disabled USE_RAW for the benchmark by default.
Also fixed the fact that the fortune table was not found as external, and reduce the threads to 64 max, since PostgreSQL allow up to 100 connections anyway.

mpv · 2022-07-19 07:41:52

@ab, it's not allowed mix raw and ORM approach in one test suite.
Each test suite (what includes all test cases) is described in benchmark_config.json should clearly indicate orm type: `full` in case we use mORMot ORM feature or raw in case of direct database access.
We can either create a separate programs+docker files or use a different endpoint names. I think I will use single program but different endpoints

I experiments today with thread count. I think thread count = CPUCount * 4 is too much. At last on real server. TFB hardware uses CPU with 28 core = 112 thread = 112 postgres connections = it's too mush IMHO

P.S.
miss your [Edit] section while wrote this post....

Last edited by mpv (2022-07-19 07:43:20)

ab · 2022-07-19 08:14:20

I have slightly updated the gist.
https://gist.github.com/synopse/f4b2c2e … 8aadff6c88

And fixed some SQLite3 related issues in the framework: the external DB was not thread-safe with SQLite3, which shares a single instance for all threads.
This won't affect the benchmark anyway.

On my PC, I have good numbers and stability, with low memory consumption. There are memory contention only in plan/json mode, as expected, but in real world process like /db or /queries, the MM is lock-free.
The raw versions are a little better, but not by a huge margin, just a few percents. So I guess we would prefer the ORM mode by default,

ab · 2022-07-19 08:40:14

I have included /cached_queries?count=x

Please try with the latest https://gist.github.com/synopse/f4b2c2e … 8aadff6c88

Using the ORM cache was pretty easy, and we could reuse the same code as with /queries?queries=x but just change the ORM class.
Numbers are very good.

ab · 2022-07-19 15:42:23

I just made a new mORMOt 2 release.
https://github.com/synopse/mORMot2/rele … g/2.0.3700

This includes latest optimizations and fixes for the benchmark.

I have also update the benchmark, especially to fix fModel early release, and enable hsoHeadersInterning new option (which reduces memory contention for /plaintext and /json very short requests).
https://gist.github.com/synopse/f4b2c2e … 8aadff6c88

mpv · 2022-07-19 18:35:02

I add an ORM tests and improvements you propose - thanks

Also combine both raw and ORM test suite and setup different URLs in benchmark_config.json

Also raw fortunes is implemented

Unfortunately ORM tests not work at all with Postgres DB - it even do not tries to connect to Postgres and SQL is generated using RowID instead of ID - something wrong with Virtual Tables ?
I add a small tutorial how to setup TFB Postgre database using Docker - it takes just a 2 minutes.

@ab - please, check what`s wrong with virtual tables ..

Last edited by mpv (2022-07-19 18:35:45)

ab · 2022-07-19 19:32:48

There is a missing

  fStore.Server.Cache.SetCache(TOrmCachedWorld);
  {$ifdef USE_SQLITE3}
  GenerateDB;
  {$else}
  fStore.Server.CreateMissingTables; // create SQlite3 virtual tables
  {$endif USE_SQLITE3}

mpv · 2022-07-19 19:54:19

Nice! Now it works.
I tries to implement `/updates` using ORM - good chance to study mORMot2 ORM and may be investigate best thread pool size for TFB bare metal

mpv · 2022-07-19 20:28:18

Not all is OK - CreateMissingTables also creates table `cachedworld` in the Postgres.
How to made a TOrmCachedWorld = class(TOrmWorld) to use the same table as TOrmWorld ?

Last edited by mpv (2022-07-19 20:28:40)

ab · 2022-07-19 20:34:03

https://github.com/TechEmpower/Framewor … ew#caching
"The request handler must retrieve a set of CachedWorld objects, equal in count to the count parameter, from the CachedWorld database table. Note that the CachedWorld database table uses a schema idential to the World table used for Test #3 but is distinct to avoid confusion."

To let TOrmCachedWorld map the same table but with another class, we can write

  fModel := TOrmModel.Create([TOrmWorld, TOrmFortune, TOrmCachedWorld]);
  VirtualTableExternalRegister(fModel, [TOrmWorld, TOrmFortune], fDbPool);
  VirtualTableExternalRegister(fModel, TOrmCachedWorld, fDbPool, 'world');
  fStore := TRestServerDB.Create(fModel, SQLITE_MEMORY_DATABASE_NAME);

mpv · 2022-07-19 21:00:08

Yes, you are right - table should exists. But this is script they use to create tables and there is no CachedWorlds there.... Strange - will see how others do.

BTW 2000 connections are allowed for test instance

Last edited by mpv (2022-07-19 21:01:11)

ab · 2022-07-19 21:06:30

Latest version available at
https://gist.github.com/synopse/f4b2c2e … 8aadff6c88

It fixes the CreateMissingTables and also pre-load all cached entries into memory.

mpv · 2022-07-19 21:06:35

All existed implementations use a `world` table - I think we also should.

ab · 2022-07-19 21:14:27

Half backed specs.

What worry me is that there was no recent round to the benchmark....

mpv · 2022-07-19 21:32:30

Your changes merged - see latest edition, now all tests (except not implemented ORM update) are passed.
Will implement it tomorrow..

Do not warry:
Framework Benchmarks Round 20 February 8, 2021
Framework Benchmarks Round 19 May 28, 2020

mpv · 2022-07-19 21:35:33

Hope our PR will be accepted before Round 21 begin..
Ok, let's hurry - please, give me a tips (or implement, please) an ORM updates and I will create a PR.

mpv · 2022-07-20 07:25:21

Intermediate test results (for master brunch?) can be found here - https://tfb-status.techempower.com/
I implement update using GenerateDB as example but have some internal error in TFB tests - investigating...

mpv · 2022-07-20 09:04:21

Final version with all endpoints implemented in ORM + RAW way (caching only for ORM) is ready
I keep thread count as CPUCount * 4 - on my desktop Rizen5 it shows the best result.

For select queries ORM is ~5% slows vs raw db access ( ~100000 RPS vs 95000 RPS for /db)
For updates our ::unnest hack speed up a raw implementation twice ( ~ 2200 RPS for /rawupdates?queries=20 vs 1400 RPS for /updates?queries=20 ) - may be this is a way for improving Postgres Batch update mode in ORM.

Any feedback is welcome. I plane to squash changes and push MR today evening.

ab · 2022-07-20 09:19:59

Round 21 was... yesterday....
But they didn't accept any new framework since may.
Let's hope they will make more often rounds, as they wrote.

Please find the updated source code:
https://gist.github.com/synopse/f4b2c2e … 8aadff6c88
- defined TFortune as record, not class
- use TDynArray for handling lists everywhere
- use data.InitArrayFrom(arr) instead of two steps with json
- implemented updates in ORM (I don't think ForUpdate=false is needed in our case)

As I can see, the ORM update already uses the ::unnest hack and from my tests is actually faster than rawupdates on my PC.

abouchez@tisab:~/dev/lib2/res/static/crypto$ wrk -c 512 -d 5s -t 4 http://localhost:8080/updates?queries=20
Requests/sec:    613.82
abouchez@tisab:~/dev/lib2/res/static/crypto$ wrk -c 512 -d 5s -t 4 http://localhost:8080/rawupdates?queries=20
Requests/sec:    458.81

Perhaps the difference you see on your PC comes from the fact that the ORM batches are run within a lock, which makes a difference with your high number of threads. But this is an unrealistic scenario for me: in such conditions, I would rather share a batch among threads requests, then use a background thread to actually send the batch to the DB asynchronously, or send it as blocking, after some input has been filled.

mpv · 2022-07-20 09:40:35

data.InitArrayFrom(arr) not compiled. should I wrote data.InitArrayFrom(arr, []) ?

ab · 2022-07-20 09:42:53

data.InitArrayFrom(arr, JSON_FAST_FLOAT);

mpv · 2022-07-20 09:44:54

for /rawfortunes it's ok. For fortunes - returns empty template

Last edited by mpv (2022-07-20 09:45:08)

ab · 2022-07-20 09:55:49

I don't have such trouble with current https://gist.github.com/synopse/f4b2c2e … 8aadff6c88

mpv · 2022-07-20 09:59:04

ab wrote:

Round 21 was... yesterday....
But they didn't accept any new framework since may.

It's a shame. I'll ask them to include us in round 21. With this war, I have a chance not to see the results of the 22nd round. And I've been waiting for so long that we will be ready to participate ...

mpv · 2022-07-20 10:06:57

ab wrote:

I don't have such trouble with current https://gist.github.com/synopse/f4b2c2e … 8aadff6c88

But I have
Free Pascal Compiler version 3.2.2 [2021/07/09] for x86_64
mORMot 2.0.3700.

Try to use setup_and_build.sh
Sorry - I must go - will return evening

ab · 2022-07-20 11:45:44

You need a fix from https://github.com/synopse/mORMot2/comm … e414370321 for it to work.

I could make another release today.

mpv · 2022-07-20 12:08:45

Ah... Round 21 results is already published Well, let's wait for 22.

I verify locally with latest variants fix - fortunes now works. Please, make a release to allow me to verify all test suite in docker environment

ab · 2022-07-20 16:25:44

I have made https://github.com/synopse/mORMot2/rele … g/2.0.3717

and some minor changes to
https://gist.github.com/synopse/f4b2c2e … 8aadff6c88

I guess we could even simplify:

 fDbPool := TSqlDBPostgresConnectionProperties.Create('tfb-database:5432', 'hello_world', 'benchmarkdbuser', 'benchmarkdbpass');

mpv · 2022-07-20 18:51:40

All tests are passed now. I wrote small (nodejs or unitybase) console utility to visualize test result
Benchmark vs lithium fastest in Full ORM class) on my AMD Ryzen5 5600G; RAM 32Gb; SSD;

./tfb --mode benchmark --test mormot lithium
node visualize_tfb_res.js

Max RPS:
┌──────────────┬────────────┬────────────┐
│   (index)    │ mormot     │ lithium    │
├──────────────┼────────────┼────────────┤
│   fortune    │   74318    │   90064    │
│  plaintext   │   920198   │  3388906   │
│      db      │   111119   │   99463    │
│    update    │   10177    │   25718    │
│     json     │   422771   │   544247   │
│    query     │   106665   │   94638    │
│ cached-query │   384818   │   528433   │
└──────────────┴────────────┴────────────┘

For `plaintext` our results is small because lithium uses HTTP pipelining. I don't think we should implement it - totally unusable in real world.

For `update` tfb put a warning what we read x2 rows - is ORM do select after update ?

Last edited by mpv (2022-07-20 18:53:48)

mpv · 2022-07-20 19:05:48

This is a warning:

./tfb --mode verify --test mormot
....
 WARN for http://tfb-server:8080/updates?queries=20
     20253 rows read in the database instead of 10240 expected. This number is excessively high.
     See https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview#specific-test-requirements
   PASS for http://tfb-server:8080/updates?queries=20
     Rows updated: 10124/10240

I think it is besause for each update `select unnest` is used... It's interesting how others do an batch update.. Will look

ab · 2022-07-20 19:36:59

Yes, the ORM does not make any SELECT after update, but I guess the "select unnest" is the culprit here - as you wrote.

I looked at lithium source code.
It is very short, using C++ templating and pre-processing a lot for compile-time serialization for instance. It is the source of all weird s:: notation within the source code.
https://matt-42.github.io/lithium/getting-started

Some remarks:
1. Its /fortunes test does not have any real template system, it is just a series of << output.
2. /plaintext is indeed using pipelining, and I don't understand why we could use it in production, as you wrote.
3. It uses fibers for DB connections, which is not always a win since we are sometimes faster, but it is more efficient for /update.
4. /cached-worlds does not follow the benchmark requirements: it is not using an ORM cache (as mORMot does), it is using a map.
5. /json is serialized at compile time I guess.
6. /update is very expressive, I like it.
7. but I wonder, with all those s:: pre-processing, how you could debug such an application?

So we can be proud of our little mORMot.

mpv · 2022-07-20 19:39:58

update query looks like:

update table set randomNumber = CASE id when $1 then $2 when $3 then $4 when $5 then $6 when $7 then $8 when $9 then $10 else randomNumber end where id in ($1,$3,$5,$7,$9)

I think we CAN use similar SQL generation, but I don't understand why it is faster

Last edited by mpv (2022-07-20 19:41:30)

ab · 2022-07-20 19:43:34

Is it faster when you benchmark it manually?

mpv · 2022-07-20 19:51:21

I did not benchmark it manually yet - just extract a query from framework sources.
My opinion what for real life out approach is better - if array length is changed we just reuse a prepared statement in our `::unnest` implementation. In `case-when` new statement is generated for each array length.
So let's keep it as-is. What do you think?

ab · 2022-07-20 20:01:20

I agree with you.
The fact that we can re-use the same SQL statement with multiple number of rows is a winner in practice.

But in lithium I see this pattern:

update table set a=tmp.a,b=tmp.b from (values (1,2,3),(4,5,6),(7,8,9),(0,1,2)) as tmp(id,a,b) where tmp.id=table.id

It is coded in bulk_update - for PostgreSQL only, and with ::int fields only (!!!)
It is similar to the case/when/then, but has a shorter syntax.
Edit: this syntax is mentioned in the official PostgreSQL doc https://www.postgresql.org/docs/current/sql-values.html

The more I look into lithium, the more I suspect it was design to shine in TechEmpoweredBenchmark, less in real life.
Their bulk update work only for PostgreSQL and integer fields. Do you still call it an ORM?

mpv · 2022-07-20 20:09:37

Agree with you about lithium

My variant of statement is from `drogon` (the fastest one in update).

But I prefer our `::unnest` case. Let's go with it - may be on dedicated Postgres it will work better when on my PC.

ab · 2022-07-20 20:17:22

I could make the mORMot TRestBatch marshalling lower, as I did with native SQLite3 to achieve the best results (about 1 million inserts per seconds on my old laptop).
There is currently an optimized JSON array generated by TRestBatch which is converted back into several arrays of JSON objects, then several arrays of SQL values, then one array of PostgreSQL JSON arrays....
I guess we could pass directly from the JSON array generated by TRestBatch into an array of JSON arrays as expected by PostgresSQL unnest.

But I am not sure it would help a lot in practice.
It would for sure reduce the memory allocation a lot. Even more than the direct raw update, which still use several arrays of SQL values in-between.
Anyway, I am still considering it. It would not hurt for PostgreSQL.

mpv · 2022-07-20 21:13:33

A long-long way is near to finish. I wanted mORMot to participate in TechEmpower benchmark since 2013

But in 2013 mORMot was Windows only.

After Linux build appears (a huge work of Arnaud and contributors) we wait for stable Free Pascal compiler version with all features needed for mORMot.

In parallel Arnaud implements a thread pull based HTTP server (I help in performance tuning and testing a little) and I add low level SynDBPostgres driver.

And now - after 9 years of work the first Pascal implementation of FrameworkBenchmarks is waiting for approval.

I think this is a good step forward not only for mORMot community, but for all Pascal'ish.

ab · 2022-07-21 11:42:21

Yes, this is a huge achievement for more visibility of both mORMot and pascal.
Thanks everyone for your interest, suggestions, testing and support.

They wrote about next round 22, which would take place around October-November.
https://github.com/TechEmpower/Framewor … ssues/7475
I hope they will include your mORMot pull request for that time!

In the meantime, they are likely to change the tags of some frameworks.
They even removed some lithium modes, which were not following the requirements.

Let's see where our little rodent will be.

mORMot Open Source

#1 2020-08-11 14:40:54

High-performance frameworks

#2 2020-08-11 15:27:28

Re: High-performance frameworks

#3 2020-08-17 10:48:37

Re: High-performance frameworks

#4 2022-06-16 08:47:39

Re: High-performance frameworks

#5 2022-07-18 08:10:16

Re: High-performance frameworks

#6 2022-07-18 11:39:06

Re: High-performance frameworks

#7 2022-07-18 12:10:23

Re: High-performance frameworks

#8 2022-07-18 13:35:49

Re: High-performance frameworks

#9 2022-07-18 14:39:33

Re: High-performance frameworks

#10 2022-07-18 15:16:16

Re: High-performance frameworks

#11 2022-07-18 16:58:09

Re: High-performance frameworks

#12 2022-07-18 20:13:28

Re: High-performance frameworks

#13 2022-07-19 07:41:52

Re: High-performance frameworks

#14 2022-07-19 08:14:20

Re: High-performance frameworks

#15 2022-07-19 08:40:14

Re: High-performance frameworks

#16 2022-07-19 15:42:23

Re: High-performance frameworks

#17 2022-07-19 18:35:02

Re: High-performance frameworks

#18 2022-07-19 19:32:48

Re: High-performance frameworks

#19 2022-07-19 19:54:19

Re: High-performance frameworks

#20 2022-07-19 20:28:18

Re: High-performance frameworks

#21 2022-07-19 20:34:03

Re: High-performance frameworks

#22 2022-07-19 21:00:08

Re: High-performance frameworks

#23 2022-07-19 21:06:30

Re: High-performance frameworks

#24 2022-07-19 21:06:35

Re: High-performance frameworks

#25 2022-07-19 21:14:27

Re: High-performance frameworks

#26 2022-07-19 21:32:30

Re: High-performance frameworks

#27 2022-07-19 21:35:33

Re: High-performance frameworks

#28 2022-07-20 07:25:21

Re: High-performance frameworks

#29 2022-07-20 09:04:21

Re: High-performance frameworks

#30 2022-07-20 09:19:59

Re: High-performance frameworks

#31 2022-07-20 09:40:35

Re: High-performance frameworks

#32 2022-07-20 09:42:53

Re: High-performance frameworks

#33 2022-07-20 09:44:54

Re: High-performance frameworks

#34 2022-07-20 09:55:49

Re: High-performance frameworks

#35 2022-07-20 09:59:04

Re: High-performance frameworks

#36 2022-07-20 10:06:57

Re: High-performance frameworks

#37 2022-07-20 11:45:44

Re: High-performance frameworks

#38 2022-07-20 12:08:45

Re: High-performance frameworks

#39 2022-07-20 16:25:44

Re: High-performance frameworks

#40 2022-07-20 18:51:40