mORMot Open Source

@zen010101 the test mentioned is very old.
Here is a good one: https://morotti.github.io/lzbench-web/

Then ZSTD is the way to go.
Thank you.

Default MM leaves the memory and does not free it to the OS as far as I know.

Your guess about FillZero is right.

Are you testing on Windows 11 and see different results?
In theory the MM can allocate big blocks (128 MB) and free them when all of the sub blocks are used. But it can be a complicated task if needed to cover general cases.

Yes! My test is done with fpcx64mm. The default memory manager allocation time is much slower. fpcx64mm allocation is very fast, but still freeing this much of memory takes time and I want to speed it up.
I just tested on Linux and it takes 100ms to free, but on Windows near 4 seconds.

As I said I like to query the list using SQLite VTab and I can not change the algorithm of VTab if you mean that. SQLite xBestIndex asks for a ordered result to speedup order or group by and that is the reason I asked the question.

In https://xxhash.com/ footer there is a list of clients.

Thank you very much. Your estimation seems solid.
Yes, a good chunk of the data is updated (hence the need to sync before letting the user know it is done), and most of the other requests would be read and not written (I should have been more clear).

You are right; there is no need to close the connections if the cache size is so low. I could keep them open indefinitely.

Having multiple servers answer write requests seems like the way to go, as if one goes down, the data will not be lost. And in this way, we can use queue and batch, as you said, to speed it up.

Great advice. Yes TRestBatch seems the way to go. I was and am mostly worried of returning success to the user, but in a crash I lose their information, hence thinkin about Full.

I should ask (as you may be one of the best people who used SQLite a lot), compare to PostgreSQL, what is your go to for servers with a lot of concurrent connection, or even for my case, many databases for many users?
I looked at benchmark (extdb-bench) test you did and SQLite is 4 to 5 times (900K/s to 190K/s) faster than PostgreSQL if you use Off and Batch and Transaction, but PostgreSQL is much faster (9.5K/s to 190/s) if I want to use Full without  Transaction (separate connections doing one at a time).

Thank you for the helpful answer. Yes it is in the wrong forum, but if I remember correctly, when I initially opened this topic, Version 2 was in Beta and there was not a dedicated forum for it. If it is possible, please move it.
Those samples were life-saving; without them, I couldn't find out what I should do. Thanks to Thomas and you for them. Martin samples are great too.
I will look forward to the updates, but please create samples for them so a newbie like me knows how to use them. Your codes are great and advanced; please dont assume finding a way in them is walking in a park. They are like being in a big library—too many great things to see and read and get overwhelmed  

Anyway, I tried my best to make it work. Can you please check it?
https://gitlab.com/-/snippets/3670020

Questions:
1- I didn't use Interface based (SOA?) because I wanted to use the server with a Browser client. Is that a right assumption?
2- I tried using TRestOrmServerDB , but it asks for a Rest, what should I give it? If TRestServerDB, then what was the reason you said I should use TRestOrmServerDB? I tried passing nil for the Rest parameter, but TRestOrm.InternalAdd raise error because it needs fRest.
3- Is there a better way to handle caching? 
4- If I don't set LockingMode, and Synchronous the speed is very low. LockingMode is fine, but setting Synchronous to smOff would be dangerous I guess. Can I ask what would you suggest in production? I can use PostgreSQL too, but I guessed SQLite would be faster and less hassle.
5- What is the best way to handle the locking problem when multiple connection come and want to use one TRestServerDB/DB?

After a while I am wondering around in the ORM part of mORMot (always funny that you call it little) and I looking for my way to implement that routing to multiple database files. So I guessed asking, maybe you did something in past years.

Honestly I expected even more powerful one. mORMot results are fantastic.
Did you configure the server machine for such high performance?
Thanks for the commit.

Thank you!

Ok good to here that. I tried two computers, one a x86 VM and one M1. Both installed Trunk for x86 with fpcupdeluxe and that was it. I didn't do anything in particular.
My current version is Lazarus 3.99 (rev main_3_99-1261-gb2b86ebaef) FPC 3.3.1 x86_64-darwin-cocoa
Lazarus on macOS is not the best in user experience, but was your problem compiler issues?

Oh believe me, compare to me, you are a god of C
I will look into it.

Btw I’m still amazed at how you made faster encryption code then OpenSSL. It is not a simple code to even understand, let alone make it this fast.
Well done!

As one who read it and know what it does exactly, can you explain how this way of encryption works?
I guess every page is encrypted when written. But a little more on how every page is encrypted in relation to the neighbors pages or HMAC storage. Also why the first bytes of the file (SQLite format) is not encrypted.
In summery I like to know more on how the encryption of SQLite is implemented without going in depth.

Thank you very much. I need to test your suggestion and will let you know if the cache is needed.

@JD thank you, although I need to have independent file for databases.
@ab thank for the notes. I want to do it with SQLite if possible. Can you elaborate on how can I route the class to different DBs while using interface based?
I was thinking about a virtual table but maybe it is over engineering? I guessed maybe there is something better in mORMot.
PS, I’m using mORMot 2 for this new project.

It was a copy and paste issue, fixed it.
Is the style of the solution correct? Registering the class and using ParseNewInstance?