mORMot Open Source

Btw I’m still amazed at how you made faster encryption code then OpenSSL. It is not a simple code to even understand, let alone make it this fast.
Well done!

As one who read it and know what it does exactly, can you explain how this way of encryption works?
I guess every page is encrypted when written. But a little more on how every page is encrypted in relation to the neighbors pages or HMAC storage. Also why the first bytes of the file (SQLite format) is not encrypted.
In summery I like to know more on how the encryption of SQLite is implemented without going in depth.

Thank you very much. I need to test your suggestion and will let you know if the cache is needed.

@JD thank you, although I need to have independent file for databases.
@ab thank for the notes. I want to do it with SQLite if possible. Can you elaborate on how can I route the class to different DBs while using interface based?
I was thinking about a virtual table but maybe it is over engineering? I guessed maybe there is something better in mORMot.
PS, I’m using mORMot 2 for this new project.

It was a copy and paste issue, fixed it.
Is the style of the solution correct? Registering the class and using ParseNewInstance?

Just checked again,
The error can be reproduced with with either the latest mORMot2 or sqlite revision 3.37.2 and FPC trunk of now.
Most mORMot tests fails, it seems a memory problem.
One interesting thing for me is that, why such an error is not detected by some catch and can mess around with other parts of code, even the parts that seem unrelated.
I was suspected of patching RTL, but activating NOPATCHRTL, makes everything worse.

Thank you!

I found one of the PDF that describes the CRC code (and has the code itself) but not the exact files that provides the exact binary.
Do you think it is a good idea to have the URL or the codes as reference?

Thank you for the clarification. Interesting as always.

Thank you for the updates.

Thanks for the update. Checked it and it works near 20GB/s! Great!

About TDynArrayHasher, it still does not work for FPC even with the new parenthesis, it seems an @ is needed.
I tried to minimize the problem in a new project and included mORMot define too, it compiles correctly, the problem is only happening in the unit. Sorry for the trouble, but I think it worth to be able to compile with the latest version of FPC.

Very nice.
But I can not compile it with the latest Trunk FPC on Win10 X64. IsValidUtf8Avx2 crashes.
mORMot test fail too on this function.
By disabling the AVX version, Pas version works 3GB/s for me on i9.
Unrelated note: In TDynArrayHasher.FindOrNew, checking  Assigned(Compare) fails on FPC trunk. Previous versions work without problem.

The validate_utf8 function is light and needs no memory using SIMD. More info: https://lemire.me/blog/2018/05/16/valid … -per-byte/
But as far as JSON needs, mORMot SAX approach uses no memory, but simdjson builds a DOM-like info that takes much (near 5X I think) more memory.

Nothing much, that was my fault anyway.
I made the pull request, please review.

Great. I inherited TSqlite3LibraryStatic with and override the BeforeInitialization with this:

  procedure TCustomSqlite3LibraryStatic.BeforeInitialization;
  begin
    sqlite3.config(SQLITE_CONFIG_MEMSTATUS, Integer(True));
    inherited BeforeInitialization;
  end;         

Now it works.
I will make a pull request for the config comments and show how it must be used to prevent future misunderstandings.

Bare metal, Windows.

Thank you.

I was reporting string dictionary without set capacity, surprised that why the new algorithm is not faster.
This time I did an another test: https://gist.github.com/OkobaPatino/cec … 358984f5c5
These are the result of latest mORMot, latest FPC, Windows and i9.

Remarks:
In increasing string AesNiHash32 is slower, near 25% with SetCapacity and near 50% without SetCapacity.
In random GUID AesNiHash32 is a bit slower.
If dictionary capacity is set beforehand, Add (400%) and Find (25%) is faster, no matter the Hasher.

Questions:
Find is slower when you do not set capacity before hand. Why?
Can I ask where much less collisions come to play with AesNiHash32 as it is always slower than crc32c?

BTW It would be nice to have the ability of choosing the Hasher of the dictionary now that we have more options.

mormot.db.raw.sqlite3 has no initialization.
mormot.db.raw.sqlite3.static has one, at it creates TSqlite3LibraryStatic. In the Create, it calls ForceToUseSharedMemoryManager and I did disable it as a test an no changes.
Am I checking a wrong place?

I thought that is not a huge code as rules stated.
If you mean ForceToUseSharedMemoryManager, I tried disabling that too, but no change.
I like to get status of the DB to have a monitor on the DB, that was one of the reasons I've ported many of these functions previously for the sqlite3 units.

Limiting prevents sudden issues. As for solution, I checked the result for 3M or 1000 bytes and load it after save, it worked correctly. I used this code for months and no problem and passed the checks until I faced this number.
Another question is why JSOn writer has a problem? I did use an external writer with owned file stream and still problem happens in Base64.

It will happen again on:
- Set CompressAlgo to nil
- Saving to JSON
- Using random value for V
I faced this problem in a project with real values with the same exact behavior.

To be clear, I tried any combination of the above in such code and also in the context of my project with real values:

begin
  D := TSynDictionary.Create(TypeInfo(TIntegerDynArray), TypeInfo(TValueDynArray));
  D.CompressAlgo := nil;
  for I := 1 to 2319851 do
  begin
    TAesPrng.Main.FillRandom(@V, SizeOf(V));
    D.Add(I, V);
  end;
  D.SaveToJson;
  WriteLn('Done');
  ReadLn;
end.