Posts by okoba

okoba · mORMot 1

Oh believe me, compare to me, you are a god of C
I will look into it.

okoba · mORMot 1

Yes I benchmarked all compression library available in mORMot and ZSTD was faster and had better compression. libdeflate is great, but ZSTD can do better.
Although I agree, libdeflate is the good way especially for server needs. But for custom client and server codes, ZSTD can be the better choice.
I could take care of adding ZSTD to mORMot 2 in the current TAlgoCompress style, but the problem is mostly managing to prepare the best static libraries (including custom emmory manger like you done for SQLite and others, or shared C Lib codes). I don't know C and building their libraries is just a pain.
If thats possible for you to prepare that, I can write the code for the current official dll.

okoba · mORMot 1

Currently mORMot supports many compression algorithm, and I wanted to ask if there is a plan for ZSTD support?

okoba · mORMot 1

@mpv I wanted to say thank you and I am following the work.

okoba · Source Code repository

Hi,

I am curious about the state of working with Fossil and Git.
Looking at mORMot and mORMot2 repositories, it seems ab uses Fossil for V1 and GitHub for V2.
Can I ask, especially from ab, how did you find working with Fossil and syncing with Git? I like to work with Fossil, but the world uses Git and GitHub, so if I even do that, I probably need a way to sync them.
I want to know, from your experience, is it worth it?

okoba · Low level and performance

Here is a post that may be interesting to check and compare with mORMot: https://www.corsix.org/content/fast-crc32c-4k
And here is another one for Apple M1: https://dougallj.wordpress.com/2022/05/ … -apple-m1/

okoba · mORMot 1

Btw I’m still amazed at how you made faster encryption code then OpenSSL. It is not a simple code to even understand, let alone make it this fast.
Well done!

okoba · mORMot 1

Great! Thank you. I suggest add them to the encryption code of SQLite unit for future. It is informative. Also adding a line for how to prepare the JSON structure is good. I can provide a patch if you see fit.

So th patch is needed mostly for WAL and adding PRAGMA, other features seems not very needed.

Why did you said the AES256 is not needed password-derivated key? I couldn’t find a reference for it?
When is suggest to use OFB or CTR?
Exactly what bytes are not encrypted? From 0 to 16 or more for page size?
Can we have them encrypted to make the file fully noise like? It will be useful when it is not needed to be used by other tools other than severs, and also storing backups of the database without leaking any information.

okoba · mORMot 1

As one who read it and know what it does exactly, can you explain how this way of encryption works?
I guess every page is encrypted when written. But a little more on how every page is encrypted in relation to the neighbors pages or HMAC storage. Also why the first bytes of the file (SQLite format) is not encrypted.
In summery I like to know more on how the encryption of SQLite is implemented without going in depth.

okoba · mORMot 1

@ab can I ask why did you use SQLite3MultipleCiphers VFS approach instead of adding VFS in runtime in using SQLite API? I mean why patching SQLite when you write most of the encryption anyway.
Could you add a VFS in Pascal like the sqlite3mc_vfs?

okoba · mORMot 1

Thank you very much. I need to test your suggestion and will let you know if the cache is needed.

okoba · mORMot 1

@ab can you help on this?

okoba · mORMot 1

@JD thank you, although I need to have independent file for databases.
@ab thank for the notes. I want to do it with SQLite if possible. Can you elaborate on how can I route the class to different DBs while using interface based?
I was thinking about a virtual table but maybe it is over engineering? I guessed maybe there is something better in mORMot.
PS, I’m using mORMot 2 for this new project.

okoba · mORMot 1

Hello,

I have multiple databases for each customer (need to be independent databases for legal reasons), with the same structure or model. And I like to use interface methods to access them. The API should have independent methods called by user, so no direct access to database tables.
What is the best way to rout incoming requests to multiple databases?

To be clear, what is the best way to rout like:
User1 calls /api/logger/add?data=XYZ&key=user1_key
Server checks user1_key, finds that the data need to be added to user1.db and like to insert to that file.

Perhaps, in the future, the Server need to call another server (User1_Server) to do the actual insert. I think this part can be done by the master/slave architecture described in the documentation, but I appreciate any needed info.

okoba · mORMot 1

It was a copy and paste issue, fixed it.
Is the style of the solution correct? Registering the class and using ParseNewInstance?

okoba · mORMot 1

Here is my response to my question. I do not know if this is the best option though.

procedure TItemReader.ReadItem(var Context: TJsonParserContext; Data: pointer);   
var
  S: PUTF8Char;
  SP: RawUTF8;
  L: integer;
  C: TItemClass;
  Instance: TObject;   
begin
  Result := nil;
  P := Context.Json;
  OldP := P;
  P := JsonObjectItem(P, 'CustomClass');
  if P = nil then
    raise Exception.Create('Invalid Item');
  JSONRetrieveStringField(P, S, L, False);
  SetString(SP, S, L);
  case SP of
    'ItemA': C := TItemA;
    'ItemB': C := TItemB;
    else
      raise Exception.Create('Unknown Item: ' + SP);
  end;
  Context.Info := Rtti.RegisterClass(C);
  Instance := TRttiJson(Context.Info).ParseNewInstance(Context);           
end;

And register it like this:

TRttiJson.RegisterCustomSerializer(TypeInfo(TItem), @ReadItem, nil);

okoba · mORMot 1

I had a code with mORMot1 like:

  TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TItems), @ReadItem, nil);

and

 
function TItemReader.ReadItem(P: PUTF8Char; var AValue; out AValid: boolean): PUTF8Char;
var
  V: TObject absolute AValue;
  OldP, S: PUTF8Char;
  SP: RawUTF8;
  L: integer;
begin
  AValid := False;
  Result := nil;
  if P = nil then
    Exit;
  OldP := P;
  P := JsonObjectItem(P, 'CustomClass');
  if P = nil then
    raise Exception.Create('Invalid Item');
  JSONRetrieveStringField(P, S, L, False);
  SetString(SP, S, L);
  case SP of
    'ItemA': V := TItemA.Create;
    'ItemB': V := TItemB.Create;
    else
      raise Exception.Create('Unknown Item: ' + SP);
  end;
  AValid := True;
  Result := JSONToObject(AValue, OldP, AValid, nil, JSONTOOBJECT_TOLERANTOPTIONS);
end;

and now with mORMot2, I can not convert it.
I thought I should do this but it leads to unknown errors.

 
function TItemReader.ReadItem(P: PUTF8Char; var AValue; out AValid: boolean): PUTF8Char;
var
  V: TObject absolute AValue;
  OldP, S: PUTF8Char;
  SP: RawUTF8;
  L: integer;
begin
  AValid := False;
  Result := nil;
  P := Context.Json;
  OldP := P;
  P := JsonObjectItem(P, 'CustomClass');
  if P = nil then
    raise Exception.Create('Invalid Item');
  JSONRetrieveStringField(P, S, L, False);
  SetString(SP, S, L);
  case SP of
    'ItemA': V := TItemA.Create;
    'ItemB': V := TItemB.Create;
    else
      raise Exception.Create('Unknown Item: ' + SP);
  end;
  AValid := True;
  Context.Json := JSONToObject(Instance, Context.Json, Context.Valid, nil, JSONTOOBJECT_TOLERANTOPTIONS);     
end;

What is the correct way to unserializing and array of classes?
Note: I can not use ParseNewObject, as the CustomClass property may not be the first properties and some other custom needs.

okoba · mORMot 1

Now it is working. Thank you.

okoba · mORMot 1

Just checked again,
The error can be reproduced with with either the latest mORMot2 or sqlite revision 3.37.2 and FPC trunk of now.
Most mORMot tests fails, it seems a memory problem.
One interesting thing for me is that, why such an error is not detected by some catch and can mess around with other parts of code, even the parts that seem unrelated.
I was suspected of patching RTL, but activating NOPATCHRTL, makes everything worse.

okoba · mORMot 1

I was using an old Trunk with mORMot2 from two weeks ago (sqlite revision 3.37.2 172d789404d400ed60447a2b757c5ac770906ae9) and everything was fine, but I pulled mORMot2 as habit and memory crash ...
So I updated my Trunk and now the problem I reported.
Unfortunately I'm stuck with Trunk (3.3.1) for generics and I cannot go back.

okoba · mORMot 1

I recently pulled mORMot2 and ran into some exceptions when setting FPC's internal string methods.I updated FPC and Lazarus to Trunk from Git and now it runs, but my SQLite Vtab codes are not working or buggy.
Just adding a useless log code like WriteLn(RawUtf8(argv^[0])); after begin of xConnect makes it work.
It solves the issue in some projects but not in others. Note that the RawUtf8 cast is the solving part, as reading the argv or WriteLn on their own does not change anything.
It is a weird case, so I'd like to know what the best version to try.

I am working on Windows 64bit and the stable version of FPC is working with the latest mORMot2, but I cannot use the stable version as the Generics code is lagging behind.
I'd appreciate any suggestions.

okoba · mORMot 1

Very interesting article.
I like to know more about how you debug these problems, like the TAsyncServer case.

okoba · mORMot 1

Thank you!

okoba · mORMot 1

Hi,

TSynQueue.Count method uses ReadOnlyLock, and it makes problems when the Queue is shared between threads that are using the count while others are reading from or writing into it.
It was a weird error to find but in some cases it trough exceptions and maybe it is a good idea to use ReadWriteLock instead in Count. In my case it solved the issue like this:

   with Queue do
        try
          Safe.ReadWriteLock;
          QueueCount := Queue.Count;
        finally
          Safe.ReadWriteUnLock;
        end;

okoba · mORMot 1

I found one of the PDF that describes the CRC code (and has the code itself) but not the exact files that provides the exact binary.
Do you think it is a good idea to have the URL or the codes as reference?

okoba · mORMot 1

Hi,

I wanted to check and compile static files, and as noted there are notes and scripts for many of them like libquickjs. But I could not find the code for some files like r crc32c64.obj or sha512-x64sse4.obj.
Are those available ?

okoba · Low level and performance

Thank you for the clarification. Interesting as always.

okoba · Low level and performance

I noticed that when I include mormot.core.data, it makes comparing string (A=B) fast. Checking that subject I found that mormot.core.rtti patches the fpc_ansistr_compare and it is very fast, excellent.
The question for me is that why mORMot do not use this functions for other parts (eg DynArray) and uses StrComp instead?

   procedure Test;
  var
    I, C: Int64;
    A, B: String;
    T: TLocalPrecisionTimer;
  begin
    A := 'Hello World';
    UniqueString(A);
    B := 'Hello World';
    UniqueString(B);
    C := 0;
    T := TLocalPrecisionTimer.CreateAndStart;
    for I := 1 to 50 * 1000 * 1000 do
      if A = B then //_ansistr_compare_equal //95ms
        //if SortDynArrayString(A, B) = 0 then
        //if StrComp(PChar(A), PChar(B)) = 0 then //367ms
        C += 1;
    Writeln(C, ' ', T.Stop);
  end;

okoba · Low level and performance

Thank you for the updates.

okoba · Low level and performance

Great!
Yes mORMot is the very fast and a blog post as an update to previous ones seems nice. May I suggest having an independent demo that clearly shows the benefits and speed? It helps to answer questions like this topic.
Also it may be a good idea to add JsonTools (from this topic) too as it is very clean and seems fast.
Topic: https://forum.lazarus.freepascal.org/in … opic=46533

PS, can you please update the TDynArrayHasher compile issue with trunk? It helps someone like me to keep up with your fast updates while maintaining daily codes.

okoba · Low level and performance

Thanks for the update. Checked it and it works near 20GB/s! Great!

About TDynArrayHasher, it still does not work for FPC even with the new parenthesis, it seems an @ is needed.
I tried to minimize the problem in a new project and included mORMot define too, it compiles correctly, the problem is only happening in the unit. Sorry for the trouble, but I think it worth to be able to compile with the latest version of FPC.

okoba · Low level and performance

Thanks for the update.
About Trunk, it let me try mORMot code with latest FPC updates as I use mORMot a lot for daily tasks like array, dictionary, Unicode, file and json. And until the latest update to the TDynArrayHasher it worked just fine and passed all the test so it may be a good idea to keep it running and testing latest things.
It is the case for V2, I agree with you on V1 of mORMot as it was always problematic to use it with Trunk. But your updates to V2 made it very compatible and comfortable to use.

okoba · Low level and performance

Very nice.
But I can not compile it with the latest Trunk FPC on Win10 X64. IsValidUtf8Avx2 crashes.
mORMot test fail too on this function.
By disabling the AVX version, Pas version works 3GB/s for me on i9.
Unrelated note: In TDynArrayHasher.FindOrNew, checking Assigned(Compare) fails on FPC trunk. Previous versions work without problem.

okoba · Low level and performance

I think you may get better results trying to rewrite it and prevent dependency.

okoba · Low level and performance

The validate_utf8 function is light and needs no memory using SIMD. More info: https://lemire.me/blog/2018/05/16/valid … -per-byte/
But as far as JSON needs, mORMot SAX approach uses no memory, but simdjson builds a DOM-like info that takes much (near 5X I think) more memory.

okoba · mORMot 1

Thank you. I will try to match the style better next time.

okoba · mORMot 1

Nothing much, that was my fault anyway.
I made the pull request, please review.

okoba · mORMot 1

Yes and I tried to propose a pull request but it had conflicts so I need to redo and check again.
As I added Configuration Options comments too.

okoba · mORMot 1

Great. I inherited TSqlite3LibraryStatic with and override the BeforeInitialization with this:

  procedure TCustomSqlite3LibraryStatic.BeforeInitialization;
  begin
    sqlite3.config(SQLITE_CONFIG_MEMSTATUS, Integer(True));
    inherited BeforeInitialization;
  end;

Now it works.
I will make a pull request for the config comments and show how it must be used to prevent future misunderstandings.

okoba · mORMot 1

@ab, can you please let me know if you meant ForceToUseSharedMemoryManager or something else? I tired to disable it and again SQLite report zero.
As documented (https://www.sqlite.org/c3ref/c_status_malloc_count.html) SQLITE_STATUS_MEMORY_USED retunes xSize results and I checked your versions, the result is non zero, so the issue may be somewhere else?
I like to have a check on memory usage and it seems the standard way for SQLite.

okoba · mORMot 1

Bare metal, Windows.

okoba · mORMot 1

Here is the new results in an i9:
https://gist.github.com/OkobaPatino/12b … 86fc1dfe06

okoba · mORMot 1

Thank you.

okoba · mORMot 1

Great update. Thank you ab!
I've checked your changes and tests, and rerun my test in an i3 too. Without SetCapacity results are 2X faster for Add() and 1.3X for find. Also Find() is almost the same with and without SetCapacity. It's great!

Two questions, is there a count limit for TSynDictionary / TDynArrayHasher? What will be the O for more than 10M?

okoba · mORMot 1

I was reporting string dictionary without set capacity, surprised that why the new algorithm is not faster.
This time I did an another test: https://gist.github.com/OkobaPatino/cec … 358984f5c5
These are the result of latest mORMot, latest FPC, Windows and i9.

Remarks:
In increasing string AesNiHash32 is slower, near 25% with SetCapacity and near 50% without SetCapacity.
In random GUID AesNiHash32 is a bit slower.
If dictionary capacity is set beforehand, Add (400%) and Find (25%) is faster, no matter the Hasher.

Questions:
Find is slower when you do not set capacity before hand. Why?
Can I ask where much less collisions come to play with AesNiHash32 as it is always slower than crc32c?

BTW It would be nice to have the ability of choosing the Hasher of the dictionary now that we have more options.

okoba · mORMot 1

I've experienced 2X slowness with TSynDictionary (in Add and Find) when I use mormot.crypt.core and it activates AES-NI.

    
  for I := 0 to High(A) do
    A[I] := IntToStr(I);       
  Dic := TSynDictionary.Create(TypeInfo(TStringDynArray), TypeInfo(TInt64DynArray));
  T := GetTickCount64();
  for I := 0 to High(A) do
    Dic.Add(A[I], I);
  WriteLn('Fill: ', GetTickCount64() - T);

  T := GetTickCount64();
  for I := 1 to High(A) do
    Dic.Find(A[I]);
  WriteLn('Find: ', GetTickCount64() - T);
  Dic.Free;

Here is the output of the mormot test:

  - crc32c: 320,100 assertions passed  69.16ms
      pas 430.8 MB/s fast 2 GB/s sse42 8.6 GB/s sse42+aesni 9.7 GB/s

Tested on Windows, latest mORMot and FPC on two machines with i3 and i9.

okoba · mORMot 1

mormot.db.raw.sqlite3 has no initialization.
mormot.db.raw.sqlite3.static has one, at it creates TSqlite3LibraryStatic. In the Create, it calls ForceToUseSharedMemoryManager and I did disable it as a test an no changes.
Am I checking a wrong place?

okoba · mORMot 1

I like to have this for memory tables too, as it helps to monitor my cache.
May it be something in the amalgamation?

okoba · mORMot 1

I thought that is not a huge code as rules stated.
If you mean ForceToUseSharedMemoryManager, I tried disabling that too, but no change.
I like to get status of the DB to have a monitor on the DB, that was one of the reasons I've ported many of these functions previously for the sqlite3 units.

okoba · mORMot 1

Hi,

I have a problem with SQLite memory status. I am trying to get the DB memory usage, but it will be always zero.
Do you have any idea why?
Also zero even with memory DB.
status64 results non-zero value for SQLITE_STATUS_PAGECACHE_OVERFLOW flag.

var
  DB: TSqlDataBase;
  Req: TSqlRequest;
  Current, HighWater: Int64;
  I: Integer;
begin
  DB := TSqlDataBase.Create('test.db');
  sqlite3.config(SQLITE_CONFIG_MEMSTATUS, Integer(True));
  WriteLn('VersionNumber: ', sqlite3.VersionNumber);

  DB.TransactionBegin;
  DB.Execute('CREATE TABLE IF NOT EXISTS "Test" ("ID" INTEGER NOT NULL PRIMARY KEY, "Value" INTEGER);');
  Req.Prepare(DB.DB, 'INSERT INTO "Test"("Value") VALUES(?);');
  with Req do
    for I := 1 to 10000 do
    begin
      Bind(1, I);
      Step;
      Reset;
    end;
  Req.Close;
  DB.Commit;
  WriteLn('RecordCount: ', DB.ExecuteNoExceptionInt64('SELECT COUNT(*) FROM Test;'));

  Current := 0;
  HighWater := 0;
  WriteLn('status64: ', sqlite3.status64(SQLITE_STATUS_MEMORY_USED, @Current, @HighWater, Integer(False)));
  WriteLn('Current: ', Current);
  WriteLn('HighWater: ', HighWater);
  WriteLn('memory_used: ', sqlite3.memory_used);
  WriteLn('memory_highwater: ', sqlite3.memory_highwater(Integer(False)));

  DB.Free;
end.

mORMot Open Source

#1 Re: mORMot 1 » ZSTD support » 2023-09-12 12:40:23

#2 Re: mORMot 1 » ZSTD support » 2023-09-12 12:06:59

#3 mORMot 1 » ZSTD support » 2023-09-12 10:36:28

#4 Re: mORMot 1 » High-performance frameworks » 2022-11-07 06:50:34

#5 Source Code repository » Source control » 2022-10-15 08:17:26

#6 Low level and performance » Faster CRC32 on x86 and M1 » 2022-08-01 14:16:23

#7 Re: mORMot 1 » 10.4 sydney issues » 2022-04-20 21:46:30

#8 Re: mORMot 1 » 10.4 sydney issues » 2022-04-20 19:51:42

#9 Re: mORMot 1 » 10.4 sydney issues » 2022-04-20 15:57:03

#10 Re: mORMot 1 » 10.4 sydney issues » 2022-04-20 13:21:57

#11 Re: mORMot 1 » Routing requests to multiple database » 2022-04-17 10:49:31

#12 Re: mORMot 1 » Routing requests to multiple database » 2022-04-17 05:33:00

#13 Re: mORMot 1 » Routing requests to multiple database » 2022-04-07 19:50:42

#14 mORMot 1 » Routing requests to multiple database » 2022-04-07 16:34:45

#15 Re: mORMot 1 » Unserializing custom array of classes with mORMot2 » 2022-02-19 20:58:41

#16 Re: mORMot 1 » Unserializing custom array of classes with mORMot2 » 2022-02-19 16:20:41

#17 mORMot 1 » Unserializing custom array of classes with mORMot2 » 2022-02-17 13:40:25

#18 Re: mORMot 1 » mORMot2 with FPC Trunk » 2022-02-14 09:58:44

#19 Re: mORMot 1 » mORMot2 with FPC Trunk » 2022-02-13 21:12:00

#20 Re: mORMot 1 » mORMot2 with FPC Trunk » 2022-02-13 20:18:38

#21 mORMot 1 » mORMot2 with FPC Trunk » 2022-02-13 14:34:57

#22 Re: mORMot 1 » Three Locks To Rule Them All » 2022-01-24 08:52:25

#23 Re: mORMot 1 » TSynQueue.Count » 2022-01-22 08:13:18

#24 mORMot 1 » TSynQueue.Count » 2022-01-20 11:37:42

#25 Re: mORMot 1 » Compile from source of static files » 2021-08-11 07:23:02

#26 mORMot 1 » Compile from source of static files » 2021-08-10 16:58:23

#27 Re: Low level and performance » Fast compare string » 2021-08-09 08:33:30

#28 Low level and performance » Fast compare string » 2021-08-08 16:49:48

#29 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-28 11:19:21

#30 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-27 20:21:45

#31 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-27 08:51:19

#32 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-26 19:36:22

#33 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-26 14:45:48

#34 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-26 07:43:58

#35 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-25 08:09:55

#36 Re: mORMot 1 » SQLite memory usage » 2021-07-20 09:51:42

#37 Re: mORMot 1 » SQLite memory usage » 2021-07-20 08:16:07

#38 Re: mORMot 1 » SQLite memory usage » 2021-07-20 08:01:25

#39 Re: mORMot 1 » SQLite memory usage » 2021-07-20 07:04:08

#40 Re: mORMot 1 » SQLite memory usage » 2021-07-19 12:00:54

#41 Re: mORMot 1 » AES-NI speed in Dictionary » 2021-06-01 13:35:48

#42 Re: mORMot 1 » AES-NI speed in Dictionary » 2021-06-01 12:33:08

#43 Re: mORMot 1 » AES-NI speed in Dictionary » 2021-05-31 19:12:37

#44 Re: mORMot 1 » AES-NI speed in Dictionary » 2021-05-31 17:49:24

#45 Re: mORMot 1 » AES-NI speed in Dictionary » 2021-05-29 08:11:58

#46 mORMot 1 » AES-NI speed in Dictionary » 2021-05-28 14:15:58

#47 Re: mORMot 1 » SQLite memory usage » 2021-05-15 17:38:33

#48 Re: mORMot 1 » SQLite memory usage » 2021-05-15 16:02:49

#49 Re: mORMot 1 » SQLite memory usage » 2021-05-15 15:54:56

#50 mORMot 1 » SQLite memory usage » 2021-05-15 15:35:36

Board footer