Posts by okoba

okoba · mORMot 2

Hi,

Does mORMot support ECH (https://blog.cloudflare.com/encrypted-client-hello) or have a plan for it?

okoba · mORMot 1

Thank you very much. Your estimation seems solid.
Yes, a good chunk of the data is updated (hence the need to sync before letting the user know it is done), and most of the other requests would be read and not written (I should have been more clear).

You are right; there is no need to close the connections if the cache size is so low. I could keep them open indefinitely.

Having multiple servers answer write requests seems like the way to go, as if one goes down, the data will not be lost. And in this way, we can use queue and batch, as you said, to speed it up.

okoba · mORMot 1

I assumed PostgreSQL does that but I am not familiar with it much.

There are max estimations:
How many users? 1K/5K (User means separate database)
How many users at the same time? 2K/10K, for each database/user there would be 1 to 3 sub-user.
How many records per user per second? 1/2 per user (2K/10K for all users at max, in a second)
How much data per record? 1KB/8KB

So if each DB/user commits in a 2 second delay, there may be 10K to 20K (at max) records that can get lost if an OS crashes happen.
I was thinking about relying each record from load balancer to two server at the same time, so if one goes down, another have a chance of commiting.

okoba · mORMot 1

Great advice. Yes TRestBatch seems the way to go. I was and am mostly worried of returning success to the user, but in a crash I lose their information, hence thinkin about Full.

I should ask (as you may be one of the best people who used SQLite a lot), compare to PostgreSQL, what is your go to for servers with a lot of concurrent connection, or even for my case, many databases for many users?
I looked at benchmark (extdb-bench) test you did and SQLite is 4 to 5 times (900K/s to 190K/s) faster than PostgreSQL if you use Off and Batch and Transaction, but PostgreSQL is much faster (9.5K/s to 190/s) if I want to use Full without Transaction (separate connections doing one at a time).

okoba · mORMot 1

Thank you very much for the explanation. I will work on it more and see what I can do.
About SQLite smOff, I would lose data in case of OS crash. Any better way?

okoba · mORMot 1

Thank you for the helpful answer. Yes it is in the wrong forum, but if I remember correctly, when I initially opened this topic, Version 2 was in Beta and there was not a dedicated forum for it. If it is possible, please move it.
Those samples were life-saving; without them, I couldn't find out what I should do. Thanks to Thomas and you for them. Martin samples are great too.
I will look forward to the updates, but please create samples for them so a newbie like me knows how to use them. Your codes are great and advanced; please dont assume finding a way in them is walking in a park. They are like being in a big library—too many great things to see and read and get overwhelmed

Anyway, I tried my best to make it work. Can you please check it?
https://gitlab.com/-/snippets/3670020

Questions:
1- I didn't use Interface based (SOA?) because I wanted to use the server with a Browser client. Is that a right assumption?
2- I tried using TRestOrmServerDB , but it asks for a Rest, what should I give it? If TRestServerDB, then what was the reason you said I should use TRestOrmServerDB? I tried passing nil for the Rest parameter, but TRestOrm.InternalAdd raise error because it needs fRest.
3- Is there a better way to handle caching?
4- If I don't set LockingMode, and Synchronous the speed is very low. LockingMode is fine, but setting Synchronous to smOff would be dangerous I guess. Can I ask what would you suggest in production? I can use PostgreSQL too, but I guessed SQLite would be faster and less hassle.
5- What is the best way to handle the locking problem when multiple connection come and want to use one TRestServerDB/DB?

okoba · mORMot 1

I am puzzled by the name of the classes.
What I understand is:
TRestServerDB is a REST server that has a DB.
TRestOrmServerDB should be a REST server that has DB and ORM support, but it is the ORM of TRestServerDB, right?

And when should I use TRestServerDB or TRestOrmServerDB? I cannot understand their usage, even after reading the readme files.

And what is the place that I should choose the TRestXDB if I am going to have multiple one per user? In OnRequest of HTTPServer?
I know mORMot2 should be easier compared to 1, but I still get lost.

okoba · mORMot 1

After a while I am wondering around in the ORM part of mORMot (always funny that you call it little) and I looking for my way to implement that routing to multiple database files. So I guessed asking, maybe you did something in past years.

okoba · mORMot 1

After a couple of years, I came back to this case again. Last time the project got canceled.
I am curious, @ab, did you make the cache mechanism or similar?

okoba · mORMot 2

Honestly I expected even more powerful one. mORMot results are fantastic.
Did you configure the server machine for such high performance?
Thanks for the commit.

okoba · mORMot 2

Hi,

In httpServerRaw project, there are sample numbers of a server. Can I ask what was the hardware for that? or maybe including them in that comment for future?

okoba · mORMot 2

Thank you!

okoba · mORMot 2

I am trying the blog post sample and I can not find First() function in the sample code.

okoba · mORMot 1

Ok good to here that. I tried two computers, one a x86 VM and one M1. Both installed Trunk for x86 with fpcupdeluxe and that was it. I didn't do anything in particular.
My current version is Lazarus 3.99 (rev main_3_99-1261-gb2b86ebaef) FPC 3.3.1 x86_64-darwin-cocoa
Lazarus on macOS is not the best in user experience, but was your problem compiler issues?

okoba · mORMot 1

Hello,

I tried a new macOS machine with latest mORMot, and FPC and compiling the packge does not work with error like: Linker: cmpl %rdx,-12(%rax)
In summery, I can not compile mORMot.
Does anyone used it recently with mac and any note I should know?

okoba · mORMot 1

Oh believe me, compare to me, you are a god of C
I will look into it.

okoba · mORMot 1

Yes I benchmarked all compression library available in mORMot and ZSTD was faster and had better compression. libdeflate is great, but ZSTD can do better.
Although I agree, libdeflate is the good way especially for server needs. But for custom client and server codes, ZSTD can be the better choice.
I could take care of adding ZSTD to mORMot 2 in the current TAlgoCompress style, but the problem is mostly managing to prepare the best static libraries (including custom emmory manger like you done for SQLite and others, or shared C Lib codes). I don't know C and building their libraries is just a pain.
If thats possible for you to prepare that, I can write the code for the current official dll.

okoba · mORMot 1

Currently mORMot supports many compression algorithm, and I wanted to ask if there is a plan for ZSTD support?

okoba · mORMot 1

@mpv I wanted to say thank you and I am following the work.

okoba · Source Code repository

Hi,

I am curious about the state of working with Fossil and Git.
Looking at mORMot and mORMot2 repositories, it seems ab uses Fossil for V1 and GitHub for V2.
Can I ask, especially from ab, how did you find working with Fossil and syncing with Git? I like to work with Fossil, but the world uses Git and GitHub, so if I even do that, I probably need a way to sync them.
I want to know, from your experience, is it worth it?

okoba · Low level and performance

Here is a post that may be interesting to check and compare with mORMot: https://www.corsix.org/content/fast-crc32c-4k
And here is another one for Apple M1: https://dougallj.wordpress.com/2022/05/ … -apple-m1/

okoba · mORMot 1

Btw I’m still amazed at how you made faster encryption code then OpenSSL. It is not a simple code to even understand, let alone make it this fast.
Well done!

okoba · mORMot 1

Great! Thank you. I suggest add them to the encryption code of SQLite unit for future. It is informative. Also adding a line for how to prepare the JSON structure is good. I can provide a patch if you see fit.

So th patch is needed mostly for WAL and adding PRAGMA, other features seems not very needed.

Why did you said the AES256 is not needed password-derivated key? I couldn’t find a reference for it?
When is suggest to use OFB or CTR?
Exactly what bytes are not encrypted? From 0 to 16 or more for page size?
Can we have them encrypted to make the file fully noise like? It will be useful when it is not needed to be used by other tools other than severs, and also storing backups of the database without leaking any information.

okoba · mORMot 1

As one who read it and know what it does exactly, can you explain how this way of encryption works?
I guess every page is encrypted when written. But a little more on how every page is encrypted in relation to the neighbors pages or HMAC storage. Also why the first bytes of the file (SQLite format) is not encrypted.
In summery I like to know more on how the encryption of SQLite is implemented without going in depth.

okoba · mORMot 1

@ab can I ask why did you use SQLite3MultipleCiphers VFS approach instead of adding VFS in runtime in using SQLite API? I mean why patching SQLite when you write most of the encryption anyway.
Could you add a VFS in Pascal like the sqlite3mc_vfs?

okoba · mORMot 1

Thank you very much. I need to test your suggestion and will let you know if the cache is needed.

okoba · mORMot 1

@ab can you help on this?

okoba · mORMot 1

@JD thank you, although I need to have independent file for databases.
@ab thank for the notes. I want to do it with SQLite if possible. Can you elaborate on how can I route the class to different DBs while using interface based?
I was thinking about a virtual table but maybe it is over engineering? I guessed maybe there is something better in mORMot.
PS, I’m using mORMot 2 for this new project.

okoba · mORMot 1

Hello,

I have multiple databases for each customer (need to be independent databases for legal reasons), with the same structure or model. And I like to use interface methods to access them. The API should have independent methods called by user, so no direct access to database tables.
What is the best way to rout incoming requests to multiple databases?

To be clear, what is the best way to rout like:
User1 calls /api/logger/add?data=XYZ&key=user1_key
Server checks user1_key, finds that the data need to be added to user1.db and like to insert to that file.

Perhaps, in the future, the Server need to call another server (User1_Server) to do the actual insert. I think this part can be done by the master/slave architecture described in the documentation, but I appreciate any needed info.

okoba · mORMot 1

It was a copy and paste issue, fixed it.
Is the style of the solution correct? Registering the class and using ParseNewInstance?

okoba · mORMot 1

Here is my response to my question. I do not know if this is the best option though.

procedure TItemReader.ReadItem(var Context: TJsonParserContext; Data: pointer);   
var
  S: PUTF8Char;
  SP: RawUTF8;
  L: integer;
  C: TItemClass;
  Instance: TObject;   
begin
  Result := nil;
  P := Context.Json;
  OldP := P;
  P := JsonObjectItem(P, 'CustomClass');
  if P = nil then
    raise Exception.Create('Invalid Item');
  JSONRetrieveStringField(P, S, L, False);
  SetString(SP, S, L);
  case SP of
    'ItemA': C := TItemA;
    'ItemB': C := TItemB;
    else
      raise Exception.Create('Unknown Item: ' + SP);
  end;
  Context.Info := Rtti.RegisterClass(C);
  Instance := TRttiJson(Context.Info).ParseNewInstance(Context);           
end;

And register it like this:

TRttiJson.RegisterCustomSerializer(TypeInfo(TItem), @ReadItem, nil);

okoba · mORMot 1

I had a code with mORMot1 like:

  TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TItems), @ReadItem, nil);

and

 
function TItemReader.ReadItem(P: PUTF8Char; var AValue; out AValid: boolean): PUTF8Char;
var
  V: TObject absolute AValue;
  OldP, S: PUTF8Char;
  SP: RawUTF8;
  L: integer;
begin
  AValid := False;
  Result := nil;
  if P = nil then
    Exit;
  OldP := P;
  P := JsonObjectItem(P, 'CustomClass');
  if P = nil then
    raise Exception.Create('Invalid Item');
  JSONRetrieveStringField(P, S, L, False);
  SetString(SP, S, L);
  case SP of
    'ItemA': V := TItemA.Create;
    'ItemB': V := TItemB.Create;
    else
      raise Exception.Create('Unknown Item: ' + SP);
  end;
  AValid := True;
  Result := JSONToObject(AValue, OldP, AValid, nil, JSONTOOBJECT_TOLERANTOPTIONS);
end;

and now with mORMot2, I can not convert it.
I thought I should do this but it leads to unknown errors.

 
function TItemReader.ReadItem(P: PUTF8Char; var AValue; out AValid: boolean): PUTF8Char;
var
  V: TObject absolute AValue;
  OldP, S: PUTF8Char;
  SP: RawUTF8;
  L: integer;
begin
  AValid := False;
  Result := nil;
  P := Context.Json;
  OldP := P;
  P := JsonObjectItem(P, 'CustomClass');
  if P = nil then
    raise Exception.Create('Invalid Item');
  JSONRetrieveStringField(P, S, L, False);
  SetString(SP, S, L);
  case SP of
    'ItemA': V := TItemA.Create;
    'ItemB': V := TItemB.Create;
    else
      raise Exception.Create('Unknown Item: ' + SP);
  end;
  AValid := True;
  Context.Json := JSONToObject(Instance, Context.Json, Context.Valid, nil, JSONTOOBJECT_TOLERANTOPTIONS);     
end;

What is the correct way to unserializing and array of classes?
Note: I can not use ParseNewObject, as the CustomClass property may not be the first properties and some other custom needs.

okoba · mORMot 1

Now it is working. Thank you.

okoba · mORMot 1

Just checked again,
The error can be reproduced with with either the latest mORMot2 or sqlite revision 3.37.2 and FPC trunk of now.
Most mORMot tests fails, it seems a memory problem.
One interesting thing for me is that, why such an error is not detected by some catch and can mess around with other parts of code, even the parts that seem unrelated.
I was suspected of patching RTL, but activating NOPATCHRTL, makes everything worse.

okoba · mORMot 1

I was using an old Trunk with mORMot2 from two weeks ago (sqlite revision 3.37.2 172d789404d400ed60447a2b757c5ac770906ae9) and everything was fine, but I pulled mORMot2 as habit and memory crash ...
So I updated my Trunk and now the problem I reported.
Unfortunately I'm stuck with Trunk (3.3.1) for generics and I cannot go back.

okoba · mORMot 1

I recently pulled mORMot2 and ran into some exceptions when setting FPC's internal string methods.I updated FPC and Lazarus to Trunk from Git and now it runs, but my SQLite Vtab codes are not working or buggy.
Just adding a useless log code like WriteLn(RawUtf8(argv^[0])); after begin of xConnect makes it work.
It solves the issue in some projects but not in others. Note that the RawUtf8 cast is the solving part, as reading the argv or WriteLn on their own does not change anything.
It is a weird case, so I'd like to know what the best version to try.

I am working on Windows 64bit and the stable version of FPC is working with the latest mORMot2, but I cannot use the stable version as the Generics code is lagging behind.
I'd appreciate any suggestions.

okoba · mORMot 1

Very interesting article.
I like to know more about how you debug these problems, like the TAsyncServer case.

okoba · mORMot 1

Thank you!

okoba · mORMot 1

Hi,

TSynQueue.Count method uses ReadOnlyLock, and it makes problems when the Queue is shared between threads that are using the count while others are reading from or writing into it.
It was a weird error to find but in some cases it trough exceptions and maybe it is a good idea to use ReadWriteLock instead in Count. In my case it solved the issue like this:

   with Queue do
        try
          Safe.ReadWriteLock;
          QueueCount := Queue.Count;
        finally
          Safe.ReadWriteUnLock;
        end;

okoba · mORMot 1

I found one of the PDF that describes the CRC code (and has the code itself) but not the exact files that provides the exact binary.
Do you think it is a good idea to have the URL or the codes as reference?

okoba · mORMot 1

Hi,

I wanted to check and compile static files, and as noted there are notes and scripts for many of them like libquickjs. But I could not find the code for some files like r crc32c64.obj or sha512-x64sse4.obj.
Are those available ?

okoba · Low level and performance

Thank you for the clarification. Interesting as always.

okoba · Low level and performance

I noticed that when I include mormot.core.data, it makes comparing string (A=B) fast. Checking that subject I found that mormot.core.rtti patches the fpc_ansistr_compare and it is very fast, excellent.
The question for me is that why mORMot do not use this functions for other parts (eg DynArray) and uses StrComp instead?

   procedure Test;
  var
    I, C: Int64;
    A, B: String;
    T: TLocalPrecisionTimer;
  begin
    A := 'Hello World';
    UniqueString(A);
    B := 'Hello World';
    UniqueString(B);
    C := 0;
    T := TLocalPrecisionTimer.CreateAndStart;
    for I := 1 to 50 * 1000 * 1000 do
      if A = B then //_ansistr_compare_equal //95ms
        //if SortDynArrayString(A, B) = 0 then
        //if StrComp(PChar(A), PChar(B)) = 0 then //367ms
        C += 1;
    Writeln(C, ' ', T.Stop);
  end;

okoba · Low level and performance

Thank you for the updates.

okoba · Low level and performance

Great!
Yes mORMot is the very fast and a blog post as an update to previous ones seems nice. May I suggest having an independent demo that clearly shows the benefits and speed? It helps to answer questions like this topic.
Also it may be a good idea to add JsonTools (from this topic) too as it is very clean and seems fast.
Topic: https://forum.lazarus.freepascal.org/in … opic=46533

PS, can you please update the TDynArrayHasher compile issue with trunk? It helps someone like me to keep up with your fast updates while maintaining daily codes.

okoba · Low level and performance

Thanks for the update. Checked it and it works near 20GB/s! Great!

About TDynArrayHasher, it still does not work for FPC even with the new parenthesis, it seems an @ is needed.
I tried to minimize the problem in a new project and included mORMot define too, it compiles correctly, the problem is only happening in the unit. Sorry for the trouble, but I think it worth to be able to compile with the latest version of FPC.

okoba · Low level and performance

Thanks for the update.
About Trunk, it let me try mORMot code with latest FPC updates as I use mORMot a lot for daily tasks like array, dictionary, Unicode, file and json. And until the latest update to the TDynArrayHasher it worked just fine and passed all the test so it may be a good idea to keep it running and testing latest things.
It is the case for V2, I agree with you on V1 of mORMot as it was always problematic to use it with Trunk. But your updates to V2 made it very compatible and comfortable to use.

okoba · Low level and performance

Very nice.
But I can not compile it with the latest Trunk FPC on Win10 X64. IsValidUtf8Avx2 crashes.
mORMot test fail too on this function.
By disabling the AVX version, Pas version works 3GB/s for me on i9.
Unrelated note: In TDynArrayHasher.FindOrNew, checking Assigned(Compare) fails on FPC trunk. Previous versions work without problem.

okoba · Low level and performance

I think you may get better results trying to rewrite it and prevent dependency.

okoba · Low level and performance

The validate_utf8 function is light and needs no memory using SIMD. More info: https://lemire.me/blog/2018/05/16/valid … -per-byte/
But as far as JSON needs, mORMot SAX approach uses no memory, but simdjson builds a DOM-like info that takes much (near 5X I think) more memory.

mORMot Open Source

#1 mORMot 2 » ECH support » 2024-02-28 11:18:05

#2 Re: mORMot 1 » Routing requests to multiple database » 2024-02-08 09:58:36

#3 Re: mORMot 1 » Routing requests to multiple database » 2024-02-07 15:08:14

#4 Re: mORMot 1 » Routing requests to multiple database » 2024-02-07 10:55:55

#5 Re: mORMot 1 » Routing requests to multiple database » 2024-02-07 09:38:07

#6 Re: mORMot 1 » Routing requests to multiple database » 2024-02-06 20:27:35

#7 Re: mORMot 1 » Routing requests to multiple database » 2024-02-06 14:22:17

#8 Re: mORMot 1 » Routing requests to multiple database » 2024-02-05 19:04:17

#9 Re: mORMot 1 » Routing requests to multiple database » 2024-02-05 18:43:04

#10 Re: mORMot 2 » httpServerRaw test machine » 2024-02-05 18:37:45

#11 mORMot 2 » httpServerRaw test machine » 2024-02-05 15:46:26

#12 Re: mORMot 2 » IDocList/IDocDict JSON for Delphi and FPC » 2024-02-03 10:19:28

#13 Re: mORMot 2 » IDocList/IDocDict JSON for Delphi and FPC » 2024-02-03 07:06:32

#14 Re: mORMot 1 » mORMot 2, macOS 14 and FPC Trunk » 2023-12-21 07:44:03

#15 mORMot 1 » mORMot 2, macOS 14 and FPC Trunk » 2023-12-20 15:03:21

#16 Re: mORMot 1 » ZSTD support » 2023-09-12 12:40:23

#17 Re: mORMot 1 » ZSTD support » 2023-09-12 12:06:59

#18 mORMot 1 » ZSTD support » 2023-09-12 10:36:28

#19 Re: mORMot 1 » High-performance frameworks » 2022-11-07 06:50:34

#20 Source Code repository » Source control » 2022-10-15 08:17:26

#21 Low level and performance » Faster CRC32 on x86 and M1 » 2022-08-01 14:16:23

#22 Re: mORMot 1 » 10.4 sydney issues » 2022-04-20 21:46:30

#23 Re: mORMot 1 » 10.4 sydney issues » 2022-04-20 19:51:42

#24 Re: mORMot 1 » 10.4 sydney issues » 2022-04-20 15:57:03

#25 Re: mORMot 1 » 10.4 sydney issues » 2022-04-20 13:21:57

#26 Re: mORMot 1 » Routing requests to multiple database » 2022-04-17 10:49:31

#27 Re: mORMot 1 » Routing requests to multiple database » 2022-04-17 05:33:00

#28 Re: mORMot 1 » Routing requests to multiple database » 2022-04-07 19:50:42

#29 mORMot 1 » Routing requests to multiple database » 2022-04-07 16:34:45

#30 Re: mORMot 1 » Unserializing custom array of classes with mORMot2 » 2022-02-19 20:58:41

#31 Re: mORMot 1 » Unserializing custom array of classes with mORMot2 » 2022-02-19 16:20:41

#32 mORMot 1 » Unserializing custom array of classes with mORMot2 » 2022-02-17 13:40:25

#33 Re: mORMot 1 » mORMot2 with FPC Trunk » 2022-02-14 09:58:44

#34 Re: mORMot 1 » mORMot2 with FPC Trunk » 2022-02-13 21:12:00

#35 Re: mORMot 1 » mORMot2 with FPC Trunk » 2022-02-13 20:18:38

#36 mORMot 1 » mORMot2 with FPC Trunk » 2022-02-13 14:34:57

#37 Re: mORMot 1 » Three Locks To Rule Them All » 2022-01-24 08:52:25

#38 Re: mORMot 1 » TSynQueue.Count » 2022-01-22 08:13:18

#39 mORMot 1 » TSynQueue.Count » 2022-01-20 11:37:42

#40 Re: mORMot 1 » Compile from source of static files » 2021-08-11 07:23:02

#41 mORMot 1 » Compile from source of static files » 2021-08-10 16:58:23

#42 Re: Low level and performance » Fast compare string » 2021-08-09 08:33:30

#43 Low level and performance » Fast compare string » 2021-08-08 16:49:48

#44 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-28 11:19:21

#45 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-27 20:21:45

#46 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-27 08:51:19

#47 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-26 19:36:22

#48 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-26 14:45:48

#49 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-26 07:43:58

#50 Re: Low level and performance » Fast (very fast) IsValidUTF8 implementation » 2021-07-25 08:09:55

Board footer