mORMot Open Source

Just checked again,
The error can be reproduced with with either the latest mORMot2 or sqlite revision 3.37.2 and FPC trunk of now.
Most mORMot tests fails, it seems a memory problem.
One interesting thing for me is that, why such an error is not detected by some catch and can mess around with other parts of code, even the parts that seem unrelated.
I was suspected of patching RTL, but activating NOPATCHRTL, makes everything worse.

Thank you!

I found one of the PDF that describes the CRC code (and has the code itself) but not the exact files that provides the exact binary.
Do you think it is a good idea to have the URL or the codes as reference?

Thank you for the clarification. Interesting as always.

Thank you for the updates.

Thanks for the update. Checked it and it works near 20GB/s! Great!

About TDynArrayHasher, it still does not work for FPC even with the new parenthesis, it seems an @ is needed.
I tried to minimize the problem in a new project and included mORMot define too, it compiles correctly, the problem is only happening in the unit. Sorry for the trouble, but I think it worth to be able to compile with the latest version of FPC.

Very nice.
But I can not compile it with the latest Trunk FPC on Win10 X64. IsValidUtf8Avx2 crashes.
mORMot test fail too on this function.
By disabling the AVX version, Pas version works 3GB/s for me on i9.
Unrelated note: In TDynArrayHasher.FindOrNew, checking  Assigned(Compare) fails on FPC trunk. Previous versions work without problem.

The validate_utf8 function is light and needs no memory using SIMD. More info: https://lemire.me/blog/2018/05/16/valid … -per-byte/
But as far as JSON needs, mORMot SAX approach uses no memory, but simdjson builds a DOM-like info that takes much (near 5X I think) more memory.

Nothing much, that was my fault anyway.
I made the pull request, please review.

Great. I inherited TSqlite3LibraryStatic with and override the BeforeInitialization with this:

  procedure TCustomSqlite3LibraryStatic.BeforeInitialization;
  begin
    sqlite3.config(SQLITE_CONFIG_MEMSTATUS, Integer(True));
    inherited BeforeInitialization;
  end;         

Now it works.
I will make a pull request for the config comments and show how it must be used to prevent future misunderstandings.

Bare metal, Windows.

Thank you.

I was reporting string dictionary without set capacity, surprised that why the new algorithm is not faster.
This time I did an another test: https://gist.github.com/OkobaPatino/cec … 358984f5c5
These are the result of latest mORMot, latest FPC, Windows and i9.

Remarks:
In increasing string AesNiHash32 is slower, near 25% with SetCapacity and near 50% without SetCapacity.
In random GUID AesNiHash32 is a bit slower.
If dictionary capacity is set beforehand, Add (400%) and Find (25%) is faster, no matter the Hasher.

Questions:
Find is slower when you do not set capacity before hand. Why?
Can I ask where much less collisions come to play with AesNiHash32 as it is always slower than crc32c?

BTW It would be nice to have the ability of choosing the Hasher of the dictionary now that we have more options.

mormot.db.raw.sqlite3 has no initialization.
mormot.db.raw.sqlite3.static has one, at it creates TSqlite3LibraryStatic. In the Create, it calls ForceToUseSharedMemoryManager and I did disable it as a test an no changes.
Am I checking a wrong place?

I thought that is not a huge code as rules stated.
If you mean ForceToUseSharedMemoryManager, I tried disabling that too, but no change.
I like to get status of the DB to have a monitor on the DB, that was one of the reasons I've ported many of these functions previously for the sqlite3 units.

Limiting prevents sudden issues. As for solution, I checked the result for 3M or 1000 bytes and load it after save, it worked correctly. I used this code for months and no problem and passed the checks until I faced this number.
Another question is why JSOn writer has a problem? I did use an external writer with owned file stream and still problem happens in Base64.

It will happen again on:
- Set CompressAlgo to nil
- Saving to JSON
- Using random value for V
I faced this problem in a project with real values with the same exact behavior.

To be clear, I tried any combination of the above in such code and also in the context of my project with real values:

begin
  D := TSynDictionary.Create(TypeInfo(TIntegerDynArray), TypeInfo(TValueDynArray));
  D.CompressAlgo := nil;
  for I := 1 to 2319851 do
  begin
    TAesPrng.Main.FillRandom(@V, SizeOf(V));
    D.Add(I, V);
  end;
  D.SaveToJson;
  WriteLn('Done');
  ReadLn;
end.                           

I answered.

Yes it is.
threadsafe is removed. Is there a reason as you did not remove it from TSqlite3Library?
I will rearrange and cleanup methods.

Thanks ab.
Please let me know when the needed flags are added so we can uncomment deactivated methods and maybe rearrange them in "SQLite natural order". Also some cleanups maybe, see "Notes" on the changelog for more info.

Leslie7, that is an interesting extension. First it needs to be added to the build by ab like other ones.

I've updated the sqlite3 units and added almost all the remaining methods from the API that seems useful.
There are 280 SQLite methods in the API, and we now have 171 (previously 91). Only VFS methods are remaining for now, other ones have a reason to not be included.

Not included ones:
These are the one that are not added and the reason. I will add any of them if you like.
- All the _16 methods as I do not see any use for them (many SQLite functions like sqlite3_bind_parameter_name return UTF8 anyway). The only one already existing is column_text16.
- Most of *64 methods like malloc64, status64, blob64 and text64. I do not see much use for them right now but SQLite says they are preferred.
- Legacy or deprecated methods like sqlite3_get_table
- Some _vX methods that has a more complete one already like sqlite3_trace
- Prohibited methods like sqlite3_os_end
- Deprecated flags like SQLITE_PREPARE_NORMALIZE
- Methods that already have a more complete option (eg sqlite3_result_error_nomem has sqlite3_result_error)
- Systemic methods like sleep or randomness as we have them already in Pascal code.
- String methods as we have them as or more powerful already in Pascal and mORMot like Format (eg sqlite3_stricmp) or can be achieved with a simple query (eg sqlite3_strlike).
  - sqlite3_str_append
  - sqlite3_str_appendall
  - sqlite3_str_appendchar
  - sqlite3_str_appendf
  - sqlite3_str_errcode
  - sqlite3_str_finish
  - sqlite3_str_length
  - sqlite3_str_new
  - sqlite3_str_reset
  - sqlite3_str_value
  - sqlite3_str_vappendf
  - sqlite3_stricmp
  - sqlite3_strlike
  - sqlite3_strnicmp
  - sqlite3_vmprintf
  - sqlite3_vsnprintf
  - sqlite3_mprintf
  - sqlite3_snprintf
  - sqlite3_strglob
- Mutex methods as we have more powerful options already in Pascal and mORMot.
  - sqlite3_mutex_alloc
  - sqlite3_mutex_enter
  - sqlite3_mutex_free
  - sqlite3_mutex_held
  - sqlite3_mutex_leave
  - sqlite3_mutex_notheld
  - sqlite3_mutex_try
  - sqlite3_db_mutex

Todos:
- If you need or like to have some of the "Not included ones".
- Session and RBU and maybe more extension are good to add, but it is better to have them in the build beforehand so we can test and verify. Also maybe a structure to add extensions is a good idea as SQLite has many good options that can be useful.
  - https://www.sqlite.org/rbu.html
  - https://sqlite.org/sessionintro.html
- Missing VFS methods. I did not know how to check or use them.
  - sqlite3_vfs_find
  - sqlite3_vfs_register
  - sqlite3_vfs_unregister
  - sqlite3_free_filename
  - sqlite3_filename_database
  - sqlite3_filename_journal
  - sqlite3_filename_wal
  - sqlite3_win32_set_directory
  - sqlite3_win32_set_directory16
  - sqlite3_win32_set_directory8
  - sqlite3_uri_boolean
  - sqlite3_uri_int64
  - sqlite3_uri_key
  - sqlite3_uri_parameter
  - sqlite3_file_control
  - sqlite3_create_filename
  - sqlite3_database_file_object

Notes:
- sqlite3_status64 is added and it seems there is no need for memory_used and memory_highwater anymore.
- I did not change the order of methods, but it may be a good idea to sort them better to match SQLite. Let me know if you like that.
- prepare_v3 is added, maybe it is a good idea to remove the v2, as v3 does the extract same thing more option.
- libversion_number is added so it is a good idea to change the Create method.
- extended_result_codes and Extended Result Code are added, maybe you like to enable it by default and use it in mORMot.
- create_collation_v2 is added, maybe it is a good idea to remove the v1, as v2 does the extact same thing with an optional destroy.

Finally these are the flags needed to be added to the build. All the missing methods are commented in TSqlite3LibraryStatic.Create, need to uncomment after adding.
- YYTRACKMAXSTACKDEPTH for SQLITE_STATUS_PARSER_STACK
- SQLITE_ENABLE_COLUMN_METADATA for column_database_name, column_table_name and
- column_origin_name
- SQLITE_ENABLE_STMT_SCANSTATUS for stmt_scanstatus and stmt_scanstatus_reset
- SQLITE_ENABLE_PREUPDATE_HOOK for sqlite3_preupdate_*_
- SQLITE_DBCONFIG_ENABLE_LOAD_EXTENSION for sqlite3_*_extension
- SQLITE_ENABLE_SNAPSHOT for sqlite3_snapshot*
- SQLITE_ENABLE_UNLOCK_NOTIFY for sqlite3_unlock_notify
- sqlite3_progress_handler is missed from the build

I would like to know your comments on the Todos and Notes.
And thank you for the clean structure, I tried to follow that and checked most of them.

Here is the result patches: https://gist.github.com/OkobaPatino/23e … 86003a094b

One question, some defined function in the static unit returning PAnsiChar, but you used PUtf8Char most of the time. Is there a particular reason?