mORMot Open Source

dcoun

Member

From: Crete, Greece

Registered: 2020-02-18

Posts: 430

TRestServerDB with TRestStorageInMemory and direct threadsafe access

I have a number of small tables that have value-sets (max ~12000 records each) that they are used by different threads at the same time and their content does not change for long time.
The above tables are used for decoding with simple CRUD queries, but at the same time I need them to be available for other queries with other tables that change all the time.
I created TRestServerDB using Mormot2 with TRestStorageInMemory  for these "static" tables having the following model:

result:=tormmodel.Create([TOrmtest1,TOrmtest12,....]); 
result.VirtualTableRegister(TOrmtest1,TOrmVirtualTableBinary);
result.Props[TOrmtest1].Kind:=ovkCustomAutoID;

I create a TRestServerDB with the above model and I find the TRestStorageInMemory instance with the following:

srvmem:=TRestServerDB.Create(creatememorymodel); 
srvmem.CreateMissingTables;

var directstorage:TRestStorageInMemory;
directstorage:=(srvmem.Orm as TRestOrmServer).StaticVirtualTable[tormtest1] as TRestStorageInMemory;
directstorage.LoadFromJson('from a rawutf8 json');

I noticed that I can issue at the same time sql queries and ORM requests to the TRestServerDB instance and CRUD read requests to the TRestStorageInMemory directly from different threads without problem
To be honest, mormot2 solved a problem I had for years in an application with the above.

My question is, if I want after a period of time to reload new content to such a TRestStorageInMemory instance what should I do to TRestServerDB to flush cache, etc in order to have a proper operation?

Thank you in advance

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: TRestServerDB with TRestStorageInMemory and direct threadsafe access

Congratulations for having found the right way to do it.
You could even make some simple SQL requests to the TRestStorageInMemory via its RetrieveList() method.
Also consider to use binary storage instead of JSON: it may be much more efficient on disk.

If you modify data in the TRestStorageInMemory only, it will indeed be thread-safe.

About the propagation to the main TRestServerDB queries, it depends on the queries:
1) If the main TRestServerDB queries are only about the TRestStorageInMemory table, then TRestServerDB will directly call the TRestStorageInMemory instance with no cache, so no need to flush the TRestServerDB cache.
2) If the main TRestServerDB queries use virtual tables (i.e. you join the TRestStorageInMemory table with other tables), then the SQLite3 Level cache may be involved, and should be flushed before the request.

dcoun

Member

From: Crete, Greece

Registered: 2020-02-18

Posts: 430

Re: TRestServerDB with TRestStorageInMemory and direct threadsafe access

I am using RetrieveList & xxFieldValue(s) and it works very fast
The TRestStorageInMemory tables are only in memory retrieved initially through a database from a rest server. Is binarystorage faster during queries?
Thank you a lot Arnauld for your work

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: TRestServerDB with TRestStorageInMemory and direct threadsafe access

Binary or JSON loading doesn't change anything during queries.

The queries are done within the TOrm array in memory, after unserialization.