Async Insert/Updates

chapa · 2014-09-16 10:54:50

Hi ab,
Is there any build in functionality or plan to do async/postponed/cached CRUD operations against mORMot server?

ab · 2014-09-16 11:02:58

Is it async AND postponed AND cached ?
Is it async OR postponed OR cached ?
What do you call "postponed"?

You can write asynchronous code using a dedicated thread.
But we won't recommend it on client side.
You can write a service which creates its own thread on the server, and do some background task.

For multiple commands sending from the client side, see the Batch*() methods.

If you find that your client application is not reactive, you can already let the client execute all process in a background thread, and still be reactive.
See http://blog.synopse.info/post/2013/07/0 … responsive

cheemeng · 2014-09-16 11:06:50

If you're using the built-in ORM, caching happens automatically (see Section 5.5 of the SAD pdf file) on the server side.

Not sure what you mean by "postponed" operations.

EDIT: Oops, looks like Arnaud replied first

Last edited by cheemeng (2014-09-16 11:08:31)

chapa · 2014-09-16 13:19:26

Indeed, I was not clear enough.
I am aware of caching mechanism of mORMot and multithreading.

Take into consideration following use case.

We have one writer and many readers. Writer try to pass real-time data source, which changes really often (50000 times per second), and latest (most up to date) state need to be able to be read from clients with as less latency.
Only recent state should be kept in the DB.

Writer can insert/update/delete at high rate. All record states can be cached for later sync with DB.
In this case 1000 updates/sec on one record can later touch DB for update only 1-2 times, with most recent state for sync.
Meantime, upon client request, server will return either from DB/internalcache, or most recent record for sync, which is not already persisted with DB.

The purpose and final goal is, with compliments to DB's "DiskSync=Off" or common DB fault tolerant mechanisms, we will have super fast in-memory sync&flush structures and possibility to update/insert and query records at really fast rate, to reflect real-time data, and only most recent record (or even only modified properties of records (column)) to be synced (flushed to db) at later time (after period of time, after number of changes, etc).

Of course this may be set up only for one table, where we can afford data loss of last record states for sync in case of disaster. They may even flush to binary file upon shutdown and later come up into the memory if server restart occur (like mORMot's Memory Tables)

All above can of course be done with custom in-memory data structures without overhead of mORMot, but it will be custom solution. Much better to have REST and mORMot over it.
Such a functionality will be Redis killer for Delphi programmers.

I know there is TSQLRestStorageInMemory and similar, which are fast, but are still not a real DB.
As one may have millions of records which are synced into a DB, and at a time only 100 records are "live" and updated at high rate.
Currently, can not load millions records into TSQLRestStorageInMemory nor update DB backed storage at high rate.
Maybe TSQLRestStorageInMemory which syncs with corresponding RestStorage and some manager in front of them will do the trick.

Such implementation may later be used for replication.

Hope it is more clear now

ab · 2014-09-16 13:30:33

Some kind of "write cache" / "delayed write" on the server side?

If I understand well, that is:
- A limited set of records are modified in the same time;
- This set of records can be overwritten/modified very often during the same time, so updating the DB is a bottleneck;
- Only send the modifications once in a while to the DB, e.g. once the record set content has been "stabilized".

What are the use case of this?
Perhaps something like user sessions statistics: they change often during the session, then once the session is closed, they would not change any more, and could be written to DB.
Could be used e.g. for http://synopse.info/fossil/tktview?name=a392945901

Is that what you mean?

chapa · 2014-09-16 14:25:19

hi ab,

You understand me well, only I should expand it a bit at "Only send the modifications once in a while to the DB, e.g. once the record set content has been "stabilized"" it can be an option when to sync when update the record.

Update the record and:
1. Immediate sync
2. Schedule sync after X milliseconds (on every update it will reschedule it, but sync if timeout) (this may break current design principles, if background thread manage it)
3. Background thread doing sync regarding above 2 settings if DB backend is idle and/or memory limit reached. On batch portions. (this definitely breaks current design principles)

Use cases may be:
1. any Forex trading systems, which will keep track of any rate change, keep change history, but without hitting the DB for every change, meantime all clients will have up to date rates which is critical for such systems.
2. Real-time network dump statistic system is also candidate for it. (ex. You may miss exact total bytes sent/received, but you do not have to hit the DB for every count network packet, or to log every packet size in order to later sum it).
3.Any other real-time processing and reporting system.
Such system may process tens of thousands updates per second for a set of records.

My use case is different, but I guess much more complicated. I do scan/parse thousand web sites, extract "rates" and process them. This is done in 1-3 seconds interval for each site and extracted entities (records) may easy hit 50000/second.
Persisted change history is essential (change miss not critical thought), along with client to have real-time up to date information.

I have custom solution over mORMot, and it will be really really good if mORMot have such general purpose functionality on the fly.

ab · 2014-09-16 16:07:12

We may implement a "proxy" TSQLRestStorageDelayed class for a given table, which would make the write buffering, and updates to the "database" TSQLRestStorage in a background thread (using BATCH process for speed).
Something similar to TSQLRestStorageRemote, with its own temporary storage for the buffering.

It could be handy for a small set of updates, but also for any kind of table, BTW.
It could be used with a TSQLRestStorageRemote class for the "database" storage to buffer outgoing content in a client-server architecture with one main database on the corporate head office, and several local sub-servers in each local offices.

ab · 2015-01-08 08:19:18

I've added a feature request for Asynch writes.
See http://synopse.info/fossil/info/cac2e379f02ea

ab · 2015-03-30 12:40:59

A new SQLRestTempStorage "asynchronous write" class has been committed.
See http://synopse.info/fossil/info/d03288afe0

Still need more testing, but a good starting point for this feature request.

mORMot Open Source

#1 2014-09-16 10:54:50

Async Insert/Updates

#2 2014-09-16 11:02:58

Re: Async Insert/Updates

#3 2014-09-16 11:06:50

Re: Async Insert/Updates

#4 2014-09-16 13:19:26

Re: Async Insert/Updates

#5 2014-09-16 13:30:33

Re: Async Insert/Updates

#6 2014-09-16 14:25:19

Re: Async Insert/Updates

#7 2014-09-16 16:07:12

Re: Async Insert/Updates

#8 2015-01-08 08:19:18

Re: Async Insert/Updates

#9 2015-03-30 12:40:59

Re: Async Insert/Updates

Board footer