mORMot and Open Source friends
View Ticket
Not logged in
2015-04-04
14:02
{1196} introducing RecordVersionSynchronizeMasterStart and RecordVersionSynchronizeSlaveStart/Stop methods for easy master/slave replication of a table for [3453f314d9] - also renamed RESTful synchronization method as RecordVersionSynchronizeSlave, and updated documentation check-in: a33c699430 user: ab tags: trunk
2015-04-03
15:20
{1188} added CurrentFrame() method recognition so that TServiceRecordVersionCallback would be able to process "jumbo frames" incoming modifications within a BATCH, for faster Master/Slave replication over WebSockets [3453f314d9] check-in: 50d534f628 user: ab tags: trunk
09:48
{1187} tested and fixed Master/Slave replication using TRecordVersion field over WebSockets [3453f314d9] check-in: 310bb783b3 user: ab tags: trunk
2015-04-02
19:26
{1186} introducing TServiceRecordVersionCallback class to implement TSQLRestServer.RecordVersionSynchronizeCallback() callbacks, and new regression tests for [3453f314d9], including fixes check-in: 6a4166cd6e user: ab tags: trunk
08:05
{1185} fixes and refactoring of TSQLRestServer.RecordVersionSynchronizeCallback() implementation - see [3453f314d9] check-in: 4462a6f626 user: ab tags: trunk
2015-04-01
21:09
{1184} introducing TSQLRestServer.RecordVersionSynchronizeCallback() method, able to register a callback interface which would be called each time a write operation is performed on a given TSQLRecord with a TRecordVersion field - see [3453f314d9] check-in: e78b7e3e24 user: ab tags: trunk
2015-03-31
20:59
{1168} allow RecordVersionSynchronize*() to work even if both source and destination TSQLModel do not match - see [3453f314d9] check-in: 8190a332aa user: ab tags: trunk
18:34
{1167} several fixes, including support for TRecordVersion up to 58 bits for [3453f314d9] (they were silently truncated to a much lower value by the compiler) check-in: e1d969fcf1 user: ab tags: trunk
18:07
{1166} added TSQLRestServer.RecordVersionSynchronizeToBatch() method which allows to retrieve a list of updates as a TSQLRestBatch, starting from a supplied TRecordVersion - see [3453f314d9] check-in: ccd5ddb087 user: ab tags: trunk
15:29
{1164} introducing TSQLRestServer.RecordVersionSynchronize() method using a TRecordVersion field to maintain a modification versioning information on any table - implement one-way synchronization for feature request [3453f314d9] check-in: ced6e37637 user: ab tags: trunk
06:51
{1158} several fixes for TRecordVersion type support - see [3453f314d9] check-in: 02de4e41d6 user: ab tags: trunk
2015-03-30
18:39
{1157} introducing new TRecordVersion type for TSQLRecord published properties, allowing easy synchronization of row values, using a monotonic version number - first part of [3453f314d9] check-in: d0664ef5bf user: ab tags: trunk
2015-01-16
09:24 Ticket [3453f314d9] Auto-Synch: between mORMot servers (as cache or for branch office cache), and potentially mORMot clients (offline mode) status still Open with 3 other changes artifact: a0be31c72e user: ab
2015-01-08
08:17 Ticket [3453f314d9]: 3 changes artifact: e151a7c410 user: ab
08:16 New ticket [cac2e379f0] Asynch ORM Writes. artifact: 6380738697 user: ab
2014-12-31
10:32 Ticket [cde2d68eb6] Automatic sharding for BigData storage status still Open with 5 other changes artifact: 427b2ea0f5 user: ab
10:32 Ticket [aa230e5299] Implements one-way callbacks from the server status still Open with 5 other changes artifact: ef1b83848c user: ab
10:32 New ticket [17958b22c0] Monitoring feature. artifact: 8a0fb77cc9 user: ab
10:14 Ticket [01da096f72] TSQLRecordMappedAutoID and TSQLRecordMappedForcedID classes status still Open with 4 other changes artifact: 364fdef097 user: ab
09:49 Ticket [3453f314d9] Auto-Synch: between mORMot servers (as cache or for branch office cache), and potentially mORMot clients (offline mode) status still Open with 3 other changes artifact: a6f7b8fb47 user: ab
09:35 New ticket [cde2d68eb6] Automatic sharding for BigData storage. artifact: 509cea8bf0 user: ab
2014-11-14
14:08 Ticket [3453f314d9] Auto-Synch: between mORMot servers (as cache or for branch office cache), and potentially mORMot clients (offline mode) status still Open with 3 other changes artifact: 4d87495c69 user: ab
2014-08-23
07:44 Ticket [3453f314d9]: 5 changes artifact: b4a8783a7c user: root
2014-08-19
09:41
enhanded documentation, mainly about TSQLRestStorageRemote class, TSQLRestServer.RemoteDataCreate() method and TSQLRestServerRemoteDB class for feature request [3453f314d97d] check-in: d9e53fd620 user: User tags: trunk
2014-08-18
13:33
added TSQLRestStorageRemote class and TSQLRestServer.RemoteDataCreate() method and fixed TSQLRestServerRemoteDB class for feature request [3453f314d97d] - only direct ORM methods are available yet: still missing a TSQLVirtualTableRemote class check-in: 796e966c81 user: User tags: trunk
2014-08-17
12:22
some refactoring for feature request [3453f314d]
  • following the Liskov substitution principle, Execute/ExecuteFmt and protected EngineExecute() are defined for TSQLRest, replacing ExecuteAll()
  • TSQLRestServerRemoteDB will now redirect into any TSQLRest instance
  • renamed TSQLRestServerDB.InternalExecute() as explicit StoredProcExecute()
check-in: 4f0c80dc03 user: User tags: trunk
2014-08-15
08:26 New ticket [3453f314d9] Auto-Synch: between mORMot servers (as cache or for branch office cache), and potentially mORMot clients (offline mode). artifact: 8c180da50b user: ab

Ticket Hash: 3453f314d97dd80cebfd13cc50661bc483fdb2c8
Title: Auto-Synch: between mORMot servers (as cache or for branch office cache), and potentially mORMot clients (offline mode)
Status: Open Type: Feature_Request
Severity: Critical Priority: Immediate
Subsystem: mORMot Resolution: Open
Last Modified: 2015-01-16 09:24:46
Version Found In: 1.18
User Comments:
ab added on 2014-08-15 08:26:42:

Following this forum thread discussion, we would like to introduce some kind of "Auto-Synch" beetween mORMot nodes.

Idea is to let some tables be defined to be synchronized.
Each table should therefore contain TModTime and TCreateTime published fields, and an index on the TModTime. It would help to identify which data is to be synchronized at startup (after disconnection).

We may imagine several use cases:

  • Synchronization of several mORMot servers, as asked here, with one master server (e.g. at the main office), and slave servers wired via Internet (e.g. at branch offices), which will serve the local clients;
  • Synchronization of several mORMot servers, to work as a farm, in which a master would be the main database (never accessed directly), and other servers will take the role of proxies, storing their own copy of the data, and able to allow horizontal scalability of the hosting solution;
  • Synchronization of mORMot clients from a server, in which the needed data will be stored on client side, to allow "offline" mode.

Server-side synchronization could be implemented via some dedicated interface-based services, published via a HTTP server.
Client-side synchronization would require a lighter approach, based on publish/subscribe as in [aa230e5299].

Some questions:

  • But how do you manage any break of the Internet link? In the simplest implementation, you may be able to read the data stored in the local slave sever, but you won't be able to push writes to the master and other slaves. The ability to update locally in offline mode, then push the modifications when the link is up again, is feasible but much more difficult to implement (a lot of conflicts may arise).
  • What if we add a WHERE clause to the slaves? For instance, some tables may need to be fully replicated locally (e.g. the customer list), but some other may benefit to be restricted to some local specifications (e.g. the orders which were performed locally). And for the potential offline mode of clients, it would definitively make sense to have only the data of client's interest cached locally.
  • How should we resolve the synchronization conflicts? E.g. if two slaves do update the same record? The easiest is to pipeline all upcoming modifications in the main server, then let the latest received be the latest written... but such an optimistic solution, even if it is not difficult to implement and safe, may not be good enough for real data.

One of my concerns is that we need to setup a good unit test pattern before implementing all this as an "official" mORMot feature.
There may be a lot of problems to handle (e.g. unsynchronized work)...
In short, if everything is OK, such a design is safe and somewhat easy to implement. But in the real life, offline mode is something difficult to work with...


root added on 2014-08-23 07:44:31:

We have added a TSQLRestStorageRemote class and TSQLRestServer.RemoteDataCreate() method.

After a call to TSQLRestServer.RemoteDataCreate(aClass: TSQLRecordClass; aRemoteRest: TSQLRest) method, any ORM operation to the specific class would be redirected to aRemoteRest.

Only direct ORM methods are available yet: still missing a TSQLVirtualTableRemote class, to allow JOINed query between those remote tables. After a call to TModel.VirtualTableRegister(TSQLRecordMyClass,TSQLVirtualTableRemote), you would be able to JOIN some SQL requests over those tables, via the main SQlite3 process of virtual tables.

You may also consider the TSQLRestServerRemoteDB class, which will redirect all tables to a given aRemoteRest. It may allow to host some services at one place, but having all ORM storage executed in another server.

Thanks to TSQLRest caching, you can already create a multi-server configuration.
Please read the "8.3.6. Redirect to an external TSQLRest" paragraphs of the SAD 1.18 pdf.


ab added on 2014-11-14 14:08:08:

See also the discussion in this forum thread.

A lot of input there, especially from real feedback when implementing similar design.

We introduced the new TID = type Int64 definition as our ORM primary key, so that we would be able to compute coherent monotonic system-wide IDs.


ab added on 2014-12-31 09:49:02:

In fact, this synchronization feature, when applied on a void database, would implement an ORM-level replication. We should take care that this process, which may be very resource intensive, would run in chunks, so that it would not block and only slightly slow down the main ORM process.

This replication feature may be used also for proper re-balancing of data sharding, as requested by [cde2d68eb60463].
In practice, we may ensure that the data sharding rule on IDs could be applied during replication, so that only the matching data is part of the new shard.
Similarly, the inverted data sharding rule may be applied on the previous shard, to purge its content from the previous balanced data.


ab added on 2015-01-08 08:17:38:

See also [cac2e379f02e] about "Asynch ORM Writes": this other feature request may be linked to this one.


ab added on 2015-01-16 09:24:46:

See also [292d00675ec] as local storage should be added for (SynCrossPlatform) clients.