mORMot road-map

ab · 2011-11-30 12:42:55

The current proposals and upcoming design orientation are listed in http://synopse.info/fossil/wiki?name=RoadMap

The current to-do list is:

- implements TSQLRecord.CreateAndFillPrepareMany method;
add a generic per-TSQLRecordClass or per-TSQLRecord caching, at TSQLRest level (with timeout and RESTful safety) - this may be used on both Client and Server side for speed enhancement of immutable objects (like configuration);
- add a generic JSON error message mechanisms in the framework;
- add Interface-based service definition (like WCF), including a per-session workflow (like WF) and a DataSnap-like JSON serialization (see serverfunctionexecutor.js - parameters may be inlined or added as request body with JSON.stringify);
- add TSQLRecordMappedAutoID and TSQLRecordMappedForcedID classes, for mapping and reverse-engineering guidance from existing database content.

ab · 2012-06-08 16:02:08

Some addition, since revision 1.16 did implement the most important of the former to-do list:

- Add 64 bit compilation under Windows (using Delphi XE2 only yet - FPC integration is planned for later);
- Add TSQLRecordMappedAutoID and TSQLRecordMappedForcedID classes, for mapping and reverse-engineering guidance from existing database content - see this blog article and this post for SynDBxplorer about to be the DB mapping help tool;
- Allow TSQLTable content to be modified - just like the BriefCase Model of TClientDataSet. It may then be saved locally (using a packed binary format), or updated remotely (using BatchUpdate) - this shall work from both TSQLRecord.FillPrepare (in reference to one list of TSQLRecord) and TSQLRecord.CreateAndFillPrepareMany (able to update a JOINed query from several TSQLRecord);
- Direct connection of interface-driven services from a rich AJAX client built with Smart Mobile Studio;
- Code-driven mapping of UI components with TSQLRecord published properties: define your form using RAD, then easily connect the field components (TEdit TComboBox TCheckBox TGroupBox TDateTimePicker) and action buttons (a KISS approach to XE2's LiveBindings, using ORM power, and its full extend of filtering and validation);
- Handle UI edition of Dynamic Array fields (i.e. sftBlobDynArray) for standard types (TIntegerDynArray TRawUTF8DynArray) and RegisterCustomJSONSerializer kind of items (from the list of serialized JSON fields);
- Handle UI edition of TSQLRecordMany fields with auto-generated master/detail fields;
- Support content encryption at the mORMot level, allowing safe and fast transfer without the speed overhead and unpleasant signature configuration on both sides;
- Finish direct ODBC driver (SynDBODBC.pas unit), since OleDB is officially deprecated;
- Implement SOA-level transactional safe process, using Event Sourcing and CQRS (aka Command Query Responsibility Segregation) implementation patterns (with associated documentation): Event Sourcing could be implemented at 3 levels - i.e. 1. interface-based service, 2. BATCH (with automatic retrieval of inserted IDs), 3. for TSQLRestServerStatic class (e.g. handling both TSQLRestServerStaticInMemory and TSQLRestServerStaticExternal kind of tables); in fact, those first two levels are needed for proper transaction handling and rollback during SOA process, and the 3rd would allow benefits of Event Sourcing at the ORM level.

I think the Event Sourcing part may be inspiring.
See http://martinfowler.com/eaaDev/EventSourcing.html

Including this concept at several levels does make sense IMHO, and will implement a nice solution for some needed features:
- Add safe transactional process within SOA / DDD application-level and domain-level (including commit/rollback), without the overhead of a cross-DB transaction service;
- Add an event-based mechanism of data replication, at both SOA level and cross-DB level (via the ORM layer);
- Offer a very fast in-memory database engine, with ACID storage (the previous implementation of TSQLRestServerStaticInMemory was slow when writing lot of data, since it is writing all data at once);
- Let a CQRS pattern be introduced easier.

chapa · 2012-06-11 08:59:37

Hi ab,

Event Sourcing is very interesting topic.
Everyone faced the problems and solutions described.

Reading about Event Sourcing I realize I had implemented it in my own custom way (regardless all the theory).
Will be glad to see your approach and realization.

Fast in-memory database engine is the first thing I may use. Due to the missing in-memory indecies of TSQLRestServerStaticInMemory I dropped it.
Also, implementation can benefit using custom thread data(event) management system, build on SRW Locks. Will break legacy OS.
In any way, the speed and scalability of event backend processing may be the key when taking decision whenever to use such approach.

Will be glad to share my ideas, plans and current sources to mORMot framework, if you find them useful you will need to document and provide some good examples.
They are not so tight around the theory, but may help mORMot users in various use cases.

Last edited by chapa (2012-06-11 09:12:18)

ab · 2012-06-11 13:46:59

@chapa Yes your point of view is very useful.
Any feedback is welcome!

TSQLRestServerStaticInMemory is pretty basic: this is just a TList of TSQLRecord instances, in memory.
I may be able to add in-memory indexing, but it needs some work and testing - for instance, we have some known issues with SynBigTable when playing with indexes.

As Martin Fowler stated in his reference article, Event Sourcing is a common approach in some applications.
For instance, it is "natural" to accounting business logic.

We will try to introduce it in order to add a "transactional" KISS approach within the framework.
I wonder if it will work as expected, but I hope yes.

But it may also be of some interest at the DB engine level also.

If you have any idea, especially about interface (methods/classes to be added) and implementation, I'd be happy.

ab · 2012-06-27 11:33:00

I've just updated the roadmap, mainly about the "briefcase model" point:

Update: perhaps not worth it since we can now export SynDB or mORMot results as TClientDataSet: why reinvent the wheel when it is not a perfect match with our ORM and SOA design?

So perhaps not worth it.
In all cases, not on the highest priority any more.
In a stateless model, it does not make much sense to use such an implementation pattern.

Event-Sourcing makes better sense to me as an implementation pattern in such case.

About in-memory databases, using an external SQLite3 in-memory engine (set ':memory:' as filename) is very efficient, and has all needed features, including indexes.
Then you can export all data as once, into a JSON file, or another SQLite3 "regular" file-based database - see http://www.sqlite.org/backup.html but I'll probably add a direct export/import method in our code base.

edwinsn · 2014-06-30 15:39:45

When FPC support is fully implemented, I believe mORMot will bring FPC/Lazurus to the next level of web development, maybe like RoR to Ruby

Junior/RO · 2015-06-29 14:05:15

@ab, what is the current road-map today?

SpanishBoy · 2015-09-11 15:21:55

+, and I saw some plans regarding supporting Delphi Web Script

mpv · 2019-10-29 11:04:03

Just to not forgot in future: implementing OpenTracing API and integrate it into TSynLog
Current LogView program solves almost all questions while analyzing log files produced by TSynLog for a single server instance. But in case of many micro-services we need to centralize a logging somewhere.

Currently I build a "bicycle" using syslogd + ELK stack to extract a small piece of information from logs what critical for my distributed system, but in general will be good to implement a OpenTracing API client (we already have almost all peaces in TSynLog) and use some kind of distributed tracing server for collecting results.
Jaeger is a perfect candidate...

mORMot Open Source

#1 2011-11-30 12:42:55

mORMot road-map

#2 2012-06-08 16:02:08

Re: mORMot road-map

#3 2012-06-11 08:59:37

Re: mORMot road-map

#4 2012-06-11 13:46:59

Re: mORMot road-map

#5 2012-06-27 11:33:00

Re: mORMot road-map

#6 2014-06-30 15:39:45

Re: mORMot road-map

#7 2015-06-29 14:05:15

Re: mORMot road-map

#8 2015-09-11 15:21:55

Re: mORMot road-map

#9 2019-10-29 11:04:03

Re: mORMot road-map

Board footer