mORMot Open Source

Rather bad idea. We have many low level details and latest stable FPC is not fully functionally with mORMot. After all NewPascal is practically just special selected FPC trunk version (FPC trunk has Management Operators too). Generally:

NewPascal means = selected Lazarus Trunk + selected FPC trunk (with features preview) + selected mORMot version.

AOG is right. Usage of pure FPC trunk for mORMot has no sense and is extremely risky. I have special version of mORMot which is *almost* ready and compatible both with current FPC trunk and with next NewPascal release:

https://github.com/maciej-izak/mORMot/tree/fpc_newrtti

The branch of mORMot is incomplete yet (we have some serious bug around - and ready to use by FPC core team related patch):

http://bugs.freepascal.org/view.php?id=31249
http://bugs.freepascal.org/view.php?id=31305

#31249 will be fixed directly in NewPascal (or even in FPC trunk).

I know that you are not happy about that but IMO is impossible to put for each change new FPC_HAS_* ... I do what I can best for improved RTTI. Finally we have small steps in right direction.

I mean moving units to directories. We should for example move

to "Common" directory or something similar.

Note: Zeos is excellent example of good designed packages for Lazarus.

IIRC for packages we need to refresh mORMot source code tree and that is main problem...

@ab this code is not used in presented "batch save/load system" (just too much copy/paste from my helper module). Btw. It works perfectly on all of my FPC targets and is very standard for Pascal: ARM, Linux/Windows... without problems.

@edwinsn you have string per operation: Add has own string, Delete  has own string, Update  has own string. The best idea is to store this "batch" strings in separate files (or in single SQLite file as records):

FileFromString(TBatchCacheFactory.Add(...), 'C:\add001.txt');
FileFromString(TBatchCacheFactory.Update(...), 'C:\update001.txt');
FileFromString(TBatchCacheFactory.Delete(...), 'C:\delete001.txt');

...

myDb.BatchSend(nil, StringFromFile('C:\add001.txt'));
myDb.BatchSend(nil, StringFromFile('C:\update001.txt'));
myDb.BatchSend(nil, StringFromFile('C:\delete001.txt'));

you can also try to use somehow single text file mORMot is super elastic.

@ab any progress ? Seems like OAuth2 work takes more than one weekend ;)

@ab: TSynUniqueIdentifierGenerator might be good for new projects but this solution is not perfect (for example is impossible to use TSynUniqueIdentifierGenerator for large existing systems).

@oz: I have 1 Master, N Slaves for described scenario (but in background I have much more complicated architecture with N Masters but that is managed outside mORMot by other tools). The idea for "1 Master - N Slaves" is simple and can be improved by ORM system. What we have:

* android clients (with local SQLite3 DB)
* 1 server mORMot server
* 1 PostgreSQL DB
* Existing big project developed for more than 10 years with 500+ tables (so I can't start from scratch - structures like TSynUniqueIdentifierGenerator are useless for me).

The first step is model:

  TSQLmaster = class(TSQLRecord)
    // ...
  end;
  TCACHE_master = class(TSQLmaster) // could be generated by ORM
  protected 
    fiid: RawUTF8;
    fcreation_time: TDateTime;
  published  
    property iid: RawUTF8 index 38 read fiid write fiid;
    property creation_time: TDateTime read fcreation_time write fcreation_time;
  end; 

  TSQLdetail = class(TSQLRecord)
    // ...
  end;
  TCACHE_detail = class(TSQLdetail) // could be generated by ORM
  protected 
    fiid: RawUTF8;
    fcreation_time: TDateTime;
  published  
    property iid: RawUTF8 index 38 read fiid write fiid;
    property creation_time: TDateTime read fcreation_time write fcreation_time;
  end;
  TCACHE_CACHE_detail = class(TCACHE_detail); // could be generated by ORM

In presented model we have 5 tables: master, CACHE_master, detail, CACHE_detail, CACHE_CACHE_detail
How it is handled by client? For the client, insertion for new records into master/detail table is forbidden. We have simple schema:

* insert for new detail record for correlated master record means hidden insertion into CACHE_detail table
* insert for new master record means hidden insertion into CACHE_master table
* insert for new detail record for correlated CACHE_master record means hidden insertion into CACHE_CACHE_detail table

Each insert above means also assigning for iid and creation_time:

  newrecord.iid := GUIDToRawUTF8(RandomGUID);
  newrecord.creation_time := DBServerDateTime;  

thanks this solution I can receive new data from server without problem and I don't need to send records from CACHE_* tables in the same time (but that scenario is also possible .

When client like to send data (or few parts) I have special record for that purpose:

  TSYNC_CACHE = class(TSQLRecord)  
  protected 
    fiid: RawUTF8;
    fcreation_time: TDateTime;
    { ... 
      structures (for example dynamic arrays) to handle CACHE_* tables and batching 
    }
  published  
    property iid: RawUTF8 index 38 read fiid write fiid;
    property creation_time: TDateTime read fcreation_time write fcreation_time;
    { ...
      structures (for example dynamic arrays) to handle CACHE_* tables and batching }
    }
  end;

on server side I have local SQLite3 file DB to track all iid which is the key for proper working, in some cases mORMot can send twice (or even more) the same data, or even corrupted data but this is different topic. Anyway TSYNC_CACHE and each item inside needs iid. When server has done all insertions described in TSYNC_CACHE (with insertions into iid table) then server sends back TSYNC_CACHE to client but with new correct ID and optionally with new values for some columns for each record inside TSYNC_CACHE.

Also very important for proper ID is right configuration for regular tables on Server side:

  procedure AddEngineComputeId(AClass: TSQLRecordClass);
  begin
    LRestStorage := TSQLRestStorageExternal.Instance(AClass, AServer);
    LRestStorage.EngineAddForcedID := 0;
    LRestStorage.OnEngineAddComputeID := aRestServer.OnEngineAddComputeID; // here inside OnEngineAddComputeID is called nextval for postgreSQL sequence
  end;

  procedure AddEngineIgnoreId(AClass: TSQLRecordClass); // for simple insert
  begin
    LRestStorage := TSQLRestStorageExternal.Instance(AClass, AServer);
    LRestStorage.EngineAddForcedID := 1;
  end;

thanks to mORMot and metaclasses, manipulations between CACHE_* and original tables on client/server side is easy. All described above can be handled on ORM level in very elastic way with many policies. The topic is complex and is very hard to decribe many aspects in one post. Thanks to CACHE_* tables, batching and iid table (and mORMot of course) I have done full CRUD + master/slave replication + online/offline modes

@AOG:

GClientPing is a global variable and is created together with regular clients (at application startup). I need that because  TSQLHttpClientWinSock is not thread safe. I have few threads in application:

1. GUI
2. Data sync
3. Ping

About security: I have private WiFi network in each place where my application is used. Additionally is used standard hcSynShaAes for data compression.

About functionality, look at new topic:

http://synopse.info/forum/viewtopic.php?id=3569

@AOG

From my experience the best values for release for production (in WiFi network) are:

const
  DEFAULT_CONNECT_TIMEOUT = 3000;

...

    AClient := TSQLHttpClientWinSock.Create(AIP, APort, GModel,
      SynCrtSock.HTTP_DEFAULT_SENDTIMEOUT, SynCrtSock.HTTP_DEFAULT_RECEIVETIMEOUT, DEFAULT_CONNECT_TIMEOUT);

In addition if connection is successful I have dedicated thread (and client GClientPing - created with the same parameters as presented above) for ping purposes (online/offline status):

const
  PING_MS_DELTA = DEFAULT_CONNECT_TIMEOUT;  

...

class function TSyncProcess.TSync.Ping: Boolean;
var
  LNow, LThen: TDateTime;
  LTryPing: Boolean;
begin
  LNow := TimeLogToDateTime(GClientPing.ServerTimeStamp);
  LThen := TimeLogToDateTime(FLastPingTimeStamp);

  LTryPing := (FLastPingTimeStamp = 0) or (MilliSecondsBetween(LNow, LThen) > PING_MS_DELTA);

  if LTryPing or FLastPingResult then
  begin
    try
      Result := GClientPing.ServerTimeStampSynchronize;
    except
      Result := False;
    end;

    Result := Sync.StatusCodeIsSuccess(GClientPing) and Result;
    FLastPingResult := Result;
    FLastPingTimeStamp := GClientPing.ServerTimeStamp;
  end
  else
    Result := FLastPingResult;
end;  

It may takes months or years.

Amazing! My inner version of NewPascal uses NodeJS for few purposes, but thanks to your work I think is worth to review my code and support this effort

FPC is the most important part. The Delphi licence (after XE2) is evil:

Seems like mORMot breaks that point in many fields, IMO very bad licence for mORMot... The comment from Embarcadero employee is for me very strange and nasty:

Another point for support for FPC: Delphi Linux compiler will be ARC only.

For extended RTTI for FPC we need to push forward NewPascal project.

@mpv presented solution will not work with TNullableFloat. For TNullableFloat better is:

NULL -> null
NaN -> -0.0
Infinity -> 1.7976931348623157e+308 (int64 with first byte 0 and all other 1)
-Infinity -> 5e-324

But maybe someone have better solution for NaN ?

Maybe for NaN: -0.0e-0 or -0.000e-0

Looks much better -,-

Can we include that into NewPascalPack Continous Integration system? Btw. you have full admin rights for newpascal repo inside dathox.

Ready thanks to patch from AOG

newpascalpack 1.0.3:
http://newpasca.org
https://github.com/dathox/newpascalpack/releases

My current "lobby goal" is to convince BaRo for using mORMot for presented Unity-style engine just amazing - game structures and fluent usage of DB backends...

The change for dynamic arrays is right move. The topic is quiet old. See my post (in 2013 my post was +/- ignored...):

http://lists.freepascal.org/fpc-devel/2 … 32586.html

You can try to track how it is done in PascalScript (with my patch for this breaking change):

http://forum.lazarus.freepascal.org/ind … #msg196402
http://bugs.freepascal.org/view.php?id=29230

@ab, @miab3

thanks for the answers. It was problem on my machine + probably old trunk ZEOS 7.2, anyway now all works as expected.