mORMot Open Source

It's really fast (i3-6100T):

32bit, Multicore:

FastMM-AVX (only ERMS enabled)  - 18 sec;
BrainMM - 8 sec;

BrainMM should be investigated how good it is on some 24/7 processes, because performance is stellar!

He already tested, but it was a bug in BrainMM and crashed on Win64.

So, any info what compression to use between iOS/Android and mORMot server?
In my iOS/Android app I have SQLite db contaning a subset of main Firebird db and sometimes when sync is needed, a bulk transfer is happening which will benefit from compression.

Thank you.

@cybexr,
In a linked post ( http://synopse.info/forum/viewtopic.php?id=3704 ), @ab says better than I could do it:

If I understand correctly, it will be only for websockets? If yes, that's to bad for me because, I would like to use SOA over websockets, but in current implementation it's only one connection per thread and it's not scalable for thousands of connections.
So, I'll still use "http.sys" http-like connections until so scalable "http.sys" websockets are fully implemented(by the way if you fully implement "http.sys" websockets, you will dust over any framework, including MS ones

Thank you for your wonderful framework and if it's possible please implement a-la SSL encryption also for normal http requests, not only for websocket.

If you want some classic approach this is what I want to test next week:

I'm using SOA - method based

1. Retrieve data as an JSON object (ex: "Order" with "OrderLine") into 2 TDataset-compatible(client side). Server-side, you could use for this ISQLDbRows.FetchAllToJSON for every table and combine the result by using TTextWriter.
2. On those 2 memory-TClientDataset you can have master-detail as you already know.
3. When you post changes back to mORMot server, on client side, scan every of those 2 TClientDataset for changed rows and you build an JSON array of changed records:
ex:

[{"table":"Order",  "op":"update", "version":1, "Data":{"ID":123, "Amount":110} },
 {"table":"OrderLine",  "op":"update", "version":1, "Data":{"ID":124, "Qty":11} },
 {"table":"OrderLine",  "op":"delete", "version":1, "Data":{"ID":125} }] 

For generating on client side the JSON from TClientDtaset you can use TDocVariant or JsonDataObjects or any Json writer.
4. On server-side, for transforming that JSON into SQL phrases use TJSONObjectDecoder record (for Fields "table", "op", "version") and GetJSONObjectAsSQL function (for "Data"; GetJSONObjectAsSQL  uses inside also TJSONObjectDecoder record):
ex:

update Order set Amount=110 where ID=123 and version=1;
update OrderLine set Qty=11 where ID=124 and version=1;
delete from OrderLine where ID=125 and version=1;

Notes:
a) "version" it's used for optimistic locking; you can use a timestamp instead; the basic idea is it prevents updating a row which is already modified by somebody-else since the time it was retrieved by me.
b) this approach is compatible with Elevate Web Builder with the difference that instead of "Data" and "version", they use 2 json-subobjects "newrow" and "oldrow".

I have order on Amazon UK on Sep 11, dispatch on Sep 13 and will be delivered between Sep 15-17.

For europeans best is Amazon UK:
"https://www.amazon.co.uk/Using-Elevate-Web-Builder-Mobile/dp/153555505X/ref=sr_1_2?ie=UTF8&qid=1473771589&sr=8-2&keywords=elevate+web+builder

Any plans to support SynLZ on FMX platform? Sometimes I will need to transfer some large blocks to iOS and Android devices.
Thank you.

Of course if you use variant a) (EWB + mORMot REST server), you can write a little code in EWB which automates marshalling from json GET http(s)://server/Customer(5) to instance of TCustomer (or from any json represented object to instance of that object). EWB knows RTTI.

So you can have your ORM on client side in EWB.

So, you can have something like this:

var
  Customer: TCustomer;
begin
  Customer :=TCustomer.Create;
  try
    GetFromServer(Customer, 5);
    label1.Caption := Customer.Name;
    label2.Caption := Customer.Adress;
  finally
    Customer.Free;
  end;
end;

where GetFromServer is your procedure written by you which calls GET http(s)://server/Customer(5), takes the json and parses Name, Adress,..

@edwinsn

1. With EWB, you can use any "outside" JS library, you have of course a component TScript. All the job is done in Object Pascal but if you want something fancy you have access to JS.
2. For EWB-mORMot relation you don't need an interface. From EWB app if you want data from mORMot you have 2 options:

a) TServerRequest send a query like this GET http(s)://server/Customers(5) , mormot responds a json with Customer with ID=5. Your logic from EWB parses the Json with help of TReader component and takes Name, Adress, FiscalCode, ... etc
So from server side-mORMot you can have:
a1) rest server with ORM enabled and a table called Customers or
a2) method based server with a method call Customers and you assembly the json for customer with ID=5; you choose.

b) EWB has a TDataset component, which is something like in-memory TClientDataset and has a method LoadRows. So, you have a in-memory TDataset with columns defined at design or runtime and you only ask for rows which will come from mORMot as json array. So you will have a mORMot method based server with only one method which assembly the response every time in the same JSON format - array of rows: ex for table Products:

{ "rows": [
{ "ProductID": "9V-BATTERY-12PK",
  "Description": "12-pack of 9-volt batteries",
  "ListPrice": 20, "Shipping": 2 },
{ "ProductID": "9V-BATTERY-4PK",
  "Description": "4-pack of 9-volt batteries",
  "ListPrice": 4.5, "Shipping": 1.5 },
{ "ProductID": "CALCULATOR-BUSINESS",
  "Description": "Business calculator",
  "ListPrice": 10, "Shipping": 1 }
] }

a) is the OO way, b) is some "tradional" Delphi way. (LoadRows of TDataset has inside a hidden TServerRequest).

@edwinsn

Yes, it supports debugging.
EWB doesn't work with CSS. It has a better way, GUI interfaces. You can easily replicate the bootstrap framework.
Check this and you can figure yourself: Elevate web builder creating custom component

Edit: By replicate, I mean the components are already there, but you can change their style as you wish.

And how much bandwidth produces http.sys?

I was able to modify Samples-06 and I was able to return a dynamic array of records. Looking from browser, Delphi 10 serializes this like normal JSON. By the way these days I have to make new samples starting from Samples-06 making method based service very clear how to return different structures of data (a result but not with names pipes, a record, an array of records, a file, ...) because o lot of beginers in mORMot like me spin their head around with so many options. This is my conclusion reading this forum together with my small learning experience. People they know what is a rest server and they want to know quickly how to create a SOA with visible (JSON) results from browser. When it's ready I'll send an email, maybe you publish them.

By the way, speaking of so many options, which functions/procedures I have to use so an array of records (or similar fast non-compatible "TDataset") aka a result set of a query should be returned by a method base service; the fastest way. What do you use to serialize to binary a result set of a query? Thank you.

"AFAIR TSQLDBProperties does support "execute block...." since we use it for batch insert/update/delete at ORM level."

That's great news. Thank you. I'll try it.

"To manage JSON in a custom way, take a look at TDocVariant in the documentation.
It is a fast and quick way of manipulating JSON."

I'll look also, but I think my solution resides in TDynArray since my clients for this replication service are only Delphi apps. My priority is to be fast as possible and eventually smallest content; so I wanted to get rid of 2 times JSON escaping:

Log = record
           LOGID: Int64;
           TRANSACTIONID: Int64;
           LOG_TEXT: RawUTF8;
         end;
LogA = array of Log;
DynLogA = TDynArray;

And as a result of my method based service will send DynLogA, which will be binary serialized compact, like you said in documentation, as Base64. So that's great. I have to try it.

"If you use TSQLDBProperties, you could retrieve the JSON directly from a SELECT.
Consider using Zeos/ZDBC, since it has direct support of JSON with TSQLDBProperties, and is very optimized for speed. Just use the 7.2/7.3 branch."

I'm considering Zeos (with or without TSQLDBProperties) because I need direct UTF8 buffer, so I can get rid of conversion.

You exposed a TZConnection from main connection of TSQLDBZEOSConnectionProperties? because I think it's not possible because like I saw in source code, TSQLDBConnection passes work to "Properties" thru a OnProcess callback.

I think you created a new TZConnection, so for my case, at the moment I can't use TSQLDBZEOSConnectionProperties.

He he, I find it myself but I didnd't have time to update here. The paper which describes the Delphi AMPQ client is here:

http://66.85.163.250/files/articles/BP37_38_AMQP.pdf

kbmMW also have their own messaging system (WIB  - publish/subcribe) which is good to use when deploy AppServer in LAN (one executable mORMot http.sys server + WIB server) - easy to make an installation script, and all the settings on default since the number of user will be small. When deploy in cloud the other scenario is good: mORMot AppServer separately and RabbitMQ+Delphi AMPQ client -  and can be made some speed optimizations conf. and installation doesn't matter because it's your server and you are in charge of.

I can't use sqllite because I have a lot of stored procedures.

I know that mormot is very refined, that's why I stopped here.

I read quickly the SAD but I still have question how to do some stuff.

I want to achieve 3 objectives:

1. To put data in cloud and to use the same thick windows app (with small modifications) - I'll use the scenario from previous post - localdb + firebird embedded + method based service on server
2. Translate to a new web app for reporting/"light use" with the excelent Elevate Web Builder - I'll use also method based service (EWB AJAX requests seems to be fine with method based services).
3. Make a REST API  for Orders, Invoices, Customers,... to  3rd party app interactions. I read a response in forum that can be achieved also with method based service but I can't figure out how enable something like this:

GET /Orders/{OrderID}/Products/{ProductID}

I read the Method based services chapter, I know that I have to create a TSQLModel 'Orders'- pass it to create method TSQLRestServerFullMemory- pass it to a THttpApiServer but I don't have a working example how to do it.

[qoute=Leslie7]It is worth testing  your usage scenarios  with mORMot for speed and resource consumption before making design decisions.[/qoute]

Yes, it's true, but I need working examples for all 3 objectives (I have solutions for the first 2), so in that way I know if I can go to the way of one big db, multiple processes + multiple dbs, I have to know how URI will be influenced,..etc It will be really helpful if someone posts some code for enabling:

GET /Orders/{OrderID}/Products/{ProductID}

@Leslie7,

Already have replication in my mind, but for the sake of question simplicity, I didn't mention it.
1. I can't use ORM(the app exists, so I'll not recode it), so I can't use mormot replication. 2. Multi master replication or merge changes configurations is out of the question.

So, my intended scenario:

Every user of MyCustomerX connects LOCALLY (on his computer thru Firebird embedded) to a READ-ONLY replicated database (of course they have only MyCustomerX databases). Every time the app needs data reads locally and load it to Firedac memory tables. At ApplyUpdates for those FDMemtables, I generate DML SQL script (with Firebird "execute block", so it's a single unit of work even it's master/detail update). That SQL script I'll send it to a method based service "ApplyUpdatesAndGetChanges" for execution. The method based service will return all changes log SINCE previous syncronization, including the just applied updates. Posted changes will be notified to other users using the same "master-database" in that moment thru a ZeroMQ or RabbitMQ server, so every notified user will call a method based service "GetChanges" .

How the "master-database" keep the changes log, you can read in excelent article  "http://www.ibexpert.net/ibe/index.php?n … Replicated"
My biggest luck to apply the methodology from that article is that I already have one unique ID=Int64 for all the records, no matter which table.

It's the best architecture I can think of. It's fast (all reads are locally), it's safe (only one true master write), minimum resources use on our cloud-server and data on every client it's up to date almost instant.

What you proposed to use one big master-database and few replicated detail-database filtered by client has 2 drawbacks:
1. 6000db x 100 mb = will be already 600GB database and growing
2. I can't use "fast download database". "Fast download database" means backup now "master-database", zip it and download it quickly to client to replace "detail-database"(less than 15 MB for a full 100 MB db). This is useful when synchonization was a long time ago and changes log is to long to be asked by SQL(part by part and buffered to client), or the client database is corrupted, or computer is reinstalled etc.

This is the model I'll go to enable "cloud" to my existing app. I'm asking those questions because later, I want to publish REST API to those databases, so I want to clear my head from the start and to consider all the aspects of this implementation.

Please write a snippet of code because I didn't find in samples.

Yes, it's true, 400 threads it's not a problem, but number of clients could grow.

But another thing is the problem: each client can have 20-30 databases = 3000-4000 databases, so in your scenario I should have "200 mormot server executable" x "2 threads https.sys" x "connection pools to 4000 db", which IMHO it's not viable.

I think I should have my own connection pool maintained by a background thread who connects dbs dinamically when requested and disconnect them after let's say 10 min of inactivity for that particular db.

The question is if I use my own connection pool and not the "official" TSQLDBConnectionProperties, I can still use later the ORM part of Mormot? because, for start, I'll execute method based serviced to serve SQL queries, but later I want to publish my REST API for Invoices/Orders/Customers/etc.