mORMot Open Source

The EWB layout system is an absolute game-changer. Additionally, combined with component creation and interfaces, you have the ability to create truly responsive, modern web apps, that look any way you wish.
You can create components by combining small elements, all in Pascal, no HTML, please see this video:

https://www.youtube.com/watch?v=dZEQBqP8NME

Regarding mobile apps, qoute from forum: "I also have to do this a lot. In my experience, the EWB Layout functionality is easily good enough for me to actually create a single app that is responsive based on device resolution. You should explore this functionality if you have not done so already, along with the Autosize functionality. The only area where I have had to "tweak" things is the removal of scrollbars on mobile devices."

These days EWB3 will be launched with many more features.

There a trial version and if you say is ok, I can contribute with a licence. I will send a private email.

He asked if is a viable option to connect locally on mobile device to mORMot server? FMX is the client and mORMot Android/iOS background app is the server.
I'm also interested in the same approach, because we can get sqlite encrypted access on mobile, plus MORE IMPORTANT: real unified code between mobile, desktop, server - and only the GUI differs.

What @ab said + "with" helps compiler to make some optimization.

ab,

FastMM4-AVX is any better? Can you make it FPC/Linux compatible?

That's what I wanted to say: mORMot+FPC packed in shared library. This library can be used by a FMX project.

Some benchmarks will tell the answer, especially in heavy multi threaded scenario.
If the improvement is real, I don't think commercial users will not spend equivalent of 16GB DDR4.

Anyway FPC/Linux needs a good MM, so this is a nice opportunity to investigate - when a quality product is offered at a decent price, people will pull the trigger.

Also , please read: fastmm5-now-released-by-pierre-le-riche-small-background-story/

Thank you very much. I'll investigate when I have some time.

What aboutL

What @sakura is asking is the same methodology implemented by TeamViewer, GoToMyPC, etc:

The client is trying to connect directly to server but if it fails, will try by connecting to a Relay-server (which is maintained by TeamViewer company) to which, the desired server is already connected as a client. So relay server acts as a forwarder between the client and the server(which is connected as a client to relay server) without making any hardware changes.
This is the most useful setup when direct connection fails and that's why software like TeamViewer works every time no matter how complicated infrastructure exists or how many firewalls are between the peers.

In my opinion is better to use a full TCP/IP connection between Office-server and Relay-server(use Indy maybe), but only when is needed because on the Relay-server side it uses one thread per connection.

I would go with this scenario:
1. Office-server is connected to Relay server by Websocket protocol (only for notification of first request of a session).
2. Client sends request to Relay.
3. On first request, Relay sends notification by websocket to Office-server and waits for full-TCP/IP-connection initiated by Office-server.
4. Office-server opens full-TCP/IP-connection to Relay.
5. Relay picks-up the full TCP/IP connection and sends the client request to Office-server, waits for processing, gets the response and send back to client, release the full-TCP/IP-connection handle without closing.
6. If there is no traffic on TCP/IP full connection for let's say 15 min, Relay closes TCP/IP full connection.
On subsequent requests only steps 2. and 5. would happen.

Or, another possibility is to use message-brokers like RabbitMQ, where Client and Office-server are connected as clients to RabbitMQ-server(Relay).

@ab

mORMot is a open-source project, so you can use C++ Builder Community Edition.

@pvn0 search for "delphi amqp".
Kbmmw also has an Amqp 0.91 library.

I also confirm VirtualBox is very easy to setup and it's free.

I see now that FastMM4-AVX is compatible with FPC-64bit:

Didn't compile under 64-bit version of FreePascal

@mpv
Can you test if the problem described here is still reproducible with latest FastMM4-AVX?
If is still reproducible, I can contact Maxim Masiutin (author of FastMM4-AVX) to look over it.

Thank you for your response.

Maybe I'm misjudging but regarding mORMot on Windows, seems old good Delphi+Win32 has the most optimizations:
Delphi+Win64 doesn't have Sqlite AES encryption and most of the routines are PUREPASCAL.
FPC+Win64 doesn't have FastMM4..

I would like to use Win64 because my design involves multi-tentant databases, but also encryption is very important. So, I have to choose between 2 strategies:
1. Use FPC+Win64, but I'm a little concerned about speed, about not having FastMM4 and other novelties..
2. Use Delphi+Win32 and every Sqlite database will have smaller cache than default (reside more on OS cache), every sqlite database connection will expire&close when it's not used more than 15min and eventually, if is needed, running many mormot servers on the same machine but different ports and to allocate fewer threads to every mormot server.

I'm glad to here pro and cons for every way. Thank you.

Any news on AES Sqlite encryption on Delphi-Win64?

And as a curiosity why Delphi-Win64 was not supported from the beginning?

First of all thank you for all responses.

I don't "feel" the need for DTOs, because I'm "cheating" - I'm Load-ing aggregates from local replica, so performance is very good.
As @ab says I'm certain that any abstraction will pay dividends sooner or later, I'm not just there to collect them

In the middle future when I'll arrive to my need of deserialize TObjectsLists, I'll try my best as I propose and make a pull request and maybe @ab will polish it. Maybe I'll even try to make a option j2oUseCocForArrayNames (Convention over Configuration) - ex: if json array name is "OrderDetails", I'll look for a class name 'T' + 'OrderDetails' + 'Class'.. I have to think...

So you say you have 2 types of objects for one entity: DTO(record&arrays) and Domain Objects(TSQLRecord with logic?)?
You have something like this: DO <-> DTO <-json-> DTO <-> DO <-> DB?

Please shed some light. Thank you.

EDIT: I read Index» mORMot Framework» Difference between DTO and entity
From that thread, this corresponds to my needs:   

I thought that is not worth it, because if it was, was already implemented

I need TObjectList because I want to have common code (at least DTO and validation) between Delphi-client/mormot-server/future-EWB-client (doesn't have T*ObjArray). Also I see a lot of people in forum try/need to use TObjectList, so I propose a solution:

RegisterClass('OrderDetails', TOrderDetailsItemClass);
RegisterClass('OrderPayments', TOrderPaymentsItemClass);

in function JSON2Object, make a new option: j2oGetItemClassFromArrayName

{"Order" = {"Client" = "XYZ",
           "Date" = "2012-04-30T02:15:12.356Z",
           "Amount" = 1000,
           "OrderDetails" = [
                            {"itemName" = "Clock",  "qty" = 1, "Value" = 100},
                            {"itemName" = "AClock", "qty" = 1, "Value" = 900}
                            ],
           "OrderPayments" = [
                             {"type" = "cash", "value" = 600, Date = "2012-04-30T02:15:12.356Z"}
                             {"type" = "card", "value" = 400, Date = "2012-04-30T02:15:12.356Z"}
                             ]
           }
}

With this solution can use many sub-TObjectLists and will solve the problems for a lot of people.

1. Forget my ignorance/lack of knowledge, but from your blog posts, I found out that avoiding LOCK instruction is crucial for good performing multi threading apps, because it will flush cache cores; that's why mORMot avoids allocating memory, char conversion, etc.. I read that atomic InterlockedXXX is cheap, but every InterlockedXXX will not generate a LOCK instruction? so it's no better to get rid of them where is not needed?

2. Since you compiled with SQLITE_THREADSAFE=1, can I change the settings to SQLITE_CONFIG_SINGLETHREAD from your code?

3. I see that every time you deserialize a JSON array, every object in the list is created. It's not worth it to have a fixed small pool of that kind of object? Let's say I can have a fixed small pool of "OrderLines" of 10 objects. Most of Orders will have maximum 10 OrderLines, otherwise if it has ex 12, 10 will be taken from pool, 2 created. I say "fixed", so the array to be maximum fast and not thinking to supplementary allocations/deallocations. A configuration setting with PoolCapacity can be created for every registered class. What do you think? Or creating an object is so fast that it doesn't worth a pool?
EDIT: Configurable fixed pool per class per thread (no synchronization between threads, no allocation for supplementary objects).

I know that what I'm asking here is premature optimization, but it's good to know and few optimizations here and there makes a better performer.

Sorry for bothering, but in your production server/s do you use "off exc"?
Thank you.

I never used Sqlite., so I ask: If somehow I have to force turn off exe-server with multiple Sqlite opened connections in "off exc" is not a big chance of corruption?

@ab
Yes, it was in a VM but these results are on metal (only 10 records - for me this is a realistic scenario):

Insertion speed
 	               Direct	Batch	Trans	Batch Trans
SQLite3 (file full)	128	1314	1150	1307
SQLite3 (file full exc)	204	1768	1643	1680
SQLite3 (file norm exc)	251	2079	1961	1831
SQLite3 (file off)	240	2711	2650	2796
SQLite3 (file off exc)	13531	20120	18691	19305
ZEOS SQlite3	        94	749	743	684
FireDAC SQlite3	        7057	9407	11428	12437
ZEOS Firebird	        1256	4290	4987	4286
FireDAC Firebird	3865	12004	9478	12484

Even that they've changed initial settings, I see that you test "Firedac Sqlite" with "smFull"

    {$ifdef USEFIREDAC}
    Test(TSQLDBFireDACConnectionProperties,FIREDAC_PROVIDER[dSQLite],'','','',' SQlite3',true,smFull);
    {$endif}

So, "Firedac Sqlite" is "smFull, exclusive" and has 11428 inserts/s(Trans), where your "SQLite3 (file full exc)" has 1643 inserts/s(Trans). So maybe they have bigger buffers or some sort of setting which increases inserts 8x.

I don't argue that reading speed is very good but for me writing speed is essential because almost all readings happen on client from slave replica.

Thank you for your great work, I'm trying to be constructive and maybe some time to time we find something you miss

@miab3
I'm interested only on "smFull".

I wanted to evaluate Sqlite vs Firebird embedded.

In example 15, I switched to 10 records because is a realistic scenario (inserting an "order" object it has usually 7-10 record).

Insertion speed:

                             Direct    Batch       Trans        Batch Trans
SQLite3 (file full)        424    3518        2673        1756
SQLite3 (file full exc)  178    3324        2130        3278
ZEOS Firebird             924    2092       2942        2464
FireDAC Firebird        3795  11507       9182       11834

Why "FireDAC Firebird" is 3 times faster than "Zeos Firebird" and also 3 times faster than "SQLite full exclusive", since Sqlite access is maximum optimized in your framework? and I mean for "Trans" because I don't think I'll use "Batch" (anyway Firedac has also a very optimized batch).

Note: I need maximum safety, and since Firebird is implicit in "Full" mode is fair that Sqlite to be also in "Full" mode.

Thank you.

You don't need to use TFDQuery. Using TFDquery will be fine but in the end it's somehow the same approach as Datasnap (many allocations in memory -> many LOCK instructions -> slower performance). Use synDBZeos unit to access Firebird and there you have ISQLDBRows.FetchAllToJson. It is the fastest mode to return database rows to corresponding Json.

Use method based service: http://blog.synopse.info/post/2010/07/1 … phi-7-2010

1. Every thread has it's own stack (usually 1 MB, but can be changed). Every variable(of a method) of basic types/records/static arrays is allocated on stack (it's private space of that thread).

2. You can't make a read-modify-write of global variable without CriticalSection because it not atomic operation (is 3 operations) and a thread switch can happen exactly between those 3 ops. So you can't make index := Inc(index) without CS.

3. Let's say you only want to write to an integer(32 bit) in multiple threads without protecting with a CS. It is an atomic operation as long as memory manager align memory addresses to at least 32bit. If you write an Int64 from multiple threads and memory is aligned 32bit you can have bad results (it's not atomic).

4. If you decide to use point 3. (write integer, with memory aligned 32 bit, without CS) you must know that operations executed are not in the order you elaborated in your program, but are executed in a different order because of optimizations in L1 cpu cache. If you need exact order you must protect with a memory barrier. CriticalSection, SRW, InterlockedIncrement, etc .. they all have memory barrier.

5. Programming without LOCK is an super-expert task. Optimize your algorithm so you minimize the number of LOCK (don't allocate strings - work with a buffer, reuse the same instance many times, etc). And when you have to LOCK use InterlockedIncrement, InterlockedExchange, etc... They are the slimmest type of LOCK (on modern cpu is one instruction). If you need more, use CriticalSection or SWR (Vista+)..they are also implemented with InterlockedIncrement, but if the LOCK is taken by another thread they degrade successfully to kernel WaitForSingleObject. You can use InitializeCriticalSectionAndSpinCount so the CS makes like 4000(as many as you want) round-trip instruction trying to acquire the lock, before degrades to much heavier WaitForSingleObject which makes the same thing. For small and fast locks this spin count try is a huge improvement.

I thought that ProcessID = Windows ProcessID or something like this. I get it now, but needs that every node to be registered and get a unique ProcessID before can post to server. It not a big deal. Also it must keep track of many instances of the client to have different ProcessID.
But I think it's not safe enough.. I saw many times (for different reasons) that client time is set back manually for let's say 2 hours, so with a "little luck" the same time can occur again in the same ProcessID.