mORMot 2 in Good Shape

ab · 2021-01-18 18:29:52

I just added regression tests for the raw SOA process...

Some output on Linux x86_64:
https://gist.github.com/synopse/d3a78e6 … 823f89881e

All green!
I guess mORMot 2 start to be almost usable in production... at least for ORM and SOA work.
Feedback is welcome!

macfly · 2021-01-18 19:51:48

Great news!

Thanks ab for everything.

Lazarus + FPC to Win32 = All green 47,79s
Lazarus + FPC to Win64 = All green 38,50s

Delphi 10.3 to Win32 = All green 51,47s
Delphi 10.3 to Win64 = All green 46,17s

Last edited by macfly (2021-01-18 20:06:16)

pvn0 · 2021-01-19 07:06:56

Excellent news! great work ab as always!

koraycayiroglu · 2021-01-19 07:15:04

looks great! thanks for your hard work.

edwinsn · 2021-01-19 08:42:06

Great job as always!

trx · 2021-01-19 09:15:14

The new mORMot is a masterpiece.
Thank you ab!

AOG · 2021-01-19 09:27:50

Hi Ab,
At the moment, mORMot2 does not run on arm. It gives a "property has no RTTI"-error. Naturally, this can be solved.
The question however:
Is mORMot2 suitable and mature now to perform deep debugging these errors ? If yes, I will dig into this and start porting my apps towards mORMot2.
Thanks.

EduardAppelhans · 2021-01-19 09:28:37

Congratulations !!!

flydev · 2021-01-19 09:35:31

Awesome, does it mean I can clone the project and recompile my fresh project under mORMot 2 ?

ab · 2021-01-19 09:45:03

@AOG
Yes, mORMot2 RTTI and JSON kernel should be stable and mature enough to fix such problems.
This would be the next logical step.
We rewrite the whole RTTI part, to call directly the TypInfo.pp FPC unit, so it should be more future-proof and cross-platform than mORMot 1, where we re-defined almost every low-level RTTI structures by hand - and twice (once in SynCommons.pas and once in mORMot.pas).
So it is possible that mormot.core.rtti.fpc.inc has some bugs with ARM, and its RTTI structures alignment requirement. I didn't take the time to validate it, so if you can, your help would be very valuable!
I focused on Intel/AMD on Windows and Linux up to now. It compiles and links on BSD, Darwin, Arm and Aarch64 but I didn't run nor debug yet on those platforms.

@flydev
You can make some experiment, but there are still some missing part, e.g. the DDD units.
I still don't know exactly what I would do with those DDD units. I am not really pleased with those, which are a bit difficult to use and adapt. Our KDD approach allows a more relaxed way of using the SOA and the ORM - we may focus on it.
So first check if all that your requirements were translated, and report here your findings.

Note that the ORM with external DB has not be fully validated yet.
Feedback is welcome, because it is hard to validate on all DB/Library/OS combinations out there...

flydev · 2021-01-19 12:59:49

ab wrote:

Note that the ORM with external DB has not be fully validated yet.
Feedback is welcome, because it is hard to validate on all DB/Library/OS combinations out there...

Thanks @ab,

I will test with external db and I will report it.

ab · 2021-01-19 16:05:24

After some patches proposed by Alfred:

So 100% green on ARM - Raspberry Pi - too

profh · 2021-01-20 05:35:49

Excellent! great work ab as always!

ab · 2021-01-22 10:03:40

Some other news from Alfred:

1. Linux aarch64 regression tests - all green!

2. And Android aarch64 regression tests - also all green!!!!

For Android, we just introduced a TSynTest.WorkDir property to store the temporary information, and use a fake console rendering for output.

So mORMot 2 has very high multi-platform and multi-compiler abilities, thanks to its new units architecture.
This is just great, and I would like to thank again Alfred for its feedback and hard work on multi-platform support since early mORMot 1 times!

xalo · 2021-01-22 11:43:32

It's amazing!! Just one word in my language:
B R A V O!!!!

Vitaly · 2021-01-22 13:48:33

Joining all kind words!
I will try to test migration starting from some simple projects. Will mORMot2 stay located at the same repository https://github.com/synopse/mORMot2?

lagodny · 2021-01-22 15:46:12

Wonderful job! Congratulations. One small question, is there any hope for Delphi support for Android?

ab · 2021-01-23 10:51:42

@Vitaly
The repository may remain as mORMot 2 in GitHub - with a fossil own repository.

@Lagodny
No Delphi Android support yet, due to the lack of shortstring.
But there are the cross-platform client units working for Delphi Android.

uian2000 · 2021-01-23 12:27:34

Good news of 2021.

I've test this trunk under Win10 X64, code page 936(zh-cn,中文简体）.

When I compile and run this test project with D7, 6 asserts failed, but with DXE, about 2,353 asserts failed.

Errors occurred when processing characters. I know it's complicated, Just commit this issue, and hope some HERO could resolve it.

Compiled with D7.

 1.2. Core process:
  - RTTI: 1,338 assertions passed  390us
  - Url encoding: 200 assertions passed  411us
!  - Encode decode JSON: 1 / 427,570 FAILED  28.77s
  - Wiki markdown to html: 56 assertions passed  731us
  - Variants: 99 assertions passed  515us
!  - Mustache renderer: 5 / 58 FAILED  36.61ms
  - TDocVariant: 91,785 assertions passed  102.82ms
  - TDecimal128: 17,446 assertions passed  2.02ms
  - BSON: 245,072 assertions passed  20.04ms
     100000 TBsonObjectID.ComputeNew in 18.91ms i.e. 5,287,368/s, aver. 0us
  - TSelectStatement: 221 assertions passed  627us
  - TSynMonitorUsage: 1,202 assertions passed  603us
  Total failed: 6 / 785,047  - Core process FAILED  28.95s

Compile with DXE.

 1.1. Core base:
  - Ini files: 7,028 assertions passed  33.74ms
!  - UTF8: 14,000 / 1,205,874 FAILED  1.18s
  - Url decoding: 1,101 assertions passed  687us
!  - Baudot code: 6,285 / 10,007 FAILED  28.90ms
  - Iso 8601 date and time: 200,831 assertions passed  17.22ms
  Total failed: 20,285 / 19,604,610  - Core base FAILED  5.54s

 1.2. Core process:
!  - Mustache renderer: 5 / 58 FAILED  38.66ms
  - TDocVariant: 91,785 assertions passed  107.47ms
  - TDecimal128: 17,446 assertions passed  1.51ms
!  - BSON: 8 / 245,072 FAILED  4.40ms
     100000 TBsonObjectID.ComputeNew in 3.11ms i.e. 32,144,005/s, aver. 0us
  Total failed: 13 / 834,650  - Core process FAILED  8.41s

 1.4. Core ecc:
!  - Certificates and signatures: 6 / 91 FAILED  80.73ms
  Total failed: 6 / 521,825  - Core ecc FAILED  1.04s

Run log is too large to be post here....

I can't access pastbin.com by now, so I've just trimed error report.

Last edited by uian2000 (2021-01-25 12:43:14)

ab · 2021-01-23 18:54:11

Please don't post such huge text in the forum directly.
Use something like Gist and make a link.

Yes, the tests have problem with complex system encoding. They expect a Win1252 system on Windows.
But the mORMot library itself is UTF-8 so has no problem with your code page in practice.
Those are tests errors, not framework error...

sakura · 2021-01-25 06:23:55

So far, looking great. I am using mORMot2 for internal production code (however, basically just the JSON-part), and it is working smoothly. The code created by that is working standalone, as we have to work on XML data, so, no fear of a pre-production version going into production code themselves.

The tests on Delphi 10.4 look promising.

Win32, Release:
Time elapsed for all tests: 56.27s
Total assertions failed for all test suits: 0 / 41,982,561
! All tests passed successfully.

Win64, Release:
Time elapsed for all tests: 45.14s
Total assertions failed for all test suits: 0 / 41,955,145
! All tests passed successfully.

However, when starting, I get a little popup warning:

Static SQLite3 library as included within mormot2tests is outdated!
Linked version is 3.31.0 whereas the current/expected is 3.34.0.
Please download latest SQLite3 3.34.0 revision from
https://github.com/synopse/mORMot2/releases/download/pre1/mormot2static.7z

And I have, just before, updated to the latest version.

Regards,
Daniel

P.S.: Du to user error, I've tested a three day old version, however, the current version performs similar, with the same popup ;-)

Last edited by sakura (2021-01-25 06:37:44)

Vitaly · 2021-01-25 06:53:10

sakura wrote:

the current version performs similar, with the same popup ;-)

Also had this warning, but then downloaded static files from another link: https://github.com/synopse/mORMot2/rele … 2static.7z

ab · 2021-01-25 10:09:09

The warning popup text is indeed wrong and confusing, because deprecated.
We have to go to https://github.com/synopse/mORMot2/releases then keep the latest version.

I have changed the text.
Please check https://github.com/synopse/mORMot2/comm … 4b1c157541
Thanks for the feedback and sorry for the confusion! 8)

ab · 2021-01-25 13:23:31

I have just upgraded the engine to the latest 3.34.1 revision.
It was the good occasion to include some SQLite3 extensions:

#define SQLITE_ENABLE_GEOPOLY
// enable GeoJSON over RTREE https://sqlite.org/geopoly.html 

#define SQLITE_ENABLE_REGEXP
// enable the compact https://www.sqlite.org/src/file?name=ext/misc/regexp.c
// - can be overloaded with any other implementation

#define SQLITE_ENABLE_SESSION
// enable Sessions https://sqlite.org/sessionintro.html
// - not used/published yet in pascal code

Those are available in both mORMot 1.18 and mORMot 2, because both share the same static binaries for SQLite3.

Please check https://synopse.info/fossil/info/d8823afb36bb687e
and https://synopse.info/fossil/info/105b6a43a19b5d76

For mORMot 2, the new binaries are at https://github.com/synopse/mORMot2/rele … ite.3.34.1

edwinsn · 2021-01-25 15:30:34

ab wrote:

#define SQLITE_ENABLE_SESSION
// enable Sessions https://sqlite.org/sessionintro.html
// - not used/published yet in pascal code

The session extension seems to be a perfect tool for implementing a one-way database sync across nodes!

sakura · 2021-01-26 07:26:38

Thanks @ab, everything works with the current repo on Delphi 10.4 (Win32/Win64) as expected. Millions of perfectly passed tests and zip failed ones. I am impressed, have always been, and likely will always be, with your work and dedication.

Vitaly · 2021-01-27 14:22:25

Just a small report. The last commit probably doesn't contain changes for mormot.orm.core. Line 15390:

[dcc64 Error] mormot.orm.core.pas(15390): E2003 Undeclared identifier: 'Utf8FirstLineToUnicodeLength'

ab · 2021-01-27 14:41:53

Yes, there are some pending changes which need to be fixed...
But I just get all tests green so I guess the commit will be almost close...

Vitaly · 2021-01-27 15:01:05

yeah, I had no doubts that it is just a "technical" issue. Decided to notify just in case

ab · 2021-01-27 18:12:43

Should be fixed now.

I have reduced the memory needed for TOrmTable by half on 64-bit systems, without touching the performance AFAICT.
Check https://github.com/synopse/mORMot2/comm … 309cfabaf6
This is a feature I wanted to do since a lot of time, and mORMot 2 was the good moment to introduce this I hope.
The number of source code lines have also slightly decreased, as a nice side effects.

Vitaly · 2021-01-28 14:30:59

ab wrote:

I have reduced the memory needed for TOrmTable by half on 64-bit systems, without touching the performance AFAICT.

I would not say that I ever regretted about mORMot memory consumption, but that sounds Very nice! Even if so far I don't understand the particular advantage of it, I already always know for sure: if Arnaud made something new, it will be better, more optimized, faster, and so on

I guess I can share some results of the first stage of my migration to mORMot2. I Hope, it'll be interesting for somebody.

I have done the migration of only one small project group (6 Delphi projects) and a part of my base classes, using mORMot. It contains several TWinHttp-based REST/JSON clients for different web-services with automatic DTO handling. It also contains some work with external DBs: FB2.5 and SQLite3 (ORM, batches, etc.). It doesn't contain any server/daemon. Custom serialization is frequently used there. Some projects are using threads. And it all works well with mORMot2, now
At first, I made it working in compatible with 1.18 mode, then I switched to PUREMORMOT2 - it was a good opportunity to review, clean, and refactor the old code.
I use my own base classes, based on mORMot classes - probably it made the whole process a bit easier for me.

My personal impressions:
- from the beginning, it was pretty hard to find needed classes and functions, but after several days I became pretty used to it.
- I missed some documentation for mORMot2 like for 1.18 - Arnaud spoiled us The documentation often helped me to understand different ways and choose the best one.
- the framework source code formatting became more readable (just for me, at least).
- naming also became much more convenient (for my taste - "Kotlin-way" and UTF8String are very nice).

My current losses:
- Zeos support (temporary).
- EngineAdd, EngineUpdate, EngineBatchSend overriding for TRestServerDB, which I used for universal transliteration (our legacy FB DB is using ISO8859_1 ) - will try to find another workaround for it. Can't afford this loss.
- i18n, I couldn't find some functions, which I used before. But it is probably will be not needed at all, so I have cut it off.

It took pretty much time to get used to mORMot2, but I consider the first stage of migration to be successful I'll continue the process with other much more complex project groups.

ab · 2021-01-28 17:34:06

Thanks for the very positive feedback, and the detailed reports whenever you reached a bug of mORMot 2!

About EngineAdd... override, I guess the problem by rather be fixed at mormot.db.sql (or mormot.db.sql.zeos) level.
Feedback is welcome, here!

You are right, i18n is not included yet, nor the TDataSet/UI part of the framework.
What do you expect/require to find in mORMot 2?

Vitaly · 2021-01-28 20:54:30

ab wrote:

Thanks for the very positive feedback, and the detailed reports whenever you reached a bug of mORMot 2!

Yeah, every time I wished to find the solution by myself to avoid disturbing you much. But the only thing I could do is to locate the problem as much as I can. I guess my level is not high enough so far

ab wrote:

About EngineAdd... override, I guess the problem by rather be fixed at mormot.db.sql (or mormot.db.sql.zeos) level.
Feedback is welcome, here!

I'll try a couple of ideas first, although they are pretty weak.

ab wrote:

You are right, i18n is not included yet, nor the TDataSet/UI part of the framework.
What do you expect/require to find in mORMot 2?

Well, I'm not using mORMotUI/TDataSet. I have a TDrawGrid-based class (inspired by TSQLTableToGrid) with some additional functionality and improvements, and it works well enough already with mORMot2 Everything else seems to present.
I guess I might easily get some troubles further with WinService, HTTP/REST/SOA-servers/clients migration, especially in our FHIR-specified project, where we've overridden mORMot classes pretty deep (I'll make its migration the last one). But I'm sure it all will be solved

ab · 2021-01-29 09:34:14

Vitaly wrote:

ab wrote:
I have reduced the memory needed for TOrmTable by half on 64-bit systems, without touching the performance AFAICT.
I would not say that I ever regretted about mORMot memory consumption, but that sounds Very nice!

An interesting (but technical) article which explains how V8 use such offsets instead of pointers is https://v8.dev/blog/pointer-compression
This blog is full of nice readings, anyway.

Vitaly · 2021-01-29 11:26:48

Will definitely study it, thanks! And look into your implementation.
I've learned so much from the mORMot. Although, sometimes it is really hard for me to understand some code from the first, second, or even tenth look

Vitaly · 2021-01-30 17:35:16

I'm not sure so far what has happened, some (not all) projects stopped working with the latest commit. Such exception appears at the start:

Project NextGeoLoadDemo.exe raised exception class EJSONException with message 'Rtti.Count=5 at mormot.core.json start'.

All works fine with the previous commit, the problem seems to appear with 544078bdf080cd6a80566461567cbc84131a948f only.
Could it be that I'm using logs incorrectly since the last commit concerns only mormot.core.log unit?

ab · 2021-01-30 22:47:54

Isn't it the Rtti.RegisterFromText([TypeInfo(TSynMapSymbol), _TSynMapSymbol,...) which fails?

Which compiler are you using?

Vitaly · 2021-01-30 23:50:23

ab wrote:

Isn't it the Rtti.RegisterFromText([TypeInfo(TSynMapSymbol), _TSynMapSymbol,...) which fails?

Nope, checked it with step over - seems it passes the whole mormot.core.log.InitializeUnit well. Exception appears at mormot.core.json.InitializeUnit.

ab wrote:

Which compiler are you using?

Delphi 10.4.1 Win64

I was curious why not all projects are failing and I think I found something, hope useful. At least, now I can overcome the exception.
This particular project contained mormot.core.log in uses list in dpr (I was setting LOG_VERBOSE sometimes directly in dpr for having a full log).
Such order causes an exception with the latest commit:

program NextGeoViewSQLite3mORMotDemo;
uses
  Vcl.Forms,
  mormot.core.log,          <--------- before the main form unit
  mormot.core.rtti,
  uFormNextGeoViewSqlite3MormotDemo in 'uFormNextGeoViewSqlite3MormotDemo.pas' {FormNextGeoViewSqlite3MormotDemo},
  uNextGeo in '..\..\Source\uNextGeo.pas',
  ...

And this order goes well, no exception:

program NextGeoViewSQLite3mORMotDemo;
uses
  Vcl.Forms,
  mormot.core.rtti,
  uFormNextGeoViewSqlite3MormotDemo in 'uFormNextGeoViewSqlite3MormotDemo.pas' {FormNextGeoViewSqlite3MormotDemo},
  mormot.core.log,          <--------- after the main form unit
  uNextGeo in '..\..\Source\uNextGeo.pas',
  ...

ab · 2021-01-31 14:14:14

It seems that Rtti.RegisterFromText() should be run before mormot.core.json is included in the uses clause...

I will move the JSON registration only when it is needed - I mean in TSynMapFile.ToJson().

Vitaly · 2021-02-01 13:51:31

Checked it with today's commit 679963edbbae2d6146dd420288ddc8ec854d3cd6. Seems the problem has gone. Thanks!

ab · 2021-02-01 13:55:01

Yes, https://github.com/synopse/mORMot2/comm … 3f90017362 was a big refactoring of the mormot.core.log unit.

Now mormot.core.log supports FPC/GDB DWARF symbols.
This was a feature I missed very much from Delphi .map.

Numbers are great: mormot2tests 4MB Delphi .map is converted into a 280KB .mab, and the 13MB FPC .dbg into a 290KB .mab!
Also the address finding is much faster than the FPC RTL version: it resolves in a few microseconds, instead of a dozen milliseconds...
So now we don't have any reason not to have source code line numbers on production logs, especially on servers.

Sidenote 1: DWARF is a powerful but awful to parse format. I spent a lot of time debugging and implementing it properly...
Sidenote 2: it seems that cross-compiling from Linux to Windows emits wrong DWARF content... is it a known FPC bug?

mpv · 2021-02-01 17:20:51

Support of DWARF symbols is a HUGE improvement. I miss it very much too. This is a good reason for me to start move my product to the mORMot2.

What type of DWARF should we use? (I mean value of "Type of debug into parameter" or in term of command line -g -gw -gw2 -gs or -gw-godwarfsets)

ab · 2021-02-01 17:28:32

It should support any DWARF version - I guess version 2 is just fine.

mpv · 2021-02-02 07:38:07

@ab - I agree with you - this message is PITA.

{$ifdef MSWINDOWS} // PITA popup
  MessageBoxA(0,pointer(error),' WARNING: deprecated SQLite3 engine',MB_OK or MB_ICONWARNING);
{$endif MSWINDOWS}

I run a regression test using CI, and it stops an execution process, but nobody can press a OK button. The same is in case application is executed as a service.
What if we throw there instead of showing a popup?

And can I ask you to enable an autoindex on location https://synopse.info/files/ by adding an nginx directive

location /files {
    autoindex on;
}

It simplify a manual file download (I can't copy a full UTL text from popup window)

Last edited by mpv (2021-02-02 07:43:26)

mpv · 2021-02-02 07:58:54

The reason of popup is what at last this files (other OS and arch folders not verified) is outdated in mORMot1 GIT repository

 x86_64-win64/libsqlite3-64.a
 x86_64-win64/sqlite3-64.dll
 x86_64-win64/sqlite3.o

compared to 7z archive. Can you, please, update a static files in mORMot1 repo?

Last edited by mpv (2021-02-02 07:59:32)

ab · 2021-02-02 17:37:22

I have updated the repository.

And replaced the MessageBox() with an exception on mORMot 1 and 2.
https://synopse.info/fossil/info/a00729a74e

mpv · 2021-02-02 19:04:58

Verified, all works as expected (at last for Win64 and Linux64). Thank you!

Vitaly · 2021-02-02 22:10:19

ab wrote:

I have updated the repository.

I'm getting the exception about outdated static lib on Delphi 10.4.1 Win64. Is it missing or my build is broken?

(had to pause my migration to 2.0 and return to 1.18 for a while - need to do some work)

ab · 2021-02-03 13:37:51

@vitaly
Oups... I forgot to push the mORMOt 1.18 files for Delphi on github...
Should be good now.

For mORMot 2, there is a release bin to download, with the exact URI displayed...

Vitaly · 2021-02-03 14:41:48

Yes, all's good now, thanks!

mORMot Open Source

#1 2021-01-18 18:29:52

mORMot 2 in Good Shape

#2 2021-01-18 19:51:48

Re: mORMot 2 in Good Shape

#3 2021-01-19 07:06:56

Re: mORMot 2 in Good Shape

#4 2021-01-19 07:15:04

Re: mORMot 2 in Good Shape

#5 2021-01-19 08:42:06

Re: mORMot 2 in Good Shape

#6 2021-01-19 09:15:14

Re: mORMot 2 in Good Shape

#7 2021-01-19 09:27:50

Re: mORMot 2 in Good Shape

#8 2021-01-19 09:28:37

Re: mORMot 2 in Good Shape

#9 2021-01-19 09:35:31

Re: mORMot 2 in Good Shape

#10 2021-01-19 09:45:03

Re: mORMot 2 in Good Shape

#11 2021-01-19 12:59:49

Re: mORMot 2 in Good Shape

#12 2021-01-19 16:05:24

Re: mORMot 2 in Good Shape

#13 2021-01-20 05:35:49

Re: mORMot 2 in Good Shape

#14 2021-01-22 10:03:40

Re: mORMot 2 in Good Shape

#15 2021-01-22 11:43:32

Re: mORMot 2 in Good Shape

#16 2021-01-22 13:48:33

Re: mORMot 2 in Good Shape

#17 2021-01-22 15:46:12

Re: mORMot 2 in Good Shape

#18 2021-01-23 10:51:42

Re: mORMot 2 in Good Shape

#19 2021-01-23 12:27:34

Re: mORMot 2 in Good Shape

#20 2021-01-23 18:54:11

Re: mORMot 2 in Good Shape

#21 2021-01-25 06:23:55

Re: mORMot 2 in Good Shape

#22 2021-01-25 06:53:10

Re: mORMot 2 in Good Shape

#23 2021-01-25 10:09:09

Re: mORMot 2 in Good Shape

#24 2021-01-25 13:23:31

Re: mORMot 2 in Good Shape

#25 2021-01-25 15:30:34

Re: mORMot 2 in Good Shape

#26 2021-01-26 07:26:38

Re: mORMot 2 in Good Shape

#27 2021-01-27 14:22:25

Re: mORMot 2 in Good Shape

#28 2021-01-27 14:41:53

Re: mORMot 2 in Good Shape

#29 2021-01-27 15:01:05

Re: mORMot 2 in Good Shape

#30 2021-01-27 18:12:43

Re: mORMot 2 in Good Shape

#31 2021-01-28 14:30:59

Re: mORMot 2 in Good Shape

#32 2021-01-28 17:34:06

Re: mORMot 2 in Good Shape

#33 2021-01-28 20:54:30

Re: mORMot 2 in Good Shape

#34 2021-01-29 09:34:14

Re: mORMot 2 in Good Shape

#35 2021-01-29 11:26:48

Re: mORMot 2 in Good Shape

#36 2021-01-30 17:35:16

Re: mORMot 2 in Good Shape

#37 2021-01-30 22:47:54

Re: mORMot 2 in Good Shape

#38 2021-01-30 23:50:23

Re: mORMot 2 in Good Shape

#39 2021-01-31 14:14:14

Re: mORMot 2 in Good Shape

#40 2021-02-01 13:51:31