mORMot 2 in Good Shape

mpv · 2021-02-08 12:05:05

I do not found a command line script to built mORMot2 test under Linux, so I add it (see pull request #9
Also I add a script to update a static folder content from latest release / pre-release.

Build script is designed to work with FPC installed from official rpm/deb (fast and do not require GUI to setup in opposite to fpcUp\fpcUpDeluxe) what allows starts adding a CI (github actions)

ab · 2021-02-08 15:26:32

@mpv
I merged your great script - thanks!
And I have also published the TSynThreadPool worker threads.

For mORMot 2, I have included a new AES-CTR x86_64 asm which outperforms OpenSSL.

   2500 aes128ctr in 2.06ms i.e. 1209482/s or 2.5 GB/s
   2500 aes256ctr in 2.68ms i.e. 931792/s or 1.9 GB/s
   2500 aes128ctrosl in 2.37ms i.e. 1053518/s or 2.1 GB/s
   2500 aes256ctrosl in 3.22ms i.e. 775193/s or 1.6 GB/s

Also tuned TAesCfbCrc x86_64 asm - this is the default WebSockets cipher when using ECDH negotiation.

  2500 aes128cfbcrc in 7.58ms i.e. 329771/s or 701.8 MB/s
  2500 aes256cfbcrc in 9.99ms i.e. 250225/s or 532.5 MB/s

Also TAesCfb which is used (hardcoded for compatibility with mORMot 1.18) in several places by the framework. Also much faster than OpenSSL:

  2500 aes128cfb in 6.98ms i.e. 357807/s or 761.4 MB/s
   2500 aes256cfb in 9.41ms i.e. 265646/s or 565.3 MB/s
   2500 aes256cfb in 13.36ms i.e. 187041/s or 398 MB/s
   2500 aes256cfbosl in 13.47ms i.e. 185473/s or 394.7 MB/s

Only AES-GCM is faster with OpenSSL. But the GCM asm is really complex... so not easy to introduce - we have a non-interleaved AES-NI / CLMUL version which is good but not so fast.

 2500 aes128gcm in 14.41ms i.e. 173418/s or 369 MB/s
 2500 aes256gcm in 17.37ms i.e. 143918/s or 306.2 MB/s
 2500 aes128gcmosl in 3.03ms i.e. 824810/s or 1.7 GB/s
 2500 aes256gcmosl in 3.56ms i.e. 701065/s or 1.4 GB/s

I have also added AES-GCM to the mormot.core.ecc algorithms.
I will continue to work on OpenSSL integration.

ab · 2021-02-10 15:59:21

After a day of patches and enhancements to both mormot.core.crypto and mormot.core.openssl which also improved.

New numbers are available at https://github.com/synopse/mORMot2/commit/964702909db6e

(Edit:)
New 8x interleaved aesni + pclmulqdq x86_64 asm for AES-GCM
https://github.com/synopse/mORMot2/commit/d9512eaa826
- from 300MB to 1.5GB/s
OpenSSL is still faster at 1.8GB/s
... but we had to let it win on one algo at least

sakura · 2021-02-11 06:20:43

ab wrote:

... but we had to let it win on one algo at least

For how long? :-D

PascalDragon · 2021-02-11 20:11:33

ab wrote:

Sidenote 2: it seems that cross-compiling from Linux to Windows emits wrong DWARF content... is it a known FPC bug?

Do you have a small, self contained example that you would be able to report as a bug? Cause that shouldn't be the case.

ab · 2021-02-12 17:12:59

@PascalDragon

When I export the dwarf when cross-compiling with FPC 3.2 from Linux to Win32 or Win64, the SetAddress fields are incorrectly set to x0.
I can see it with an objdump output:

  0x00000031  Extended opcode 2: set Address to 0x0

Check https://gist.github.com/synopse/8f3e45d … 0ec1ea77cb

Sounds like a FPC bug since sometimes the Address is set, sometimes it is not.
With objdump in line mode:

mormot2tests.exe:     file format pei-i386

Contents of the .debug_line section:

CU: mormot2tests.dpr:
File name                            Line number    Starting address    View    Stmt
mormot2tests.dpr                             131                   0               x
mormot2tests.dpr                             135                 0x4               x
mormot2tests.dpr                             137                 0x9               x
mormot2tests.dpr                             135                 0xc               x
.......
mormot2tests.dpr                             159                   0               x

mormot2tests.dpr                             172            0x401f00               x
mormot2tests.dpr                             162            0x401f03               x
mormot2tests.dpr                             162            0x401f0d               x

mormot2tests.dpr                             162            0x401f10               x
mormot2tests.dpr                             162            0x401f1d               x
....

okoba · 2021-02-21 12:42:04

Can you add Lazarus package for mORMot 2?

mdbs99 · 2021-02-21 14:24:07

@okoba mORMot 1 has packages. I guess v2 should be released first, than we can add the packages following what was done in v1.

macfly · 2021-03-19 18:27:58

TSynTests is not accepting the creation of the instance:

  with TSynTestsLogged.Create('mORMot2 Regression Tests') do
  try
    ...
  finally
  end;

Exception on SETPARSERTYPE,  line 9658 of ../../../../mORMot2/trunk/src/core/mormot.core.json.pas

Edit: I forgot that it is necessary to disable -CR option in project properties.

Last edited by macfly (2021-03-20 02:09:48)

HollosCs · 2021-03-21 07:21:16

Hello ab! Your product is fantastical (hungarian slang: “nagyon Gromek”). In the new project I use mormot2.
In the new update I not found 3.35.0 static binaries for mormot2. Mormot2 test not run. Are any timeframe for update or just fix the version string in the file?

mormot.db.sqlite3.static.pas:
{$ifdef OSANDROID} '3.34' {$else} '3.35.0' {$endif};

Thx

Last edited by HollosCs (2021-03-21 09:22:55)

Leslie7 · 2021-03-21 10:48:54

HollosCs wrote:

Hello ab! Your product is fantastical (hungarian slang: “nagyon Gromek”). In the new project I use mormot2.
In the new update I not found 3.35.0 static binaries for mormot2. Mormot2 test not run. Are any timeframe for update or just fix the version string in the file?
mormot.db.sqlite3.static.pas:
{$ifdef OSANDROID} '3.34' {$else} '3.35.0' {$endif};
Thx

It is usually worth waiting a bit with new SQLite releases until the newly introduced bugs are fixed.

AOG · 2021-03-21 12:14:23

I have compiled some sqlite3 Android libs v3.35.0000
Use them as you see fit. Greetings.
https://github.com/LongDirtyAnimAlf/fpc … 350000.zip

Last edited by AOG (2021-03-21 12:14:55)

ab · 2021-03-21 16:27:18

The 3.35.0 is sadly already outdated.
3.35.2 is out..
But I am waiting for the upcoming 3.35.3 which should appear beginning of next week, including latest fixes.

flydev · 2021-03-22 07:38:49

Downloading the source and the static files version 3.34.1 from there : https://github.com/synopse/mORMot2/releases

ask for the 3.34.0 static files when running an executable, is this expected ?

ab · 2021-03-22 08:14:58

Please wait a little bit - I am waiting for SQlite 3.35.3 to fix all this.
And I made a huge refactoring of the statically linked libraries for FPC, so please be a little patient.

flydev · 2021-03-22 08:15:51

Oke @ab, thanks you.

ab · 2021-03-22 18:53:35

3.35.3 on its way....

https://www.sqlite.org/src/timeline?c=8 … 4b019f&y=a

edwinsn · 2021-03-23 07:13:59

flydev wrote:

https://i.ibb.co/pLnmX9h/Capture-d-cran … 083337.png

@ab, the screenshot posted by flydev doesn't look right to me - Sqlite is backward compatible, what do you think about allowing mORMot to be compatible with **new versions** of SQLite?
Take this screenshot as an example, mORMot complains about 3.34.0 is missing while 3.34.1 (the newer version) is already provided?

ab · 2021-03-23 08:08:42

@edwinsn
There are at least two reasons.

Even if SQLite would work with older binaries, sometimes some new functions were added in the API, which expect new binaries.

Second reason is because we use SQlite as reference, but there are other static files, which may need an update.
For instance, in the next static release we will include libdeflate and QuickJS for FPC, and we also rebuilt the liblizard binaries, and get rid of libgcc.a.

damiand · 2021-07-08 08:58:57

Hi

I cloned the mORMot2 repository, downloaded the static files with SQLite 3.35.5, compiled and run the regression tests, using FPC 3.2.0 on OpenSUSE Linux. Unfortunately, some tests failed:

/home/damian/mORMot2/test/fpc/bin/x86_64-linux/mormot2tests 0.0.0.0 (2021-07-07 16:36:50)
Host=XXXX User=damian CPU=4xIntel(R)Core(TM)i5-4460CPU@3.20GHz(x64):FFFBEBBFBFFBFA7FAB270000000000000006009C OS=Suse=Linux-5.3.18-lp152.78-default#1-SMP-Tue-Jun-1-14:53:21-UTC-2021-(556d823) Wow64=0 Freq=1000000
TSynLog 2.0.1 2021-07-07T13:38:23

0000000000000001  ! fail  #5 ../../src/core/mormot.core.test.pas tsyntestcase.testfailed (944) ../../src/core/mormot.core.test.pas tsyntests.run (1171) mormot2tests.dpr main (96) 
00000000000156FF  ! fail  #7 ../../src/core/mormot.core.test.pas tsyntestcase.testfailed (944) ../../src/core/mormot.core.test.pas tsyntests.run (1171) mormot2tests.dpr main (96) 
0000000000016CF9  ! fail  #9 ../../src/core/mormot.core.test.pas tsyntestcase.testfailed (944) ../../src/core/mormot.core.test.pas tsyntests.run (1171) mormot2tests.dpr main (96) 
00000000027A9CFE  ! fail  #6046 777<>777.777 ../../src/core/mormot.core.test.pas tsyntestcase.testfailed (944) test.core.script.pas ttestcorescript.quickjslowlevel (170) ../../src/core/mormot.core.test.pas tsyntests.run (1171) mormot2tests.dpr main (96) 
... (the error above is repeated 1000 times up to #40012)

Any hints?

ab · 2021-07-09 07:59:28

What are the problems of the first issues?
The stack trace is not very accurate: it does not state where the error occurred.
Could you use the debugger and state what is wrong?

The last lines are about QuickJS values.
There seems to be some QuickJS problem on your system.
I will look into it. Perhaps the test is not correct on some OS.

Note that I don't have any issue on my Debian system.

damiand · 2021-07-09 11:57:37

With the debugging options defined in the Lazarus project, I can't debug it. I get the following GDB error:

The GDB command:
"-break-insert +0"
did not return any result.

I changed the build mode to lin64, rebuilt and run the project again. Now all tests are passed, except the 1000 quickjs related errors. All these are raised in section 3.1 (SOA/Core script).

I changed the debugging options but I'm still getting the same GDB error.

Lazarus (2.0.11)/FPC (3.2.0) installation is made via fpcupdeluxe.

UPDATE: I created another Lazarus project besides the mormot2tests and tried to re edit it from scratch. I kept the program body minimum (only a writeln statement) and I added progressively the uses section. Debugging was available and OK, till I added the test.core.base unit.

Last edited by damiand (2021-07-09 14:14:27)

ttomas · 2021-07-09 23:44:59

@damiand, I have the same debugger error when try to debug MVC Blog demo with mORMot2.
In Lazarus menu Tools-Options in Debugger-General options try to change InternalStartBreak to gdsbMain.

damiand · 2021-07-10 09:46:28

Thanks a lot @ttomas for the hint. With this option I can debug the tests project. The failed test is at line 166 of test.core.script unit:

        CheckEqual(Run(
          'add(434.732,343.045)'), '777.777');

The evaluation of the javascript expression gives the following result:

<JSVALUE> = { U = { I32 = 777,  TAG = 65536,  F64 = 1.3906711615708398e-309,  PTR = $1000000000309,  U64 = 281474976711433}}

which is apparently integer and the equality check with the float value '777.777' fails.

ab · 2021-07-11 06:47:06

About 777.777, this is really weird.
The JSValue comes directly from the QuickJS C library, which is statically compiled and linked, so I don't understand why there may be something diverse on your PC and others...
I will try to investigate further. But since it is within the QuickJS C code, it is really weird.
Have you:

abouchez@tisab:~/dev/lib2/static/x86_64-linux$ ll quickjs.o
-rw-rw-r-- 1 abouchez 489610513 1.1M Apr  2 21:24 quickjs.o

What is your CPU?

damiand · 2021-07-11 07:07:45

@ab

damian@ifestos:~/mORMot2/static/x86_64-linux> ll quickjs.o
-rw-r--r-- 1 damian users 1072632 Απρ   2 22:24 quickjs.o

My CPU is Intel Core i5-4460 @ 3.20GHz

stj · 2021-07-12 05:42:33

The quickjs issue seems to be a locale problem as tthomas found.

See https://synopse.info/forum/viewtopic.php?id=5933

@ab

damiand seems to have the same issues with the hashes.
This time on intel. But what might be the reason?

Shouldn't we continue in one forum thread?

damiand · 2021-07-12 09:49:06

I confirm that this is a locale related problem. My current locale setting is el_GR.UTF-8. When I run the mormot2tests program inside a container (via DDEV-Local) which by default, has locale C.UTF-8 all tests are passed successfully.

Is it possible to have a locale agnostic static version of quickjs?

Last edited by damiand (2021-07-12 09:54:40)

ab · 2021-07-12 10:52:11

But the QuickJS library itself relies on libc for its parsing... it may be difficult to change the C source code. I will look into it.

I have added the proper SetLocale() call as a SetLibcNumericLocale cross-platform function, and called it from mormot.lib.quickjs.
Please report if it solves the issue for you.

damiand · 2021-07-12 11:38:46

@ab

I pulled the changes from the repository, built and run again the program. Now all tests pass successfully!

ab · 2021-07-12 17:01:40

Now https://github.com/synopse/mORMot2/comm … d7d9915b19 also fixes AES-NI hash, I hope.

mORMot Open Source

#51 2021-02-08 12:05:05

Re: mORMot 2 in Good Shape

#52 2021-02-08 15:26:32

Re: mORMot 2 in Good Shape

#53 2021-02-10 15:59:21

Re: mORMot 2 in Good Shape

#54 2021-02-11 06:20:43

Re: mORMot 2 in Good Shape

#55 2021-02-11 20:11:33

Re: mORMot 2 in Good Shape

#56 2021-02-12 17:12:59

Re: mORMot 2 in Good Shape

#57 2021-02-21 12:42:04

Re: mORMot 2 in Good Shape

#58 2021-02-21 14:24:07

Re: mORMot 2 in Good Shape

#59 2021-03-19 18:27:58

Re: mORMot 2 in Good Shape

#60 2021-03-21 07:21:16

Re: mORMot 2 in Good Shape

#61 2021-03-21 10:48:54

Re: mORMot 2 in Good Shape

#62 2021-03-21 12:14:23

Re: mORMot 2 in Good Shape

#63 2021-03-21 16:27:18

Re: mORMot 2 in Good Shape

#64 2021-03-22 07:38:49

Re: mORMot 2 in Good Shape

#65 2021-03-22 08:14:58

Re: mORMot 2 in Good Shape

#66 2021-03-22 08:15:51

Re: mORMot 2 in Good Shape

#67 2021-03-22 18:53:35

Re: mORMot 2 in Good Shape

#68 2021-03-23 07:13:59

Re: mORMot 2 in Good Shape

#69 2021-03-23 08:08:42

Re: mORMot 2 in Good Shape

#70 2021-07-08 08:58:57

Re: mORMot 2 in Good Shape

#71 2021-07-09 07:59:28

Re: mORMot 2 in Good Shape

#72 2021-07-09 11:57:37

Re: mORMot 2 in Good Shape

#73 2021-07-09 23:44:59

Re: mORMot 2 in Good Shape

#74 2021-07-10 09:46:28

Re: mORMot 2 in Good Shape

#75 2021-07-11 06:47:06

Re: mORMot 2 in Good Shape

#76 2021-07-11 07:07:45

Re: mORMot 2 in Good Shape

#77 2021-07-12 05:42:33

Re: mORMot 2 in Good Shape

#78 2021-07-12 09:49:06

Re: mORMot 2 in Good Shape

#79 2021-07-12 10:52:11

Re: mORMot 2 in Good Shape

#80 2021-07-12 11:38:46

Re: mORMot 2 in Good Shape

#81 2021-07-12 17:01:40

Re: mORMot 2 in Good Shape

Board footer