mORMot2 on ARM

mdoyle · 2022-01-29 16:54:44

I am wondering if ARM will be a focus target for mORMot2 in the future. Linux aarch64 is supported but I believe it is not as maintained as the x64 platform. I do not know if the Oracle cloud is still used in the current development. When running the tests there are a couple of errors.
ARM is a hot topic for Apple and Windows 11 is also available for ARM now.
I have a new Mac M1 and I got FPC and Lazarus running on MacOS which is pretty cool. A client application with ORM and HTTP already works. Next I will test a server app with external sqlite3 libs since there are no arm static files other than for Linux. So, if this interesting I would be happy to help testing and building.

Martin

ab · 2022-01-29 21:24:12

You are right, Oracle closed my "perpetual free account" after a few weeks.
So I don't have acess to the Ampere machine any more.

We could ask Alfred from Mac M1 support with fpcupdeluxe.
I guess the included NDK cross compiler toolchain could be used to build SQLite3 static libraries, as we do for Darwin Intel i386/x86_64.

I need to setup my Raspberry Pi 4 again, and try to fix the regressions in Linux Aarch64.
But was very pleased with the Ampere CPU, and found FPC good enough to start serious work on ARM.

Therefore your feedback is very welcome!

profh · 2022-01-30 04:21:59

thanks to ab and Alfred, i cross compiled my project with mormot1.8 using fpcupdeluxe AARch64 cross compiler under windows 11, it worked fine for months.

i also compiled it with mormot2 using fpcupdeluxe AARch64 cross compiler under windows 11, it worked also.

install fpcupdeluxe on my Raspberry Pi 4, compile it again and it also works.

the key is that i should use free pascal version 3.2.2 and lazarus 2.2.0, higher version will produce kinds of errors which are of my own codes.

thanks to you two again!

Last edited by profh (2022-01-30 04:25:10)

ab · 2022-01-30 06:56:37

I tried to be compatible with the latest FPC trunk this week (for mORMot 2).
https://github.com/synopse/mORMot2/comm … a650cfeb0f
They changed the way strings are encoded in memory (the TStrRec header before the text itself). See https://gitlab.com/freepascal.org/fpc/s … bdd0018f2e

But I didn't test it yet - it "should work" (tm). Feedback is welcome.

AOG · 2022-01-31 10:38:09

The latest mORMot2 (with sqlite3-statics) with the latest FPC trunk (both today) works very well on ARM32 bit.
Will try ARM64 also, if time permits.

ab · 2022-01-31 13:04:13

AOG wrote:

The latest mORMot2 (with sqlite3-statics) with the latest FPC trunk (both today) works very well on ARM32 bit.

Thanks Alfred for the feedback!

Note that I have backported the FPC trunk fix about TStrRec.RefCnt in mORMot 1 this morning.
See https://synopse.info/fossil/info/b701a6f3f93d532a

mdoyle · 2022-02-03 20:58:59

As far as I can see the only issue with aarch64 is TDocVariant. All tests with TDocVariantData like AddValue fail. The tests with variants are fine. I do not know what to do at the moment since Step into AddValue already causes an error message -> log message TSysVariant {Message: TDocVariant.DispInvoke: invalid (1) call“} [Main] at 696618 … variants.pas (3490) variants (3597) variants (8938).

ab · 2022-02-04 11:24:56

This is weird, because TDocVariantData.AddValue does not use the late-binding so won't call DispInvoke.
Your error message means that DispInvoke() was called with a late-binding method with no name.

When occurs exactly the problem (the full call stack and the error line in the test case itself)?
Did you try with FPC 3.2?
By chance, did you force string=UnicodeString?
Perhaps there is an unresolved bug in FPC trunk, or something backward incompatible in trunk DispInvoke. I didn't find anything change with latest source.
Is https://gitlab.com/freepascal.org/fpc/s … 5dfdb6b33b included in your trunk revision?

AOG · 2022-02-05 10:31:49

I can confirm the described problem(s) on aarch64. And, I have a hard time debugging the issue(s).
At the moment, most severe problems (on my system) stem from a memory leak on fastsetstring (and derivatives). String-memory is not released, causing a (out-of-memory) system-crash.
Still looking into this. Will report if any news.

ab · 2022-02-05 10:43:27

FastNewString() uses PStrRec to define the string header.
So it should be fine with the new layout of FPC trunk - if you are using FPC trunk.
Perhaps the memory leak may come from a wrong refCnt.

On AArch64, FPC trunk should compile SizeOf(TStrRec) = 16 (= 2 + 2 + 4 + 8). (STRCNT32 defined)
And FPC 3.2 has SizeOf(TStrRec) = 24. (STRCNT32 not defined)

mdoyle · 2022-02-05 10:54:44

OK. I checked again my setup. FPC is available for aarch64 only since version 3.2.2.
First I cross compiled from FPC 3.2.0 Debian X64 to aarch64 and I did not get the above error messages. All Variant tests were fine.
Second I reinstalled FPC fixes-3.2 and Lazarus fixes-2.2 on Debian aarch64 with FPCUPdeluxe 2.2.0h. The installed versions are now FPC 3.2.3-582-gdba65567f1 and Lazarus 2.2.0-64-gff329cf453. And guess what, no Variant test issues anymore.
There are a few tests left which fail and I will keep checking. Thanks ab and Alfred for the great job.

AOG · 2022-02-05 11:07:16

@ab
Yes, in theory all should work. That is why I am still looking into the real source of the issues mentioned.
Sidenote:
FPC trunk has
TAnsiRec = {packed} Record
TUnicodeRec = {packed} Record
tdynarray = { packed } Record
The packed is not used anymore of these records.

mdoyle · 2022-02-05 16:30:27

Please find the output from the tests on gist https://gist.github.com/martin-doyle/44 … 99bc915c56. Might be also interesting to see the numbers.

ab · 2022-02-05 20:11:36

@Alfred
I suspect it makes no difference to get rid of the "packed" keyword with TStrRec.
But I will delete it in mormot.core.base.pas for FPC, to be in synch with the FPC RTL source code.

@mdoyle
The M1 numbers are very interresting.
Some are very high, like memory copy/fill using clib, Lizard process, OpenSSL AES CTR or GCM... The unified ARM memory model gives very good numbers, even on single thread for sure. They are in pair withthe Ampere number I already published.
We need to understand why some tests do fail. They should not.
Side note: please run the tests a second time. The JSONBEnchmark tests need to have a people.json local file, which is created at the first pass. Then we will know a bit more about UTF8 and JSON performance numbers.

mdoyle · 2022-02-06 08:40:06

I ran the tests several times. Please find the numbers here https://gist.github.com/martin-doyle/8b … c7d68cfd7c.

The failed tests starting at 2.1 ORM Core seem to have the same root cause. The InMemory storage reading existing data - i.e. TTestOrmCore._TRestServerFullMemory line 299 - fails due to duplicate entries. TRestStorageInMemory.LoadFromBinariy loads all entries correctly but in ComputeStateAfterLoad it seems that the PtrInt dup variable is not set in/after fValues.Hasher.ForceReHash(@dup).

ab · 2022-02-06 13:51:11

Thanks to your feedback, I have fixed an integer/PtrInt pointer issue in TRestStorageInMemory.ComputeStateAfterLoad.
The compiler should track that a PInteger is not a PInt64/PPtrInt!

Please check https://github.com/synopse/mORMot2/commit/0cc9d1f2

It should fix the problem you encounter.
Weirdly, it was not triggered on x86_64, but it was an Heisenbug anyway.

mdoyle · 2022-02-06 16:50:35

Wow. AAALLL green now.

Total assertions failed for all test suits: 0 / 71,390,226
! All tests passed successfully.

https://gist.Github.com/martin-doyle/a8 … 7985572920.

ab · 2022-02-06 17:28:38

Good news.

From the tests, it seems that the only place where my Intel Core i5 is slightly faster, is when the heap is involved. For instance, with TDocVariantData.
Our x86_64 asm heap manager is very fast.

Web server numbers are also impressive: on single thread tests, about 200MB/s is twice more than my Core i5...
But the multi thread tests are deceiving. It should scale much better. How many cores did you give to your VM?

BTW the CPU detection code is not working as expected. It just returns aarch64 with no detection at all.
What does cat /proc/cpuinfo return on your VM?

mdoyle · 2022-02-06 19:05:57

Oops. My VM had only 2 cores with 4 GB RAM.
For 4 cores and 8GB it looks like this https://gist.github.com/martin-doyle/e0 … 22075e6de0.

cat /proc/cpuinfo does not give any information about the cpu details only some cryptic numbers. sysctl -a machdep.cpu only works on the macOS terminal. Let me further check.

ab · 2022-02-06 20:27:42

I guess cat /proc/cpuinfo should return something on the VM terminal (not MacOs).

The numbers are weird: some are better, other are worse than your previous tests.
But very good numbers in all cases.

Thanks for the feedback.

ab · 2022-02-08 09:44:31

I tried to enhance the CPU detection with https://github.com/synopse/mORMot2/commit/43c47fa6

mdoyle · 2022-02-08 11:04:23

„4x generic aes sha crc aarch64 cpu“ on Linux.
„8x MacBookPro18,3 (aarch64)“ on MacOS. I just started some tests with the external sqlite 3.36.0 library. Would be interesting to do tests with the static libraries.

AOG · 2022-02-08 13:00:57

Update about FPC trunk + mORMot2 on AArch64 (Odroid C2 running Arch Linux).
The memory leak reported by me earlier seems to be caused by FPC. Even Lazarus crashes (and gets killed) due to huge and increasing memory consumption, even when idle.
Will keep you informed.

ab · 2022-02-08 14:51:35

Thanks you both for your updates.
We stay tuned!

ab · 2022-02-08 18:24:38

@AOG
Is the current fpcupdeluxe able to cross-compile from Linux to Darwin AArch64?
Could we use these cross-compile tools to statically compile the SQlite3 static library?

AOG · 2022-02-09 12:52:44

Final Aarch64 Linux FPC trunk update.
This issue: https://gitlab.com/freepascal.org/fpc/s … sues/38018
This commit: https://gitlab.com/freepascal.org/fpc/s … bdd0018f2e
Are the cause of the failures and memory leaks on current FPC trunk.
If I run the mORMot2 on the previous (see below) commit, all greens, no leaks.
Previous: https://gitlab.com/freepascal.org/fpc/s … 84d1432048
So, no mORMot2 issue !

AOG · 2022-02-09 12:53:47

@ab

Is the current fpcupdeluxe able to cross-compile from Linux to Darwin AArch64?

Yes !

ab · 2022-02-09 13:11:03

What I don't understand is that issues/38018 should be supported by the my https://synopse.info/forum/viewtopic.ph … 626#p36626 commit above.
What if we force the STRCNT32 conditional?

I will try to update to latest FpcUpDeluxe and use FPC 3.2.2 fixes.
When will you publish the next 2.2.0i release?

AOG · 2022-02-09 13:20:25

@ab.
Yes, 38018 is supported by your commit ! Its not a mORMot2 problem.
However, as stated, this is a FPC problem. It seems that the calculation of the header size in the compiler sources is not correct for aarch64.
Lazarus trunk crashes within few minutes. Its memory consumption goes up with an MB per second whenever the source editor is used.

AOG · 2022-02-09 13:22:44

When will you publish the next 2.2.0i release?

Should be within a week. Still waiting for confirmation of some fixes.

ab · 2022-02-09 13:27:48

Thanks for the feedback.

Very interesting.

AOG · 2022-02-10 10:26:45

Bugger reported.
https://gitlab.com/freepascal.org/fpc/s … sues/39569

AOG · 2022-02-12 09:26:27

Bugger solved.
https://gitlab.com/freepascal.org/fpc/s … 1dfbc523a8

All greens now on Linux AArch64 with FPC trunk. Very good !
However, a small workaround for the FPC source parser was needed.
In function GetJsonField, a label (lit) is used for a goto. This results in code errors on AArch64.
If the label is removed, all is ok and 100% error free mORMot2 can be enjoyed.

ab · 2022-02-12 10:47:17

Very good news!

So I guess https://github.com/synopse/mORMot2/commit/04050181 would do the trick for AARCH64.
It may enhance a little bit the parsing performance, too.

Thanks for the feedback.

AOG · 2022-02-12 11:25:45

Your "trick" works !
Thanks.

mdoyle · 2022-02-13 09:13:46

I kept testing on MacOS aarch64.

First, I compiled the newest sqlite3 3.37.2 and the installation worked without any problem. Then I thought, maybe compiling the static libraries works as well. I could compile armv8.c from the res directory. This made building the mormot2 package possible.
Compiling the sqlite3 library was a little trickier. I had to patch the amalgamation file on Linux since „sed“ works a little different on MacOS.

To make a long story short, I could compile and run the mormot tests with static libraries.
I had to disable a couple of tests which froze but it is a good start. https://gist.github.com/martin-doyle/0f … e9dacdc252

A first issue I found was a segmentation fault when creating the BloomFilter. ln(fFalsePositivePercent / 100) crashes. ln(0.01) works just fine. I created a little fpc test program in order to see if it is a fpc, type whatever issue. But no problem here - ln seems to work as expected but behaves strangely within the Create method of Bloomfilter.

sstillwell · 2022-04-18 20:51:13

ab wrote:

You are right, Oracle closed my "perpetual free account" after a few weeks.
So I don't have acess to the Ampere machine any more.
We could ask Alfred from Mac M1 support with fpcupdeluxe.
I guess the included NDK cross compiler toolchain could be used to build SQLite3 static libraries, as we do for Darwin Intel i386/x86_64.
I need to setup my Raspberry Pi 4 again, and try to fix the regressions in Linux Aarch64.
But was very pleased with the Ampere CPU, and found FPC good enough to start serious work on ARM.
Therefore your feedback is very welcome!

It would be worth trying to re-create your Oracle account - I currently have two Ampere VMs running with 2 cores and 12 GB RAM each - they're quite performant! I haven't been able to use fpcupdeluxe on them yet to build a working environment - I get a access violation that I can continue execution past, but the lists of FPC and Lazarus version numbers are invisible or incompletely populated. I also run an M1 Mac Mini as my everyday machine and I DO have it built there and working quite well.

If I can be of assistance, let me know!

Scott

sstillwell · 2022-04-19 03:00:23

I'll counter my offer of help with a request for help...it appears that most or all of the required static libs are not included in the downloadable libraries in aarch64-darwin format, nor is there an easy way to build them. It appears that several of the build scripts are assuming that they'll use cross compilers from Windows to the target format, and that syntax won't work when compiling natively on macOS. For example, using AT&T syntax in .s files - gcc on recent macOS is using clang/llvm and doesn't understand the .type or .size directives that I can tell.

The documentation is very comprehensive from a design standpoint, but it appears to be a miss that there isn't an installation guide. My apologies if that offends. My "day job" is as a consultant/systems engineer installing, upgrading, and troubleshooting ERP systems, and I'm used to digging through documentation to find THOSE documents, so that's where I'm coming from. I've done quite a bit of development in the past, but it's also quite a while in the past, so I'm a bit rusty.

If I'm missing the information stashed somewhere on the site, in the forums, in documents, wherever, just let me know and I'll go dig them up - I just haven't found them so far.

AOG · 2022-04-19 05:12:21

but the lists of FPC and Lazarus version numbers are invisible or incompletely populated

This is probably due to the fact that fpcupdeluxe is not able to create some initial settings files. Create a folder somewhere (desktop) and please move fpcupdeluxe to this directory of your choice. The same is needed on Apple (when fpcupdeluxe is run from the Download folder).

dcoun · 2022-04-19 05:40:22

Ubuntu 22 LTS is optimized for Raspberry 4 too, getting out this week

sstillwell · 2022-04-19 06:32:11

AOG wrote:

but the lists of FPC and Lazarus version numbers are invisible or incompletely populated
This is probably due to the fact that fpcupdeluxe is not able to create some initial settings files. Create a folder somewhere (desktop) and please move fpcupdeluxe to this directory of your choice. The same is needed on Apple (when fpcupdeluxe is run from the Download folder).

That appears to be part of the problem, thanks. I had it in /usr/local/bin and moved it into my home directory - after that the error went away. One issue down, more to go.

FPC and Lazarus both build and install, and mORMot2 and mORMot2ui packages build and install into Lazarus...which is more than they do on M1 macOS 12. For some reason, the install of mORMot2ui into Lazarus on M1 macOS fails looking for the aarch64-linux version of armv8.o. Even if I download latest static libs, there is no aarch64-darwin directory in it.

Last edited by sstillwell (2022-04-19 07:00:17)

ab · 2022-04-19 07:28:46

Note that AARCH64 armv8.o is now only validated on Linux. It would not try to use it any more on Mac M1.
https://github.com/synopse/mORMot2/commit/abdc3eb2 and
https://github.com/synopse/mORMot2/commit/2e19dfcd

And aarch64-darwin target is not supported yet in the static folder.
But if you use the system SQLite3 library (.so) it may work.
I don't have access to any M1 system, so I can't say how much...

sstillwell · 2022-04-19 13:53:14

ab wrote:

Note that AARCH64 armv8.o is now only validated on Linux. It would not try to use it any more on Mac M1.
https://github.com/synopse/mORMot2/commit/abdc3eb2 and
https://github.com/synopse/mORMot2/commit/2e19dfcd
And aarch64-darwin target is not supported yet in the static folder.
But if you use the system SQLite3 library (.so) it may work.
I don't have access to any M1 system, so I can't say how much...

Okay, then that means that the packages can't be installed into Lazarus since that requires static build, correct? I'm a bit uncertain as to how to change to using .so if it's allowed.

EDIT: Okay, I've looked at the commits. I'll see if I can figure out how to pull latest mORMot and build on M1. In the meantime I have it running on AARCH64 Linux on my Oracle Cloud Ampere VM - it's a little slower with only 2 cores and doesn't have access to the DB servers here in-house, but with 12 GB of RAM I can install Postgresql or MariaDB or whatever and tinker with it there locally, or with just plain SQLite itself, of course.

Last edited by sstillwell (2022-04-19 15:41:50)

ab · 2022-04-19 17:33:08

All dependent libraries (e.g. SQLite3) are loaded using late binding, so nothing should prevent you to install the packages in Lazarus.

sstillwell · 2022-04-19 18:03:02

For what it's worth, fpcupdeluxe has a failure on macOS when installing mORMot2 as:

fpcupdeluxe: info: mORMot2Installer (GetModule: mORMot2): Going to download 2.0.181 of mormot sqlite3 static libs [mormot2static.7z] from https://github.com/synopse/mORMot2/releases/download/2.0.181/mormot2static.7z
Download progress FPCUPTMP00000.7z: starting.
Download progress FPCUPTMP00000.7z: Starting download.
Download progress FPCUPTMP00000.7z: download ready !
fpcupdeluxe: info: mORMot2Installer (GetModule: mORMot2): Download ok. Extracting files. Please wait.
fpcupdeluxe: WARNING: mORMot2Installer (GetModule: mORMot2): Unpack of /Users/scott/fpcupdeluxe/tmp/FPCUPTMP00000.7z failed with resultcode: -1
fpcupdeluxe: info: mORMot2Installer (GetModule: mORMot2): Getting mORMot2 (statics) failure. Will continue anyhow.

Manually running ./get_latest_static.sh works properly, assuming you've used homebrew to install p7zip package to give you the 7za command. This is the kind of thing that needs to go in an installation guide.

sstillwell · 2022-04-19 18:10:31

Should the following be in mormot.lib.openssl11?

        {$ifdef CPUX64}
        LIB_CRYPTO = 'libssl-merged-osx64.dylib';
        LIB_SSL = 'libssl-merged-osx64.dylib';
        _PU = '_';
        {$endif CPUX64}
        {$ifdef CPUX64_static}
        LIB_CRYPTO = 'libcrypto-osx64.a';
        LIB_SSL = 'libssl-osx64.a';
        _PU = '';
        {$define OPENSSLSTATIC}
        {$endif CPUX64}
        {$ifdef CPUAARCH64}
        LIB_CRYPTO = 'libssl-merged-osx64.dylib';
        LIB_SSL = 'libssl-merged-osx64.dylib';
        _PU = '_';
        {$endif CPUAARCH64}
        {$ifdef CPUAARCH64_static}
        LIB_CRYPTO = 'libcrypto-osx64.a';
        LIB_SSL = 'libssl-osx64.a';
        _PU = '';
        {$define OPENSSLSTATIC}
        {$endif CPUAARCH64}

The compile of the tests project fails since CPUAARCH64 isn't allowed for in the code.

sstillwell · 2022-04-19 20:09:18

I'll continue this in another thread - I don't want to clog this up with my issues.

mORMot Open Source

#1 2022-01-29 16:54:44

mORMot2 on ARM

#2 2022-01-29 21:24:12

Re: mORMot2 on ARM

#3 2022-01-30 04:21:59

Re: mORMot2 on ARM

#4 2022-01-30 06:56:37

Re: mORMot2 on ARM

#5 2022-01-31 10:38:09

Re: mORMot2 on ARM

#6 2022-01-31 13:04:13

Re: mORMot2 on ARM

#7 2022-02-03 20:58:59

Re: mORMot2 on ARM

#8 2022-02-04 11:24:56

Re: mORMot2 on ARM

#9 2022-02-05 10:31:49

Re: mORMot2 on ARM

#10 2022-02-05 10:43:27

Re: mORMot2 on ARM

#11 2022-02-05 10:54:44

Re: mORMot2 on ARM

#12 2022-02-05 11:07:16

Re: mORMot2 on ARM

#13 2022-02-05 16:30:27

Re: mORMot2 on ARM

#14 2022-02-05 20:11:36

Re: mORMot2 on ARM

#15 2022-02-06 08:40:06

Re: mORMot2 on ARM

#16 2022-02-06 13:51:11

Re: mORMot2 on ARM

#17 2022-02-06 16:50:35

Re: mORMot2 on ARM

#18 2022-02-06 17:28:38

Re: mORMot2 on ARM

#19 2022-02-06 19:05:57

Re: mORMot2 on ARM

#20 2022-02-06 20:27:42

Re: mORMot2 on ARM

#21 2022-02-08 09:44:31

Re: mORMot2 on ARM

#22 2022-02-08 11:04:23

Re: mORMot2 on ARM

#23 2022-02-08 13:00:57

Re: mORMot2 on ARM

#24 2022-02-08 14:51:35

Re: mORMot2 on ARM

#25 2022-02-08 18:24:38

Re: mORMot2 on ARM

#26 2022-02-09 12:52:44

Re: mORMot2 on ARM

#27 2022-02-09 12:53:47

Re: mORMot2 on ARM

#28 2022-02-09 13:11:03

Re: mORMot2 on ARM

#29 2022-02-09 13:20:25

Re: mORMot2 on ARM

#30 2022-02-09 13:22:44

Re: mORMot2 on ARM

#31 2022-02-09 13:27:48

Re: mORMot2 on ARM

#32 2022-02-10 10:26:45

Re: mORMot2 on ARM

#33 2022-02-12 09:26:27

Re: mORMot2 on ARM

#34 2022-02-12 10:47:17

Re: mORMot2 on ARM

#35 2022-02-12 11:25:45

Re: mORMot2 on ARM

#36 2022-02-13 09:13:46

Re: mORMot2 on ARM

#37 2022-04-18 20:51:13

Re: mORMot2 on ARM

#38 2022-04-19 03:00:23

Re: mORMot2 on ARM

#39 2022-04-19 05:12:21

Re: mORMot2 on ARM

#40 2022-04-19 05:40:22