mORMot2 (Latest Release) regression test on Win64 failed. Critical?

stj · 2021-07-07 15:04:17

Hi.

I just wanted to start (again) with mORMot ...

Installed FPC 3.2.0 and Lazarus 2.0.12 with fpcupdeluxe and
downloaded the latest mORMot2 release (mORMot2-sqlite.3.35.5) with the corresponding static files (mormot2static.7z).

Compiling mormot2.lpk and fpcunittestrunner.lpk was fine.
Compiling mormot2test.lpi went fine.

Running mormot2test.exe gives

1.3. Core crypto:
...
! - Hashes: 3 / 9 FAILED 692.09ms
...

and

Generated with: Free Pascal 3.2 64 bit compiler

Time elapsed for all tests: 11m11
Performed 2021-07-07 17:02:01 by Stephan on STEPHAN-NB2

Total assertions failed for all test suits: 3 / 41,061,139
! Some tests FAILED: please correct the code.

Is this critical? Can I help to find the problem?

Logfile on demand.

Regards
Stephan

Last edited by stj (2021-07-07 15:10:07)

ab · 2021-07-07 16:51:02

I am not able to reproduce it with FPC 3.2 and current mORMot 2 trunk.

Ensure you download the latest source code from github - the one included with the mORMOt 2 release may have some problems.

stj · 2021-07-08 04:56:05

Too bad.
Same with latest code from github.

0000000000000001 ! fail #5 ... test.core.crypt.pas ttestcorecrypto.hashes (1240) ...
0000000000094C41 ! fail #7 ... test.core.crypt.pas ttestcorecrypto.hashes (1243) ...
000000000009AD2D ! fail #9 .. test.core.crypt.pas ttestcorecrypto.hashes (1246) ...

Any idea?

Edit:

Just restarted with a complete fresh installation.
Same problem.

Failed tests leads to AES-NI hasher in _AesNiHashXmm0.

Is it probably a CPU problem?

CPU is AMD A8-7410 with Radeon R5 graphics.
Family F, Model 0, Stepping 1, Ext. Family 16, Ext. Model 30, Revision ML-A1

SSE4.2 and AES is supported according CPU-Z (www.cpuid.com).

At least I would like to disable AES-NI hasher to get started.
Is there a global compiler define for this (maybe USEAESNIHASH) and where should I unset it?

Last edited by stj (2021-07-08 10:38:20)

ab · 2021-07-08 20:52:31

Could you find where exactly the test fail: on which byte of the input buf[] content?

I have no AMD so I can't reproduce the problem.
On my Core i5 CPU I don't have any such issue with AesNi hashes.

Please also try on Win32 and Linux x86_64 if you can.

stj · 2021-07-09 08:30:30

Did a quick test:

Hash32Test: Failed. L = 249 at modif = 120
Hash64Test: Failed. L = 249 at modif = 120
Hash128Test: Failed. L = 249 at modif = 120

I'm just setting up a VM with Ubuntu for a Liinux test and will setup a cross-compiler for Win32.
Stay tuned :-)

stj · 2021-07-09 09:52:16

Hm. I do not have a real Win32 at hand, so I installed a cross-compiler for i386-win32.

All hashes-tests passed but this gave me alot of other failed tests with a logfile of approx. 1.5 MB :-(

Shouldn't a crosscompile work?

Last edited by stj (2021-07-09 09:53:13)

ab · 2021-07-09 10:02:01

I cross compile daily from Linux x86_64 to i386-win32 with no problem...
Please check https://gist.github.com/synopse/8ffdb40 … 95b64ff61f

Did you run the tests once with administrator rights?
It is necessary on Windows to register the http.sys web server URI and ports.

stj · 2021-07-09 10:58:08

Hm. Checked what you wrote.
I'm using Windows 10 - 64 bit and mORMot2.

I do not know how to register http.sys in this environment. Is this needed?

From mORMot 1.18 in TestSQL3Register.dpr:

{ in order to be able to use http.sys server for TestSQL3.exe under
Vista or Seven, call first this program with Administrator rights
- you can unregister it later with command line parameter /delete }

This is not in mORMot 2.

Edit:

Same problem with Hashes in a brand new Debian 64-bit VM with FPC 3.2.
And additionally in 3.1 Core Script a lot of

#xxxx 777 <> 777.777

see https://gist.github.com/stj-mv/68e79698 … 8230681159

Edit:

Crosscompiled on linux x86-64 for win 64 gives the same as native on win 64.
Test for the hashes failed.

For me this sounds really like a CPU problem ...

Edit:

Crosscompiled on Win 64 for linux x86-64 gives lots of failed tests.
Most of them seems to be related to floating point values.

see https://gist.github.com/stj-mv/e49fd632 … f0f5933a16

Last edited by stj (2021-07-09 13:18:37)

ab · 2021-07-09 14:15:01

0. If you run mormot2tests with adminstrative rights once, it is enough. It includes TestSQL3Register

1. Please try to run the tests without heaptrc.

2. Never install a Win64 FPC compiler: install Win32 version of the FPC compilers on Windows.

3. What is weird is that the regular AES vector tests do pass. Only the AESNIHash has troubles.

stj · 2021-07-09 15:03:44

ab wrote:

0. If you run mormot2tests with adminstrative rights once, it is enough. It includes TestSQL3Register

Fine. Always did it as Administrator (on Windows) or as root (on Linux)

ab wrote:

1. Please try to run the tests without heaptrc.

Same problems in native Linux x86_64 without heaptrc.
Hashes failed and this irritating "777 <> 777.777".

ab wrote:

2. Never install a Win64 FPC compiler: install Win32 version of the FPC compilers on Windows.

Oh. Didn't know that. Are there any known limitations with Win64 FPC compiler?

As I said ...
Crosscompiled on linux x86-64 for win 64 gives the same as native on win 64.
Test for the hashes failed.

I used a i386-win32 cross-compiler running in FPC 64bit on Windows.
Shouldn't this work?

ab wrote:

3. What is weird is that the regular AES vector tests do pass. Only the AESNIHash has troubles.

What about those lots of floating point problems?
Is this related?

Edit:

Compiled native with 32-bit FPC on Windows 64:

Same problems with the AES-NI hashes.
All other tests pass.

Last edited by stj (2021-07-09 15:27:43)

ab · 2021-07-09 15:18:33

(no need on Linux - this root registration is a Windows/http.sys specific thing)

Perhaps the FPU flags are not correctly set on Win64.
I am reviewing the FPU flags to be used between C code and pascal code (not the same about exceptions).
So I don't think it is related to AES-NI.

Win64 FPC compiler has troubles with "currency" and "extended" constants.

stj · 2021-07-09 15:29:56

I see. This might be fixed in 3.2.2.
But as I've read ... there are problems with variants.

Well, this is software :-)

ab · 2021-07-09 20:42:45

Could you try to download and run https://synopse.info/tmp/mormot2tests.zip on Linux x86_64 ?
On my computer there is no issue:
https://gist.github.com/synopse/30c68ff … 060ceac356
On the synopse server (slow Atom) either:
https://gist.github.com/synopse/83250a8 … b227b3cc6d

So we will see if this is a compiler problem or a CPU issue.

stj · 2021-07-10 07:44:22

Still not working.

Hashes are ok but issues at 2.8 and 3.1.

see https://gist.github.com/stj-mv/267b6d37 … 67434d8919

My compiler version on linux is 3.2.0 Rev. 45643 build from sources with fpcupdeluxe 1.8.2t

stj · 2021-07-10 10:58:40

In short the test passes only with 32bit compiler on windows
and only with a 32bit target on 64bit linux with 64bit compiler.

Here's the summary:

Compiler FPC 3.0.2 32bit on Windows 10, 64bit
- Target Windows 32, i386 - native, Run on Windows 64
PASS
- Target Windows 64, x86_64 - cross compiled, Run on Windows 64
FAILED: Hashes at 1.3
- Target Linux, x86_64 - cross compiled, Run on Debian VM, 64bit
FAILED: Hashes at 1.3, Core Script at 3.1

Compiler FPC 3.0.2 64bit on Windows 10, 64bit

- Target Windows 32, i386 - cross, Run on Windows 64
FAILED: Hashes OK, Lots of errors related to floating point or currency
- Target Windows 64, x86_64 - native, Run on Windows 64
FAILED: Hashes at 1.3
- Target Linux, x86_64 - cross compiled, Run on Debian VM, 64bit
FAILED: Hashes at 1.3, Lots of errors related to floating point or currency

Compiler FPC 3.0.2 64bit on Debian 10 VM, 64bit

- Target Windows 32, i386 - cross compiled, Run on Windows 64
PASS
- Target Windows 64, x86_64 - cross compiled, Run on Windows 64
FAILED: Hashes at 1.3
- Target Linux, x86_64 - native, Run on Debian VM, 64bit
FAILED: Hashes at 1.3, Core Script at 3.1

AOG · 2021-07-10 12:03:02

Compiler FPC 3.0.2

I guess and hope you mean FPC 3.2.0 ?

Crossing from win64 to win32 is very ill advised and not supported by both FPC and mORMot[2].

As I am myself running the mORMot[2] with FPC trunk on various systems, it would be interesting (for me) to see your results when using trunk.

As a sidenote: with trunk I am also experiencing some errors with Core Script at 3.1 ! Its caused by this line (test.core.script.pas):

v := Run('Date() + "=" + new Date().toString()');

If I comment this line (and the accompanying next line), the tests run fine.

AOG · 2021-07-10 12:35:16

FPCtrunk

stj · 2021-07-10 13:57:30

@AOG

Of cause I'm running FPC 3.2.0.
Just a typo and copy & paste :-)

Commenting out a test looks a little bit suspicious to me.
A testsuite is for me a "don't touch the source".
IMO test are there to assure the expected behaviour.

AOG · 2021-07-10 14:18:28

You report an error. I did have a look at this error. And tested the mORMot2 with FPC trunk. On my system, the error seems to be related to a particular statement. If the same is valid on your system, you have helped us in isolating this error. And making the bug-hunt and bug-solving a bit easier. If the bug is fixed, the test-suit will run again at 100% greens.

stj · 2021-07-10 14:23:59

AOG wrote:

You report an error. I did have a look at this error. And tested the mORMot2 with FPC trunk. On my system, the error seems to be related to a particular statement. If the same is valid on your system, you have helped us in isolating this error. And making the bug-hunt and bug-solving a bit easier. If the bug is fixed, the test-suit will run again at 100% greens.

OK. I see your point :-)
I'll cross check if this is the same statement failing here :-)

stj · 2021-07-10 14:31:16

AOG wrote:

Compiler FPC 3.0.2
I guess and hope you mean FPC 3.2.0 ?
Crossing from win64 to win32 is very ill advised and not supported by both FPC and mORMot[2].
As I am myself running the mORMot[2] with FPC trunk on various systems, it would be interesting (for me) to see your results when using trunk.
As a sidenote: with trunk I am also experiencing some errors with Core Script at 3.1 ! Its caused by this line (test.core.script.pas):
v := Run('Date() + "=" + new Date().toString()');
If I comment this line (and the accompanying next line), the tests run fine.

Well, this isn't the case here. This one works fine.

AOG · 2021-07-10 14:46:47

Well, this isn't the case here. This one works fine.

Please state the system you used to test. I presume you used 64bit FPC trunk of today running on Windows 10 64 bit ?

stj · 2021-07-10 15:04:07

AOG wrote:

Well, this isn't the case here. This one works fine.
Please state the system you used to test. I presume you used 64bit FPC trunk of today running on Windows 10 64 bit ?

No. I use, as stated by ab, the officially supported FPC Version 3.2.0 Rev. 45643, linux_x86_64, running on a 64bit debian 10. CPU is AMD.

stj · 2021-07-10 15:14:51

I nailed it down :-)

On line 166/167 in test.core.script.pas

        CheckEqual(Run(
          'add(434.732,343.045)'), '777.777');

fails.

I'm nearly sure this is kind of a rounding or FPU problem.
What CPU are you using? Intel or AMD?

ab · 2021-07-10 20:23:27

If hashes are OK with my executable, I guess this is because there is something wrong with your FPC compiler.
The only difference is the compiler.
So at least, we don't have AES-NI issue with your AMD CPU.

About 777.777, this is really weird.
The value comes directly from the QuickJS C library, which is statically compiled and linked, so I don't understand why there may be something diverse on your PC and others...
I will try to investigate further. But since it is within the QuickJS C code, it is really weird.
Have you:

abouchez@tisab:~/dev/lib2/static/x86_64-linux$ ll quickjs.o
-rw-rw-r-- 1 abouchez 489610513 1.1M Apr  2 21:24 quickjs.o

stj · 2021-07-11 07:13:44

ab wrote:

If hashes are OK with my executable, I guess this is because there is something wrong with your FPC compiler.
The only difference is the compiler.
So at least, we don't have AES-NI issue with your AMD CPU.

Hm. I thought building the FPC with fpcupdeluxe is a reasonable way ...
The FPC testsuite gave the expected and known errors and problems.
Could you give me the exact revision of your compiler? (fpc -iW)

ab wrote:

About 777.777, this is really weird.
The value comes directly from the QuickJS C library, which is statically compiled and linked, so I don't understand why there may be something diverse on your PC and others...
I will try to investigate further. But since it is within the QuickJS C code, it is really weird.
Have you:
abouchez@tisab:~/dev/lib2/static/x86_64-linux$ ll quickjs.o
-rw-rw-r-- 1 abouchez 489610513 1.1M Apr  2 21:24 quickjs.o

Just checked: Same date, size is 1072632 bytes.
I downloaded mormot2static.7z on 8. july. Size is 18989699 bytes.

stj · 2021-07-11 08:23:52

Hm.
I just downloaded the official FPC/Lazarus release 3.2.0 from Sourceforge (via the links on https://www.lazarus-ide.org/index.php?page=downloads).
Installed them on my Debian VM.

Same Problem.
Hashes and Script failed the test.

I've no more ideas

I'll look if I find an unused PC with a real Debian on AMD.
Should be somewhere in the attic ...

Last edited by stj (2021-07-11 08:26:03)

ttomas · 2021-07-11 12:50:19

@ab @stj, about 777.777, I have the same error, this is not cpu related only Locales settings, I change in Manjaro Settings Manager - Locale Settings change settings for Numbers and Currency to en_US and test pass no more 777<>777.777. How many times we will have problem with decimal dot or comma, my country locale is decimal comma :-)
Check your /etc/locale.conf

Last edited by ttomas (2021-07-11 13:52:38)

stj · 2021-07-11 14:54:50

ttomas wrote:

@ab @stj, about 777.777, I have the same error, this is not cpu related only Locales settings, I change in Manjaro Settings Manager - Locale Settings change settings for Numbers and Currency to en_US and test pass no more 777<>777.777. How many times we will have problem with decimal dot or comma, my country locale is decimal comma :-)
Check your /etc/locale.conf

Sorry ttomas. I always thought that first ...
The result of

add(434.732,343.045)

should be 777.777 (or maybe something else as decimal sep) but
it returns a plain 777 without decimals.

So I don't think this is a locale problem.
(BTW LANG=de_DE.UTF8 and no other LC_* are set)

ttomas · 2021-07-11 16:10:21

It's strange I know, 777 come from quickjs, after change locale test fails, but after linux restart test is OK. Try reboot.
I return locale for numbers to my country and test fail again after reboot. Look like quickjs use locale internally for some conversions!

ttomas · 2021-07-11 16:27:39

My locale.conf is:
LANG=en_US.UTF-8
LC_NUMERIC=mk_MK.UTF-8
LC_TIME=mk_MK.UTF-8
LC_PAPER=mk_MK.UTF-8
LC_NAME=mk_MK.UTF-8
LC_ADDRESS=mk_MK.UTF-8
LC_TELEPHONE=mk_MK.UTF-8
LC_MEASUREMENT=mk_MK.UTF-8
LC_IDENTIFICATION=mk_MK.UTF-8

English language with locale formatings only. With this config I have 1000 fails 777<>777.777
When I change locales for numbers line LC_NUMERIC= is missing from locale.conf

 3.1. Core script: 
  - Quick JS low level: 40,057 assertions passed  1.10s
  Total failed: 0 / 40,057  - Core script PASSED  1.10s

Try with
LANG=de_DE.UTF8
LC_NUMERIC=en_US.UTF-8

Last edited by ttomas (2021-07-11 16:32:07)

stj · 2021-07-11 16:53:42

ttomas wrote:

My locale.conf is:
LANG=en_US.UTF-8
LC_NUMERIC=mk_MK.UTF-8
LC_TIME=mk_MK.UTF-8
LC_PAPER=mk_MK.UTF-8
LC_NAME=mk_MK.UTF-8
LC_ADDRESS=mk_MK.UTF-8
LC_TELEPHONE=mk_MK.UTF-8
LC_MEASUREMENT=mk_MK.UTF-8
LC_IDENTIFICATION=mk_MK.UTF-8
English language with locale formatings only. With this config I have 1000 fails 777<>777.777
When I change locales for numbers line LC_NUMERIC= is missing from locale.conf
 3.1. Core script: 
  - Quick JS low level: 40,057 assertions passed  1.10s
  Total failed: 0 / 40,057  - Core script PASSED  1.10s
Try with
LANG=de_DE.UTF8
LC_NUMERIC=en_US.UTF-8

Wow @ttomas. Well done

I just tried it with LANG=en_US and ...

- Quick JS low level: 40,057 assertions passed 322.41ms
Total failed: 0 / 40,057 - Core script PASSED 322.46ms

This is a workaround but IMHO quickjs shouldn't be affected by the user's locale.

@ab: Do you know about this issue or any preconditions for quickjs?

mpv · 2021-07-11 21:09:15

Usually such errors comes from C printf/scanf with %f formatting. I saw such with many libs. In case
LC_ALL=C test
solves a problem, consider to put
setlocale(LC_ALL, 'C');
in the beginning of program.

stj · 2021-07-12 08:41:40

@ab

I think I've found the problem whith the hashes.

I setup my system on linux with official FPC 3.2.0 release from sourceforge.

Compiling mormot2's trunk this morning with supplied script (build_fpc.sh):
All tests pass (except the known quickjs issue)

Compiling mormot2's trunk within lazarus:
Hashes failes as before.

As far as I can see, the lazarus project options do not match with the options used in the script.

Maybe the mormot2test.lpi should be adjusted to match the build script for those dumb users like
me who try to work with an IDE

Happy that this one is solved for me. Now I start evaluating ...

Last edited by stj (2021-07-12 08:43:31)

ab · 2021-07-12 10:14:07

@stj
This is weird because I use the IDE and the mORMot 2 package here, and mormot2test.lpi project...

@mpv
Do you think we should try to add setlocale(LC_ALL, 'C') within mormot.lib.static.pas ?
I have added it as a SetLibcNumericLocale cross-platform function, and called it from mormot.lib.quickjs.

ttomas · 2021-07-12 10:58:33

Better to report Issue in quickjs repo.
If you add setlocale(LC_ALL, 'C') this will impact every app using mORMot2 or can be set only for quickjs?

OK I see you edit post, SetLibcNumericLocale is nice workaround.

Last edited by ttomas (2021-07-12 11:01:58)

stj · 2021-07-12 12:25:51

ab wrote:

@stj
This is weird because I use the IDE and the mORMot 2 package here, and mormot2test.lpi project...

Found the problem.
The script uses -O2, the .lpi from github uses -O1.

Changing optimization in lazarus project to -O2 solved the failing hashes-test.

With your change, also the failing quickjs-test passed with my locale (de_DE.UTF-8).

Thanks for your efforts.

mpv · 2021-07-12 15:53:12

setlocale for LC_NUMERIC and not for LC_ALL do not help with dates . While there is no issue with dates, it is ok.

Last edited by mpv (2021-07-12 16:01:01)

mpv · 2021-07-12 16:06:40

In case results from O1 and O2 optimisation level differs, this mostly mean some local variable is not initialized propertly. Or wrong parameter type for some C function. @ab - remember an issue with SynDBOracle where we incorrectly define a size of parameter....

ab · 2021-07-12 17:00:00

I guess the O1/O2 problem is now fixed with AesNiHash.
https://github.com/synopse/mORMot2/comm … d7d9915b19
(it also includes some optimizations for smaller size)

There was an data alignment issue in the GoLang RTL I used as reference - a bug in it which has been fixed lately.

Note that I usually compile for Linux x86_64 as -O3 with no problem, for production code.

About LC_NUMERIC it is enough for QuickJS, I guess.

stj · 2021-07-13 07:26:06

ab wrote:

I guess the O1/O2 problem is now fixed with AesNiHash.
https://github.com/synopse/mORMot2/comm … d7d9915b19
(it also includes some optimizations for smaller size)

There was an data alignment issue in the GoLang RTL I used as reference - a bug in it which has been fixed lately.
Note that I usually compile for Linux x86_64 as -O3 with no problem, for production code.
About LC_NUMERIC it is enough for QuickJS, I guess.

Sorry to say, but still the same problem with O1 at least on Win64.

Passes with O2/O3

Last edited by stj (2021-07-13 07:26:46)

ab · 2021-07-13 16:39:58

I am not able to reproduce the problem with O0 or O1 on Win64...

stj · 2021-07-14 07:23:28

ab wrote:

I am not able to reproduce the problem with O0 or O1 on Win64...

Maybe I wasn't clear

The default settings in the package is -O3. and in mormot2test is -O1.
The build script uses -O2.

I did some tests wit always recompiling the package and mormot2test.
For each test I adjusted the optimization settings for package and mormot2test.

On Windows 10, 64bit:

If I compile package and mormot2test with -O0 or -O1 the hashes test fails.

If I compile package and mormot2test with -O2 or -O3 the hashes test pass.

On Linux, 64bit:

All tests pass with -O0 to -O3.

mORMot Open Source

#1 2021-07-07 15:04:17

mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#2 2021-07-07 16:51:02

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#3 2021-07-08 04:56:05

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#4 2021-07-08 20:52:31

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#5 2021-07-09 08:30:30

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#6 2021-07-09 09:52:16

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#7 2021-07-09 10:02:01

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#8 2021-07-09 10:58:08

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#9 2021-07-09 14:15:01

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#10 2021-07-09 15:03:44

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#11 2021-07-09 15:18:33

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#12 2021-07-09 15:29:56

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#13 2021-07-09 20:42:45

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#14 2021-07-10 07:44:22

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#15 2021-07-10 10:58:40

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#16 2021-07-10 12:03:02

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#17 2021-07-10 12:35:16

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#18 2021-07-10 13:57:30

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#19 2021-07-10 14:18:28

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#20 2021-07-10 14:23:59

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#21 2021-07-10 14:31:16

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#22 2021-07-10 14:46:47

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#23 2021-07-10 15:04:07

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#24 2021-07-10 15:14:51

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#25 2021-07-10 20:23:27

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#26 2021-07-11 07:13:44

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#27 2021-07-11 08:23:52

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#28 2021-07-11 12:50:19

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#29 2021-07-11 14:54:50

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#30 2021-07-11 16:10:21

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#31 2021-07-11 16:27:39

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#32 2021-07-11 16:53:42

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#33 2021-07-11 21:09:15

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#34 2021-07-12 08:41:40

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#35 2021-07-12 10:14:07

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#36 2021-07-12 10:58:33

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#37 2021-07-12 12:25:51

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#38 2021-07-12 15:53:12

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#39 2021-07-12 16:06:40

Re: mORMot2 (Latest Release) regression test on Win64 failed. Critical?

#40 2021-07-12 17:00:00