mORMot.pas

ComingNine · 2014-11-01 09:11:13

Can you confirm that you have succeeded in cross-compiling mORMot from win32 to linux32 with static (linux32)library of SQLite ?

I still get complaints

c:\codetyphon\fpc\fpc32\bin\i386-win32\i386-linux-ld.exe: warning: link.res contains output sections; did you forget -T?
c:\codetyphon\fpc\fpc32\bin\i386-win32\i386-linux-ld.exe: cannot find /lib/libpthread.so.0
c:\codetyphon\fpc\fpc32\bin\i386-win32\i386-linux-ld.exe: cannot find /usr/lib/libpthread_nonshared.a
LinuxSynTestFPCLinuxi386.lpr(22,1) Error: Error while linking

The reason seems to be that the static linux32 library of SQLite, which is compiled under a VMWare CentOS 5.11 x86, has internal linkage to "/lib/libpthread.so.0" and "/usr/lib/libpthread_nonshared.a" ?

Last edited by ComingNine (2014-11-01 09:21:55)

AOG · 2014-11-01 09:31:16

Yes, I can confirm. All apps for Linux I am building now are static linked against SQLite !

My advice for crosscompiling:

Start with the most easy FPC program to check your setup:

program Project1;
uses
  cthreads;
begin
  writeln('Hello world. Press enter to continue');
  readln;
  writeln('Thank you !');
end.

Next, start with easy dynamic linking for mORMot to check your mORMot setup.

Next, compile your own sqlite3fts3.o with the provided Linux-script on your Linux machine.

Then, compile static.

ab · 2014-11-01 12:49:29

@AOG
Perhaps it is time for us to officially release and document the FPC/Windows/Linux feature.

Users would need easy step-by-step instructions to be able to compile for FPC/Lazarus.
We may consider CodeTyphon as our official IDE, since we both use it AFAIR, and since it is more regularly releases (5.1 is just out! - with CrossBuild Support for Rapberry Pi and other ARM platforms you like so much).
In fact, why not propose to help including mORMot within the official CodeTyphon release?

ComingNine · 2014-11-01 13:13:40

program LinuxSynTestFPCLinuxi386;
{$I Synopse.inc} // define HASINLINE USETYPEINFO CPU32 CPU64 OWNNORMTOUPPER
{$APPTYPE CONSOLE}
uses {$ifdef Linux} cthreads, cwstring, {$endif} mORMotSelfTests;
begin SQLite3ConsoleTests; end.

Thank you for your help very much !
I have succeeded in cross-compiling mORMot from win32 to linux32 with static (linux32) library of SQLite !

OS: Windows 7 SP1 X64
CodeTyphon: 5.1
Typhon: x86
FPC cross-build element: i386-linux
mORMot: 014f4dd542e609ba2e27b81eccc6890f66ff1a68
modification to mORMot: nothing

File system:
Note: (1) The files under the "linuxlibrary" are from the VMWare CentOS 5.11 x86. That is to say, they are not from what CodeTyphon provides, which are from Debian 7.2. (2) The reason is that I would think that the files under "linuxlibrary" may need to be compatible with the sqlite*.o which are compiled using "mORMot\SQLite3\c-fpcgcclin.sh" under the VMWare CentOS 5.11 x86. (3) I am not sure whether the OS library from CodeTyphon works fine with the SQLite library compiled under the VMWare CentOS 5.11 x86. (4) The files under the "linuxlibrary" are only sufficient for LinuxSynTestFPCLinuxi386.lpr shown above. That is to say, if more functionality is needed, more library would be needed.

IDE settings:

Last edited by ComingNine (2014-11-01 13:29:30)

ComingNine · 2014-11-01 13:19:03

I have tried to ran the "LinuxSynTestFPCLinuxi386" executable on several of my linux machines. On one of them, there is one error:

The output from the "LinuxSynTestFPCLinuxi386"

   Synopse mORMot Framework Automated tests
  ------------------------------------------


1. Synopse libraries

 1.1. Low level common: 
  - System copy record: 20 assertions passed  22.00s
  - TRawUTF8List: 70,005 assertions passed  38713.00s
  - TDynArray: 1,027,706 assertions passed  278982.00s
  - TDynArrayHashed: 1,200,629 assertions passed  126158.00s
  - TObjectListHashed: 999,780 assertions passed  335741.00s
  - TObjectDynArrayWrapper: 167,501 assertions passed  34004.00s
  - Fast string compare: 7 assertions passed  5.00s
  - IdemPropName: 30 assertions passed  8.00s
  - Url encoding: 132 assertions passed  1246.00s
  - GUID: 9,005 assertions passed  6018.00s
  - IsMatch: 599 assertions passed  180.00s
  - Soundex: 35 assertions passed  11.00s
  - Numerical conversions: 1,119,286 assertions passed  430866.00s
  - crc32c: 20,020 assertions passed  79207.00s
      pas 23080.00s 271 B/s fast 10425.00s 600 B/s
  - Curr 64: 20,053 assertions passed  3109.00s
  - CamelCase: 11 assertions passed  18.00s
  - Bits: 4,774 assertions passed  109.00s
  - Ini files: 7,004 assertions passed  65814.00s
  - UTF8: 82,106 assertions passed  1076475.00s
  - Iso 8601 date and time: 36,015 assertions passed  5079.00s
  - Url decoding: 1,100 assertions passed  268.00s
  - Mime types: 23 assertions passed  28.00s
!  - TSynTable: 1 / 50 FAILED  5379.00s
  - TSynCache: 404 assertions passed  119.00s
  - TSynFilter: 804 assertions passed  2610.00s
  - TSynValidate: 677 assertions passed  709.00s
  - TSynLogFile: 36 assertions passed  899.00s
  Total failed: 1 / 4,767,812  - Low level common FAILED  2491962.00s

 1.2. Low level types: 
  - Url encoding: 200 assertions passed  992.00s
  - Encode decode JSON: 250,616 assertions passed  160159.00s
  - Variants: 5 assertions passed  6.00s
  Total failed: 0 / 250,821  - Low level types PASSED  161175.00s

 1.3. Cryptographic routines: 
  - Adler32: 1 assertion passed  21.00s
  - MD5: 1 assertion passed  10.00s
  - SHA1: 5 assertions passed  15.00s
  - SHA256: 5 assertions passed  25.00s
  - AES256: 12,177 assertions passed  963996.00s
  - RC4: 1 assertion passed  19.00s
  - Base64: 11,994 assertions passed  282174.00s
  - CompressShaAes: 1,683 assertions passed  4448.00s
  Total failed: 0 / 25,867  - Cryptographic routines PASSED  1250751.00s

 1.4. Compression: 
  - In memory compression: 12 assertions passed  695995.00s
  - GZIP format: 19 assertions passed  1254797.00s
  - SynLZO: 3,006 assertions passed  137542.00s
  - SynLZ: 21,010 assertions passed  723858.00s
  Total failed: 0 / 24,047  - Compression PASSED  2812237.00s


2. mORMot

 2.1. File based: 
  - Database direct access: 10,138 assertions passed  378921.00s
  - Virtual table direct access: 12 assertions passed  1811.00s
  - TSQLTableJSON: 106,068 assertions passed  130316.00s
  - TSQLRestClientDB: 310,072 assertions passed  1029241.00s
  Total failed: 0 / 426,290  - File based PASSED  1540337.00s

 2.2. File based memory map: 
  - Database direct access: 10,136 assertions passed  220219.00s
  - Virtual table direct access: 12 assertions passed  1264.00s
  - TSQLTableJSON: 106,068 assertions passed  99050.00s
  - TSQLRestClientDB: 310,071 assertions passed  1237888.00s
  Total failed: 0 / 426,287  - File based memory map PASSED  1558456.00s

 2.3. File based WAL: 
  - Database direct access: 10,138 assertions passed  260996.00s
  - Virtual table direct access: 12 assertions passed  1248.00s
  - TSQLTableJSON: 106,068 assertions passed  105735.00s
  - TSQLRestClientDB: 310,072 assertions passed  1061406.00s
  Total failed: 0 / 426,290  - File based WAL PASSED  1429446.00s

 2.4. Memory based: 
  - Database direct access: 10,136 assertions passed  200254.00s
  - Virtual table direct access: 12 assertions passed  1129.00s
  - TSQLTableJSON: 106,068 assertions passed  98096.00s
  - TSQLRestClientDB: 402,396 assertions passed  1747835.00s
  - RTree: 140,000 assertions passed  1048069.00s
  Total failed: 0 / 658,612  - Memory based PASSED  3095432.00s

 2.5. Basic classes: 
  - TSQLRecord: 77 assertions passed  629.00s
  - TSQLRecordSigned: 200 assertions passed  10346.00s
  - TSQLModel: 3 assertions passed  85.00s
  - TSQLRestServerFullMemory: 562,308 assertions passed  1504564.00s
  Total failed: 0 / 562,588  - Basic classes PASSED  1515670.00s

 2.6. Client server access: 
  - TSQLHttpServer: 2 assertions passed  44355.00s
     using Synsock - Synapse Platform Independent Socket Layer.514
  - TSQLHttpClient: 3 assertions passed  72193.00s
  - HTTP client keep alive: 387 assertions passed  5346723.00s
  - HTTP client multi connect: 387 assertions passed  86186.00s
  - HTTP client encrypted: 387 assertions passed  5361698.00s
  - Direct in process access: 356 assertions passed  21841.00s
  - HTTP several DB servers: 1,504 assertions passed  34536038.00s
  Total failed: 0 / 3,026  - Client server access PASSED  2519456.04s

 2.7. External database: 
  - TQuery: 2,003 assertions passed  97492.00s
  - External records: 2 assertions passed  395.00s
  - Auto adapt SQL: 624 assertions passed  42994.00s
  - Crypted database: 253,275 assertions passed  231756.00s
  - External via REST: 168,339 assertions passed  1417085.00s
  - External via virtual table: 168,339 assertions passed  1623631.00s
  - External via REST with change tracking: 178,439 assertions passed  2253432.00s
  Total failed: 0 / 771,021  - External database PASSED  5666860.00s

 2.8. Multi thread process: 
  - Create thread pool: 1 assertion passed  9366.00s
  - TSQLRestServerDB: 4,822 assertions passed  259597.00s
     1=0/s  2=0/s  5=0/s  10=0/s  30=0/s  50=0/s  
  - TSQLRestClientDB: 4,822 assertions passed  217509.00s
     1=0/s  2=0/s  5=0/s  10=0/s  30=0/s  50=0/s  
  - Locked: 4,822 assertions passed  213061.00s
     1=0/s  2=0/s  5=0/s  10=0/s  30=0/s  50=0/s  
  - Unlocked: 4,822 assertions passed  199824.00s
     1=0/s  2=0/s  5=0/s  10=0/s  30=0/s  50=0/s  
  - Background thread: 4,822 assertions passed  324339.00s
     1=0/s  2=0/s  5=0/s  10=0/s  30=0/s  50=0/s  
  - Main thread: 4,822 assertions passed  548101.00s
     1=0/s  2=0/s  5=0/s  10=0/s  30=0/s  50=0/s  
  Total failed: 0 / 28,933  - Multi thread process PASSED  1771903.00s


Synopse framework used: 1.18.451
SQlite3 engine used: 3.8.7
Generated with: Free Pascal 2.7.1 compiler
Tests performed at 11/01/2014 09:49:54 PM

Total assertions failed for all test suits:  1 / 8,371,594
! Some tests FAILED: please correct the code.
                                             Done - Press ENTER to Exit

The "LinuxSynTestFPCLinuxi386 20141101 214843.log"

Host=linux User=user CPU=1*0-15-1027 OS=2.3=5.1.2600 Wow64=0 Freq=1234
TSQLLog 1.18.451 2014-11-01T21:48:43

20141101 21484323 warn  Test failed Out of memory
20141101 21484323 fail  TTestLowLevelCommon(555562F0) Low level common: TSynTable "Out of memory"

The output from "top" command

top - 22:16:39 up 3 days, 20:59,  4 users,  load average: 0.20, 0.20, 0.18
Tasks: 269 total,   1 running, 268 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  24676464k total, 13231312k used, 11445152k free,   249460k buffers
Swap: 49150856k total,        0k used, 49150856k free, 12487296k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                       
29319 xxe       16   0 17108 1336  900 R  0.3  0.0   0:00.04 top

Could you help to comment ?

ComingNine · 2014-11-01 13:27:32

ab wrote:

@AOG
Perhaps it is time for us to officially release and document the FPC/Windows/Linux feature.
Users would need easy step-by-step instructions to be able to compile for FPC/Lazarus.
We may consider CodeTyphon as our official IDE, since we both use it AFAIR, and since it is more regularly releases (5.1 is just out! - with CrossBuild Support for Rapberry Pi and other ARM platforms you like so much).
In fact, why not propose to help including mORMot within the official CodeTyphon release?

Great idea ! Once the FPC team/community knows mORMot, they may help to fix that crucial RTTI bug...

Furthermore, stack trace should be a very useful feature for mORMot using FPC/Windows/Linux.

AOG · 2014-11-01 13:27:57

@ComingNine.

This is very good news ! Thank you for sharing your setup. Enjoy mORMot on Linux !!

One minor thing: you can try removing the -dPUREPASCAL.
mORMot should work without it on i386. It will speed up some parts of mORMot by using assembler.

@Ab.
Very (very) good ideas !!

But, IMHO, before we go the official way, we would have to include FPC projects in the samples.
E.g. make the samples 100% compatible with FPC.
And include an *.lpi for very easy FPC deployment.
I volunteer in making them ... ;-)

I have been in contact with Sternas (CodeTyphon CEO) before, for making CodeTyphon do ARMHF things.
If we have our FPC-samples ready, I am positive about inclusion in CodeTyphon.

However, mORMot runs also fine with FPC 2.6.4 (and Lazarus).

Alfred.

AOG · 2014-11-01 13:34:46

@ComingNine.

Yes, I saw this error before, in your outputs.

As far as I now, it is this statement inside SynSelfTests that is causing trouble:

check(data.Field['text']=u);

(line 2989)

But I am not yet sure about it. Still checking and testing.
If I now more, I will report about it !

ComingNine · 2014-11-01 13:54:31

@AOG and @ab
Congratulations for your great work ! Thank you for your efforts very much !

@AOG
Can you help to comment how you identified the line number (line 2989) of the exception ? It will help me to debug mORMot-based applications using FPC on Linux. Maybe your way will even help @ab to implement the stack trace feature ...

Thank you for your suggestions ! Removing -dPUREPASCAL does not introduce errors. But it does not improve speed either, as shown below (left is with -dPUREPASCAL, right is without -dPUREPASCAL)...

Last edited by ComingNine (2014-11-01 13:55:40)

ab · 2014-11-01 14:19:44

You can see a difference where it was optimized, e.g. for crc32c computation.

Otherwise, do not expect to much of the optimization, but perhaps also on numerical conversion into text.

AOG · 2014-11-01 14:22:14

With my (Virtual) Linux machine, I get the same numbers as you do !
So not much improvement, but still a little benefit from assembler.
Proves that FPC is a good compiler for Linux !
And Ab a very efficient programmer !!

Debugging.
Well, your previous output showed an error in the SynTable test.
Inserting this:

writeln('AssertionsFailed: ',AssertionsFailed);

into different places inside the SynTable test run, shows you the place of the error.

Then, I use (some version of) Delphi to step through the code, until the lowest level.
Until the bug surfaces.

Consumes (a lot of) time, but bug-hunting consumes a lot of time unfortunately.

Last edited by AOG (2014-11-01 14:28:21)

ComingNine · 2014-11-01 15:13:59

Congratulation for the ASM crc32c achievement !

Thank you for your comments regarding debugging ! That does sound a lot of work ...

I have another question: This error occurs on only one of my linux box. How do you manage to reproduce this error on your linux ...

Last edited by ComingNine (2014-11-01 15:35:27)

ab · 2014-11-01 21:32:57

I suspect stack trace during debugging may help.
See http://synopse.info/fossil/info/b64b01bca

mORMot Open Source

#51 2014-11-01 09:11:13

Re: mORMot.pas

#52 2014-11-01 09:31:16

Re: mORMot.pas

#53 2014-11-01 12:49:29

Re: mORMot.pas

#54 2014-11-01 13:13:40

Re: mORMot.pas

#55 2014-11-01 13:19:03

Re: mORMot.pas

#56 2014-11-01 13:27:32

Re: mORMot.pas

#57 2014-11-01 13:27:57

Re: mORMot.pas

#58 2014-11-01 13:34:46

Re: mORMot.pas

#59 2014-11-01 13:54:31

Re: mORMot.pas

#60 2014-11-01 14:19:44

Re: mORMot.pas

#61 2014-11-01 14:22:14

Re: mORMot.pas

#62 2014-11-01 15:13:59

Re: mORMot.pas

#63 2014-11-01 21:32:57

Re: mORMot.pas

Board footer