You are not logged in.
Ubuntu64bit running on Virtualbox hosted on Windows-laptop. FPC built by fpcupdeluxe
Below is TestSQL3 console
blabla
...
Using mORMot 1.18.3554 FTS3
Running on Linux-4.4.0-70-generic #91-Ubuntu SMP Wed Mar 22 12:47:43 UTC 2017
TSQLite3LibraryStatic 3.17.0 with internal MM
Generated with: Free Pascal 3.1.1 MOP 64 bit compiler
Time elapsed for all tests: 154.89s
Tests performed at 2017年03月29日 上午 11时18分48秒 上午
Total assertions failed for all test suits: 10 / 23,646,794
! Some tests FAILED: please correct the code.
Heap dump by heaptrc unit
59735153 memory blocks allocated : 22346965872/22506485944
59735118 memory blocks freed : 22346952752/22506472768
35 unfreed memory blocks : 13120
True heap size : 2392064
True free heap : 2373280
Should be : 2374408
Call trace for block $00007FC87C61A320 size 55
$00000000006F2791 INTERNALREQUEST, line 409 of ../SynDBRemote.pas
$00000000006EBEA8 PROCESS, line 8111 of ../SynDB.pas
$00000000006EBEA8 PROCESS, line 8111 of ../SynDB.pas
$00000000006EE555 EXECUTEPREPARED, line 8531 of ../SynDB.pas
$00000000006501D5 MULTIFIELDVALUES, line 34324 of mORMot.pas
Call trace for block $00007FC8758DE900 size 29
$00000000006EE57A EXECUTEPREPARED, line 8535 of ../SynDB.pas
$00000000006DA6B9 REMOTEPROCESSMESSAGE, line 4592 of ../SynDB.pas
$00000000006EC3BA PROCESSMESSAGE, line 8162 of ../SynDB.pas
$00000000006EE555 EXECUTEPREPARED, line 8531 of ../SynDB.pas
$000000000069EEFB INTERNALPROCESS, line 52609 of mORMot.pas
$0000000000618BDC X64FAKESTUB
$0000000000618BDC X64FAKESTUB
$0000000000618BDC X64FAKESTUB
Call trace for block $00007FC8758E9700 size 34
$00000000006ECA21 INTHEADERPROCESS, line 8261 of ../SynDB.pas
$00000000006EE57A EXECUTEPREPARED, line 8535 of ../SynDB.pas
$00000000006DA355 REMOTEPROCESSMESSAGE, line 4562 of ../SynDB.pas
$00000000006EC3BA PROCESSMESSAGE, line 8162 of ../SynDB.pas
$00000000006EE555 EXECUTEPREPARED, line 8531 of ../SynDB.pas
$0000000000618BDC X64FAKESTUB
$0000000000618BDC X64FAKESTUB
Call trace for block $00007FC87C4AE950 size 472
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
Call trace for block $00007FC87CDAF290 size 4843
$0000000000484A4C REQUEST, line 4362 of ../SynCrtSock.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
Call trace for block $00007FC8758CF780 size 47
$0000000000488AA3 GETHEADER, line 5203 of ../SynCrtSock.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$0000000000636EC7 CREATEFROMTABLES, line 27840 of mORMot.pas
$0000000000659B38 LIST, line 36606 of mORMot.pas
Call trace for block $00007FC87C620FA0 size 62
$0000000000488AA3 GETHEADER, line 5203 of ../SynCrtSock.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$0000000000519BCF LOGINTERNAL, line 4210 of ../SynLog.pas
$0000000000518319 LOG, line 3775 of ../SynLog.pas
$00000000006CF4FE CREATE, line 749 of mORMotHttpServer.pas
Call trace for block $00007FC87C6237E0 size 56
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$000000000068F4DA DESTGETJOINEDTABLE, line 48718 of mORMot.pas
$000000000066CAF0 URI, line 40797 of mORMot.pas
$000000000068F4DA DESTGETJOINEDTABLE, line 48718 of mORMot.pas
Call trace for block $00007FC87C6216A0 size 70
$0000000000488AA3 GETHEADER, line 5203 of ../SynCrtSock.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$00000000006581AE SERVERTIMESTAMPSYNCHRONIZE, line 36236 of mORMot.pas
$0000000000519BCF LOGINTERNAL, line 4210 of ../SynLog.pas
$0000000000518319 LOG, line 3775 of ../SynLog.pas
$00000000006CF4FE CREATE, line 749 of mORMotHttpServer.pas
Call trace for block $00007FC8758CEAC0 size 45
$0000000000488AA3 GETHEADER, line 5203 of ../SynCrtSock.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$0000000000636EC7 CREATEFROMTABLES, line 27840 of mORMot.pas
$0000000000659B38 LIST, line 36606 of mORMot.pas
Call trace for block $00007FC87C622BA0 size 52
$0000000000488AA3 GETHEADER, line 5203 of ../SynCrtSock.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$000000000066CAF0 URI, line 40797 of mORMot.pas
$00000000006501D5 MULTIFIELDVALUES, line 34324 of mORMot.pas
Call trace for block $00007FC8758CF540 size 47
$0000000000488AA3 GETHEADER, line 5203 of ../SynCrtSock.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$0000000000636EC7 CREATEFROMTABLES, line 27840 of mORMot.pas
$0000000000659B38 LIST, line 36606 of mORMot.pas
Call trace for block $00007FC87C622820 size 63
$0000000000488AA3 GETHEADER, line 5203 of ../SynCrtSock.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$000000000068F4DA DESTGETJOINEDTABLE, line 48718 of mORMot.pas
$000000000066CAF0 URI, line 40797 of mORMot.pas
$000000000068F4DA DESTGETJOINEDTABLE, line 48718 of mORMot.pas
Call trace for block $00007FC87C622F20 size 49
$0000000000488AA3 GETHEADER, line 5203 of ../SynCrtSock.pas
$00000000006CD046 INTERNALREQUEST, line 683 of mORMotHttpClient.pas
$000000000068F4DA DESTGETJOINEDTABLE, line 48718 of mORMot.pas
$000000000066CAF0 URI, line 40797 of mORMot.pas
$000000000068F4DA DESTGETJOINEDTABLE, line 48718 of mORMot.pas
Offline
I have seen this leaks also.
And spend a lot of time digging into them, but to no avail.
However, AFAIK, functionality of the mORMot is not in jeopardy !
Offline
Thanks for replying. @AOG, Yes, I agree, TestSQL3 is so complicate, so many (20 million) Testcase, it's hard to guaranty of 100% memory management.
And if we can't get perfect code, is there specific mormot version, that is stable enough and suitable for Linux server ? ( the newpascal-ccr on github is DEPRECATED ). Memory leak on a 24*7 services could be a disaster, depending where it leaks from ( the SynCrtSock.pas shown in trace makes me nervous ) . Is there some successful linux-service showcase ?
Offline
I just checked my mORMot server that runs on an Amazon T2 tier (linux64).
Its is now running for 6 months 24/7 without a single restart.
No (memory) problems whatsoever.
In fact, I have looked deeply into the SynCrtSock.pas problem.
It has to to with the compression of data.
It could be that a line as below (SynCommons, CompressSynLZ) does not work 100% on FPC, but I am not yet sure.
Data: RawByteString absolute DataRawByteString;
Offline
In the above routines, SetString could also be the culprit.
See ZEOS-code here:
https://sourceforge.net/p/zeoslib/code- … y.pas#l715
and further.
But again: still not sure what is causing this.
Offline
Good news, there are real-production Linux64server, @AOG, thank you for your replying&exploring . What we planned is a moromt/SQLite3 cloud on Amazon ELB+EC2, maybe it will similar to your business-solution, hope for further discussion on linux. Does not familiar with SynLZ, but the WEB&APP world will always prefer deflate/gzip, so maybe we should tailor TestSQL3 to our business-needs.
AFAIK, ab seems prefer delphi+windows, the significant difference between FPC&Delphi is Memorymanager, and I discovered that FastMM support FPC
https://github.com/pleriche/FastMM4/issues/20
so maybe we can switch to FastMM under FPC, and make a try.
Offline
That's weird, because SynLZ is indeed a proprietary protocol, so only mORMot clients would now it.
You can disable it on production, for sure.
Between mORMot nodes, you could rather rely on WebSockets communication.
Which is encrypted, compressed, more efficient, and allows asynchronous callbacks notifications.
I have to look into this Linux specific problems.
The regression tests should NOT trigger any problem at shutdown, for sure!
Offline
During the test, there are an Exception :
! External database - SynDBRemote
! Exception ESQLDBRemote raised with messsage:
! TSQLDBCurlConnectionProperties.ProcessMessage: Error 404 from http://127.0.0.1:8888/
It seems some memory leaks because of this exception, and the source-code below ask us to use 32-bit libcurl:
// - under a 64-bit Linux system, you should install the 32-bit flavor of
// libcurl, e.g. on Ubuntu:
// $ sudo apt-get install libcurl3:i386
// - will use in fact libcurl.so, so either libcurl.so.3 or libcurl.so.4,
// depending on the default version installation on the system
TCurlHTTP = class(THttpRequest)
But, TestSQL3 used a 64bit- libcurl (see below ), does it matters ?
sa@sa-VirtualBox:~$ lsof -a -p 3989
XTestSQL3 3989 sa mem REG 8,1 452992 400666 /usr/lib/x86_64-linux-gnu/libcurl.so.4.4.0
Offline
I have many memory leaks too. TestSQL3 tested for Windows 10.
All memory leaks are in the same place (CREATE, line 2827 of ../SynBidirSock.pas):
C:\_projects\FreePascal\trunk_pure_install\bin\i386-win32\mormot\SQLite3\fpc\bin\i386-win32\TestSQL3.exe
Heap dump by heaptrc unit
60531400 memory blocks allocated : 534546464/727925560
60531200 memory blocks freed : 534530856/727909560
200 unfreed memory blocks : 15608
True heap size : 4294082560 (160 used in System startup)
True free heap : 4294082400
Should be : 4294053600
Call trace for block $1BB327D0 size 80
$004110E5 GETMEM, line 284 of ../inc/heap.inc
$0040D814 NEWINSTANCE, line 427 of ../inc/objpas.inc
$004584DF CREATE, line 2827 of ../SynBidirSock.pas
$0044C5FB TASK, line 5713 of ../SynCrtSock.pas
$0044C364 EXECUTE, line 5671 of ../SynCrtSock.pas
$006C6045 THREADPROC, line 203 of ../objpas/classes/classes.inc
$0041314E THREADMAIN, line 235 of ../win/systhrd.inc
$77498744
Call trace for block $1BB32690 size 80
$004110E5 GETMEM, line 284 of ../inc/heap.inc
$0040D814 NEWINSTANCE, line 427 of ../inc/objpas.inc
$004584DF CREATE, line 2827 of ../SynBidirSock.pas
$0044C5FB TASK, line 5713 of ../SynCrtSock.pas
$0044C364 EXECUTE, line 5671 of ../SynCrtSock.pas
$006C6045 THREADPROC, line 203 of ../objpas/classes/classes.inc
$0041314E THREADMAIN, line 235 of ../win/systhrd.inc
$77498744
Call trace for block $1BB32550 size 80
$004110E5 GETMEM, line 284 of ../inc/heap.inc
$0040D814 NEWINSTANCE, line 427 of ../inc/objpas.inc
$004584DF CREATE, line 2827 of ../SynBidirSock.pas
$0044C5FB TASK, line 5713 of ../SynCrtSock.pas
$0044C364 EXECUTE, line 5671 of ../SynCrtSock.pas
$006C6045 THREADPROC, line 203 of ../objpas/classes/classes.inc
$0041314E THREADMAIN, line 235 of ../win/systhrd.inc
$77498744
... and so one
any new ideas?
best regards,
Maciej Izak
Offline
Today I found this problem: https://bugs.freepascal.org/view.php?id=31987
best regards,
Maciej Izak
Offline
@hnb
If you run TestSQL3, but without
TTestBidirectionalRemoteConnection
TTestMultiThreadProcess
all is ok !
Perhaps this helps a bit.
2.10. DDD multi thread:
- Delete old database: 1 assertion passed 1.50ms
- Start server: 1 assertion passed 31.90ms
- Single client test: 1,002 assertions passed 483.15ms
- Multi threaded clients test: 21 assertions passed 338.64ms
Total failed: 0 / 1,025 - DDD multi thread PASSED 855.36ms
Using mORMot 1.18.3684 FTS3
Running on Windows 10 64bit (10.0.14393) with code page 1252
TSQLite3LibraryStatic 3.19.2 with internal MM
Generated with: Free Pascal 3.1.1 MOP compiler
Time elapsed for all tests: 154.33s
Tests performed at 10-6-2017 11:43:39
Total assertions failed for all test suits: 0 / 25,759,320
! All tests passed successfully.
Heap dump by heaptrc unit
57661255 memory blocks allocated : 1575202168/1760679880
57661255 memory blocks freed : 1575202168/1760679880
0 unfreed memory blocks : 0
True heap size : 4294508544 (128 used in System startup)
True free heap : 4294508416
Offline
Additional:
If your do run TTestMultiThreadProcess, but without TTestMultiThreadProcess.SocketAPI and TTestMultiThreadProcess.Websockets, all is again ok.
Offline
Alright. Found the culprit !
Its the TSynThread.Destroy;
Its only enabled for FPC.
Remove it : no memory leaks anymore !!
Not sure however if this is correct.
2.12. DDD multi thread:
- Delete old database: 1 assertion passed 675us
- Start server: 1 assertion passed 16.89ms
- Single client test: 1,002 assertions passed 567.87ms
- Multi threaded clients test: 21 assertions passed 366.20ms
Total failed: 0 / 1,025 - DDD multi thread PASSED 951.75ms
Using mORMot 1.18.3684 FTS3
Running on Windows 10 64bit (10.0.14393) with code page 1252
TSQLite3LibraryStatic 3.19.2 with internal MM
Generated with: Free Pascal 3.1.1 MOP compiler
Time elapsed for all tests: 183.84s
Tests performed at 10-6-2017 12:29:23
Total assertions failed for all test suits: 0 / 25,842,666
! All tests passed successfully.
Done - Press ENTER to Exit
Heap dump by heaptrc unit
59206543 memory blocks allocated : 1960090725/2150608728
59206543 memory blocks freed : 1960090725/2150608728
0 unfreed memory blocks : 0
True heap size : 4294344704 (128 used in System startup)
True free heap : 4294344576
Offline
Great News !! test with my windows laptop success
Using mORMot 1.18.3698 FTS3
Running on Windows 10 64bit (10.0.14393) with code page 936
TSQLite3LibraryStatic 3.19.2 with internal MM
Generated with: Free Pascal 3.1.1 MOP compiler
Time elapsed for all tests: 118.40s
Tests performed at 2017-06-27 9:08:54
Total assertions failed for all test suits: 0 / 26,955,672
! All tests passed successfully.
Done - Press ENTER to Exit
Heap dump by heaptrc unit
60964465 memory blocks allocated : 625514488/820540480
60964465 memory blocks freed : 625514488/820540480
0 unfreed memory blocks : 0
True heap size : 4294115328 (112 used in System startup)
True free heap : 4294115216
test on ubuntu, still with some exception and memleak
Using mORMot 1.18.3698 FTS3
Running on Linux-4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017
TSQLite3LibraryStatic 3.19.2 with internal MM
Generated with: Free Pascal 3.1.1 MOP 64 bit compiler
Time elapsed for all tests: 369.20s
Tests performed at 2017年06月27日 下午 01时57分12秒 下午
Total assertions failed for all test suits: 4,033 / 25,734,184
! Some tests FAILED: please correct the code.
Heap dump by heaptrc unit
55791900 memory blocks allocated : 21569492832/21723426616
55791865 memory blocks freed : 21569483394/21723417120
35 unfreed memory blocks : 9438
True heap size : 2392064
True free heap : 2376960
Should be : 2378088
Call trace for block $00007F1603643E20 size 55
Internal error: unknown dwarf form: $17
Internal error: unknown dwarf form: $19
$000000000043D393
$000000000048BF48 LIBCURLINITIALIZE, line 8676 of ../SynCrtSock.pas
$000000000048C20E INTERNALCONNECT, line 8731 of ../SynCrtSock.pas
$00000000006F47B5 EXECUTEPREPARED, line 8531 of ../SynDB.pas
$000000000061DC0C X64FAKESTUB
$000000000061DC0C X64FAKESTUB
Call trace for block $00007F160375A3C0 size 29
$00000000006F47DA EXECUTEPREPARED, line 8535 of ../SynDB.pas
$00000000006E0919 REMOTEPROCESSMESSAGE, line 4592 of ../SynDB.pas
$00000000006F261A PROCESSMESSAGE, line 8162 of ../SynDB.pas
$00000000006F47B5 EXECUTEPREPARED, line 8531 of ../SynDB.pas
$00000000006F47B5 EXECUTEPREPARED, line 8531 of ../SynDB.pas
Call trace for block $00007F16015A7F40 size 34
$00000000006F2C81 INTHEADERPROCESS, line 8261 of ../SynDB.pas
$00000000006F47DA EXECUTEPREPARED, line 8535 of ../SynDB.pas
$00000000006F261A PROCESSMESSAGE, line 8162 of ../SynDB.pas
$00000000006F47B5 EXECUTEPREPARED, line 8531 of ../SynDB.pas
$00000000006F47B5 EXECUTEPREPARED, line 8531 of ../SynDB.pas
Call trace for block $00007F1603D4FB90 size 472
$F0F0F0F0F0F0F0F0
$F0F0F0F0F0F0F0F0
Offline
After lengthy investigation, many of the above errors seem to be caused by compression not working as expected on Linux. Will investigate further.
Offline
May I ask if this issue has been resolved?
I just installed NewPascal (Windows 8.1 32bits) and when running TestSQL3 I get the same leak.
Removing TSynThread.Destroy only worked for TTestBidirectionalRemoteConnection but for TTestMultiThreadProcess it's gotten worse.
The same thing happens with the provided samples related to multithreading.
Last edited by slowpoke (2017-10-01 00:58:05)
Offline
the bug was related to FPC & mORMot. I've fixed the bug some time ago for FPC:
https://svn.freepascal.org/cgi-bin/view … sion=35717
and thanks to help of Alfred also for mORMot:
https://github.com/synopse/mORMot/commi … 3cceb24ee8
New version of NewPascal should be released very soon.
best regards,
Maciej Izak
Offline
Thanks @hnb, works without problems now.
Offline