Pages: 1

#1 2017-03-29 06:18:23

cybexr
Member
Registered: 2016-09-14
Posts: 37

TestSQL3 compiled by FPC memoryleak

Ubuntu64bit running on Virtualbox hosted on Windows-laptop.  FPC built by fpcupdeluxe

Below is TestSQL3 console 

blabla
...

Using mORMot 1.18.3554 FTS3
Running on Linux-4.4.0-70-generic #91-Ubuntu SMP Wed Mar 22 12:47:43 UTC 2017
TSQLite3LibraryStatic 3.17.0 with internal MM
Generated with: Free Pascal 3.1.1 MOP 64 bit compiler

Time elapsed for all tests: 154.89s
Tests performed at 2017年03月29日 上午 11时18分48秒 上午

Total assertions failed for all test suits:  10 / 23,646,794
! Some tests FAILED: please correct the code.
Heap dump by heaptrc unit
59735153 memory blocks allocated : 22346965872/22506485944
59735118 memory blocks freed     : 22346952752/22506472768
35 unfreed memory blocks : 13120
True heap size : 2392064
True free heap : 2373280
Should be : 2374408
Call trace for block $00007FC87C61A320 size 55
  $00000000006F2791  INTERNALREQUEST,  line 409 of ../SynDBRemote.pas
  $00000000006EBEA8  PROCESS,  line 8111 of ../SynDB.pas
  $00000000006EBEA8  PROCESS,  line 8111 of ../SynDB.pas
  $00000000006EE555  EXECUTEPREPARED,  line 8531 of ../SynDB.pas
  $00000000006501D5  MULTIFIELDVALUES,  line 34324 of mORMot.pas
Call trace for block $00007FC8758DE900 size 29
  $00000000006EE57A  EXECUTEPREPARED,  line 8535 of ../SynDB.pas
  $00000000006DA6B9  REMOTEPROCESSMESSAGE,  line 4592 of ../SynDB.pas
  $00000000006EC3BA  PROCESSMESSAGE,  line 8162 of ../SynDB.pas
  $00000000006EE555  EXECUTEPREPARED,  line 8531 of ../SynDB.pas
  $000000000069EEFB  INTERNALPROCESS,  line 52609 of mORMot.pas
  $0000000000618BDC  X64FAKESTUB
  $0000000000618BDC  X64FAKESTUB
  $0000000000618BDC  X64FAKESTUB
Call trace for block $00007FC8758E9700 size 34
  $00000000006ECA21  INTHEADERPROCESS,  line 8261 of ../SynDB.pas
  $00000000006EE57A  EXECUTEPREPARED,  line 8535 of ../SynDB.pas
  $00000000006DA355  REMOTEPROCESSMESSAGE,  line 4562 of ../SynDB.pas
  $00000000006EC3BA  PROCESSMESSAGE,  line 8162 of ../SynDB.pas
  $00000000006EE555  EXECUTEPREPARED,  line 8531 of ../SynDB.pas
  $0000000000618BDC  X64FAKESTUB
  $0000000000618BDC  X64FAKESTUB
Call trace for block $00007FC87C4AE950 size 472
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
Call trace for block $00007FC87CDAF290 size 4843
  $0000000000484A4C  REQUEST,  line 4362 of ../SynCrtSock.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0
Call trace for block $00007FC8758CF780 size 47
  $0000000000488AA3  GETHEADER,  line 5203 of ../SynCrtSock.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $0000000000636EC7  CREATEFROMTABLES,  line 27840 of mORMot.pas
  $0000000000659B38  LIST,  line 36606 of mORMot.pas
Call trace for block $00007FC87C620FA0 size 62
  $0000000000488AA3  GETHEADER,  line 5203 of ../SynCrtSock.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $0000000000519BCF  LOGINTERNAL,  line 4210 of ../SynLog.pas
  $0000000000518319  LOG,  line 3775 of ../SynLog.pas
  $00000000006CF4FE  CREATE,  line 749 of mORMotHttpServer.pas
Call trace for block $00007FC87C6237E0 size 56
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $000000000068F4DA  DESTGETJOINEDTABLE,  line 48718 of mORMot.pas
  $000000000066CAF0  URI,  line 40797 of mORMot.pas
  $000000000068F4DA  DESTGETJOINEDTABLE,  line 48718 of mORMot.pas
Call trace for block $00007FC87C6216A0 size 70
  $0000000000488AA3  GETHEADER,  line 5203 of ../SynCrtSock.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $00000000006581AE  SERVERTIMESTAMPSYNCHRONIZE,  line 36236 of mORMot.pas
  $0000000000519BCF  LOGINTERNAL,  line 4210 of ../SynLog.pas
  $0000000000518319  LOG,  line 3775 of ../SynLog.pas
  $00000000006CF4FE  CREATE,  line 749 of mORMotHttpServer.pas
Call trace for block $00007FC8758CEAC0 size 45
  $0000000000488AA3  GETHEADER,  line 5203 of ../SynCrtSock.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $0000000000636EC7  CREATEFROMTABLES,  line 27840 of mORMot.pas
  $0000000000659B38  LIST,  line 36606 of mORMot.pas
Call trace for block $00007FC87C622BA0 size 52
  $0000000000488AA3  GETHEADER,  line 5203 of ../SynCrtSock.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $000000000066CAF0  URI,  line 40797 of mORMot.pas
  $00000000006501D5  MULTIFIELDVALUES,  line 34324 of mORMot.pas
Call trace for block $00007FC8758CF540 size 47
  $0000000000488AA3  GETHEADER,  line 5203 of ../SynCrtSock.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $0000000000636EC7  CREATEFROMTABLES,  line 27840 of mORMot.pas
  $0000000000659B38  LIST,  line 36606 of mORMot.pas
Call trace for block $00007FC87C622820 size 63
  $0000000000488AA3  GETHEADER,  line 5203 of ../SynCrtSock.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $000000000068F4DA  DESTGETJOINEDTABLE,  line 48718 of mORMot.pas
  $000000000066CAF0  URI,  line 40797 of mORMot.pas
  $000000000068F4DA  DESTGETJOINEDTABLE,  line 48718 of mORMot.pas
Call trace for block $00007FC87C622F20 size 49
  $0000000000488AA3  GETHEADER,  line 5203 of ../SynCrtSock.pas
  $00000000006CD046  INTERNALREQUEST,  line 683 of mORMotHttpClient.pas
  $000000000068F4DA  DESTGETJOINEDTABLE,  line 48718 of mORMot.pas
  $000000000066CAF0  URI,  line 40797 of mORMot.pas
  $000000000068F4DA  DESTGETJOINEDTABLE,  line 48718 of mORMot.pas

Offline

#2 2017-03-29 06:25:45

AOG
Member
Registered: 2014-02-24
Posts: 346

Re: TestSQL3 compiled by FPC memoryleak

I have seen this leaks also.
And spend a lot of time digging into them, but to no avail.

However, AFAIK, functionality of the mORMot is not in jeopardy !

Offline

#3 2017-03-29 07:34:29

cybexr
Member
Registered: 2016-09-14
Posts: 37

Re: TestSQL3 compiled by FPC memoryleak

Thanks for replying.  @AOG,  Yes, I agree, TestSQL3 is so complicate, so many (20 million) Testcase, it's hard to guaranty of 100% memory management.

And if we can't get perfect code, is there specific mormot version, that is stable enough and suitable for Linux server ? ( the newpascal-ccr on github is DEPRECATED ).  Memory leak on a 24*7 services could be a disaster, depending where it leaks from ( the SynCrtSock.pas shown in trace makes me nervous ) . Is there some successful linux-service showcase ?

Offline

#4 2017-03-29 08:03:14

AOG
Member
Registered: 2014-02-24
Posts: 346

Re: TestSQL3 compiled by FPC memoryleak

I just checked my mORMot server that runs on an Amazon T2 tier (linux64).
Its is now running for 6 months 24/7 without a single restart.
No (memory) problems whatsoever.

In fact, I have looked deeply into the SynCrtSock.pas problem.
It has to to with the compression of data.
It could be that a line as below (SynCommons, CompressSynLZ) does not work 100% on FPC, but I am not yet sure.
Data: RawByteString absolute DataRawByteString;

Offline

#5 2017-03-29 08:20:59

AOG
Member
Registered: 2014-02-24
Posts: 346

Re: TestSQL3 compiled by FPC memoryleak

In the above routines, SetString could also be the culprit.
See ZEOS-code here:
https://sourceforge.net/p/zeoslib/code- … y.pas#l715
and further.

But again: still not sure what is causing this.

Offline

#6 2017-03-30 02:37:36

cybexr
Member
Registered: 2016-09-14
Posts: 37

Re: TestSQL3 compiled by FPC memoryleak

Good news, there are real-production Linux64server, @AOG, thank you for your replying&exploring . What we planned is a moromt/SQLite3 cloud on Amazon ELB+EC2, maybe it will similar to your business-solution,  hope for further discussion on linux.  Does not familiar with SynLZ, but the WEB&APP world will always prefer deflate/gzip, so maybe we should tailor TestSQL3 to our business-needs.

AFAIK,  ab seems prefer delphi+windows, the significant difference between FPC&Delphi is Memorymanager, and I discovered that FastMM support FPC
https://github.com/pleriche/FastMM4/issues/20
so maybe we can switch to FastMM under FPC, and make a try.

Offline

#7 2017-03-30 10:19:27

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 9,274
Website

Re: TestSQL3 compiled by FPC memoryleak

That's weird, because SynLZ is indeed a proprietary protocol, so only mORMot clients would now it.
You can disable it on production, for sure.

Between mORMot nodes, you could rather rely on WebSockets communication.
Which is encrypted, compressed, more efficient, and allows asynchronous callbacks notifications.

I have to look into this Linux specific problems.
The regression tests should NOT trigger any problem at shutdown, for sure!

Offline

#8 2017-03-31 06:41:59

cybexr
Member
Registered: 2016-09-14
Posts: 37

Re: TestSQL3 compiled by FPC memoryleak

During the test, there are an Exception :
! External database - SynDBRemote
! Exception ESQLDBRemote raised with messsage:
!  TSQLDBCurlConnectionProperties.ProcessMessage: Error 404 from http://127.0.0.1:8888/

It seems some memory leaks because of this exception, and the source-code below ask us to use 32-bit libcurl: 

  // - under a 64-bit Linux system, you should install the 32-bit flavor of
  // libcurl, e.g. on Ubuntu:
  // $ sudo apt-get install libcurl3:i386
  // - will use in fact libcurl.so, so either libcurl.so.3 or libcurl.so.4,
  // depending on the default version installation on the system
  TCurlHTTP = class(THttpRequest)

But, TestSQL3 used a 64bit- libcurl (see below ), does it matters ?

sa@sa-VirtualBox:~$ lsof -a -p 3989
XTestSQL3 3989   sa  mem    REG    8,1   452992 400666 /usr/lib/x86_64-linux-gnu/libcurl.so.4.4.0

Offline

#9 2017-06-08 22:52:00

hnb
Member
Registered: 2015-06-15
Posts: 246

Re: TestSQL3 compiled by FPC memoryleak

I have many memory leaks too. TestSQL3 tested for Windows 10.

All memory leaks are in the same place (CREATE,  line 2827 of ../SynBidirSock.pas):

C:\_projects\FreePascal\trunk_pure_install\bin\i386-win32\mormot\SQLite3\fpc\bin\i386-win32\TestSQL3.exe 
Heap dump by heaptrc unit
60531400 memory blocks allocated : 534546464/727925560
60531200 memory blocks freed     : 534530856/727909560
200 unfreed memory blocks : 15608
True heap size : 4294082560 (160 used in System startup)
True free heap : 4294082400
Should be : 4294053600
Call trace for block $1BB327D0 size 80
  $004110E5  GETMEM,  line 284 of ../inc/heap.inc
  $0040D814  NEWINSTANCE,  line 427 of ../inc/objpas.inc
  $004584DF  CREATE,  line 2827 of ../SynBidirSock.pas
  $0044C5FB  TASK,  line 5713 of ../SynCrtSock.pas
  $0044C364  EXECUTE,  line 5671 of ../SynCrtSock.pas
  $006C6045  THREADPROC,  line 203 of ../objpas/classes/classes.inc
  $0041314E  THREADMAIN,  line 235 of ../win/systhrd.inc
  $77498744
Call trace for block $1BB32690 size 80
  $004110E5  GETMEM,  line 284 of ../inc/heap.inc
  $0040D814  NEWINSTANCE,  line 427 of ../inc/objpas.inc
  $004584DF  CREATE,  line 2827 of ../SynBidirSock.pas
  $0044C5FB  TASK,  line 5713 of ../SynCrtSock.pas
  $0044C364  EXECUTE,  line 5671 of ../SynCrtSock.pas
  $006C6045  THREADPROC,  line 203 of ../objpas/classes/classes.inc
  $0041314E  THREADMAIN,  line 235 of ../win/systhrd.inc
  $77498744
Call trace for block $1BB32550 size 80
  $004110E5  GETMEM,  line 284 of ../inc/heap.inc
  $0040D814  NEWINSTANCE,  line 427 of ../inc/objpas.inc
  $004584DF  CREATE,  line 2827 of ../SynBidirSock.pas
  $0044C5FB  TASK,  line 5713 of ../SynCrtSock.pas
  $0044C364  EXECUTE,  line 5671 of ../SynCrtSock.pas
  $006C6045  THREADPROC,  line 203 of ../objpas/classes/classes.inc
  $0041314E  THREADMAIN,  line 235 of ../win/systhrd.inc
  $77498744
... and so one

any new ideas?

best regards,
Maciej Izak

Offline

#10 2017-06-09 08:25:22

hnb
Member
Registered: 2015-06-15
Posts: 246

Re: TestSQL3 compiled by FPC memoryleak

Today I found this problem: https://bugs.freepascal.org/view.php?id=31987

best regards,
Maciej Izak

Offline

#11 2017-06-10 09:44:31

AOG
Member
Registered: 2014-02-24
Posts: 346

Re: TestSQL3 compiled by FPC memoryleak

@hnb
If you run TestSQL3, but without
TTestBidirectionalRemoteConnection
TTestMultiThreadProcess
all is ok !
Perhaps this helps a bit.

2.10. DDD multi thread: 
  - Delete old database: 1 assertion passed  1.50ms
  - Start server: 1 assertion passed  31.90ms
  - Single client test: 1,002 assertions passed  483.15ms
  - Multi threaded clients test: 21 assertions passed  338.64ms
  Total failed: 0 / 1,025  - DDD multi thread PASSED  855.36ms


Using mORMot 1.18.3684 FTS3
Running on Windows 10 64bit (10.0.14393) with code page 1252
TSQLite3LibraryStatic 3.19.2 with internal MM
Generated with: Free Pascal 3.1.1 MOP compiler

Time elapsed for all tests: 154.33s
Tests performed at 10-6-2017 11:43:39

Total assertions failed for all test suits:  0 / 25,759,320
! All tests passed successfully.
Heap dump by heaptrc unit
57661255 memory blocks allocated : 1575202168/1760679880
57661255 memory blocks freed     : 1575202168/1760679880
0 unfreed memory blocks : 0
True heap size : 4294508544 (128 used in System startup)
True free heap : 4294508416

Offline

#12 2017-06-10 09:55:33

AOG
Member
Registered: 2014-02-24
Posts: 346

Re: TestSQL3 compiled by FPC memoryleak

Additional:
If your do run TTestMultiThreadProcess, but without TTestMultiThreadProcess.SocketAPI and TTestMultiThreadProcess.Websockets, all is again ok.

Offline

#13 2017-06-10 10:33:22

AOG
Member
Registered: 2014-02-24
Posts: 346

Re: TestSQL3 compiled by FPC memoryleak

Alright. Found the culprit !

Its the TSynThread.Destroy;
Its only enabled for FPC.
Remove it : no memory leaks anymore !!

Not sure however if this is correct.

2.12. DDD multi thread: 
  - Delete old database: 1 assertion passed  675us
  - Start server: 1 assertion passed  16.89ms
  - Single client test: 1,002 assertions passed  567.87ms
  - Multi threaded clients test: 21 assertions passed  366.20ms
  Total failed: 0 / 1,025  - DDD multi thread PASSED  951.75ms


Using mORMot 1.18.3684 FTS3
Running on Windows 10 64bit (10.0.14393) with code page 1252
TSQLite3LibraryStatic 3.19.2 with internal MM
Generated with: Free Pascal 3.1.1 MOP compiler

Time elapsed for all tests: 183.84s
Tests performed at 10-6-2017 12:29:23

Total assertions failed for all test suits:  0 / 25,842,666
! All tests passed successfully.

Done - Press ENTER to Exit
Heap dump by heaptrc unit
59206543 memory blocks allocated : 1960090725/2150608728
59206543 memory blocks freed     : 1960090725/2150608728
0 unfreed memory blocks : 0
True heap size : 4294344704 (128 used in System startup)
True free heap : 4294344576

Offline

#14 2017-06-27 06:15:49

cybexr
Member
Registered: 2016-09-14
Posts: 37

Re: TestSQL3 compiled by FPC memoryleak

Great News !!  test with my windows laptop success 

Using mORMot 1.18.3698 FTS3
Running on Windows 10 64bit (10.0.14393) with code page 936
TSQLite3LibraryStatic 3.19.2 with internal MM
Generated with: Free Pascal 3.1.1 MOP compiler

Time elapsed for all tests: 118.40s
Tests performed at 2017-06-27 9:08:54

Total assertions failed for all test suits:  0 / 26,955,672
! All tests passed successfully.

Done - Press ENTER to Exit

Heap dump by heaptrc unit
60964465 memory blocks allocated : 625514488/820540480
60964465 memory blocks freed     : 625514488/820540480
0 unfreed memory blocks : 0
True heap size : 4294115328 (112 used in System startup)
True free heap : 4294115216

test on ubuntu, still with some exception and memleak

Using mORMot 1.18.3698 FTS3
Running on Linux-4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017
TSQLite3LibraryStatic 3.19.2 with internal MM
Generated with: Free Pascal 3.1.1 MOP 64 bit compiler

Time elapsed for all tests: 369.20s
Tests performed at 2017年06月27日 下午 01时57分12秒 下午

Total assertions failed for all test suits:  4,033 / 25,734,184
! Some tests FAILED: please correct the code.
Heap dump by heaptrc unit
55791900 memory blocks allocated : 21569492832/21723426616
55791865 memory blocks freed     : 21569483394/21723417120
35 unfreed memory blocks : 9438
True heap size : 2392064
True free heap : 2376960
Should be : 2378088
Call trace for block $00007F1603643E20 size 55
Internal error: unknown dwarf form: $17
Internal error: unknown dwarf form: $19
  $000000000043D393
  $000000000048BF48  LIBCURLINITIALIZE,  line 8676 of ../SynCrtSock.pas
  $000000000048C20E  INTERNALCONNECT,  line 8731 of ../SynCrtSock.pas
  $00000000006F47B5  EXECUTEPREPARED,  line 8531 of ../SynDB.pas
  $000000000061DC0C  X64FAKESTUB
  $000000000061DC0C  X64FAKESTUB
Call trace for block $00007F160375A3C0 size 29
  $00000000006F47DA  EXECUTEPREPARED,  line 8535 of ../SynDB.pas
  $00000000006E0919  REMOTEPROCESSMESSAGE,  line 4592 of ../SynDB.pas
  $00000000006F261A  PROCESSMESSAGE,  line 8162 of ../SynDB.pas
  $00000000006F47B5  EXECUTEPREPARED,  line 8531 of ../SynDB.pas
  $00000000006F47B5  EXECUTEPREPARED,  line 8531 of ../SynDB.pas
Call trace for block $00007F16015A7F40 size 34
  $00000000006F2C81  INTHEADERPROCESS,  line 8261 of ../SynDB.pas
  $00000000006F47DA  EXECUTEPREPARED,  line 8535 of ../SynDB.pas
  $00000000006F261A  PROCESSMESSAGE,  line 8162 of ../SynDB.pas
  $00000000006F47B5  EXECUTEPREPARED,  line 8531 of ../SynDB.pas
  $00000000006F47B5  EXECUTEPREPARED,  line 8531 of ../SynDB.pas
Call trace for block $00007F1603D4FB90 size 472
  $F0F0F0F0F0F0F0F0
  $F0F0F0F0F0F0F0F0

Offline

#15 2017-07-01 09:23:16

AOG
Member
Registered: 2014-02-24
Posts: 346

Re: TestSQL3 compiled by FPC memoryleak

After lengthy investigation, many of the above errors seem to be caused by compression not working as expected on Linux. Will investigate further.

Offline

#16 2017-07-01 21:56:38

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 9,274
Website

Re: TestSQL3 compiled by FPC memoryleak

Two fixes have been just committed - should work better now.

Thanks a lot Alfred for the debugging and report!
smile

Offline

Pages: 1

Board footer

Powered by FluxBB