#1 2020-11-12 11:53:27

zhao maosen
Member
Registered: 2020-07-20
Posts: 4

Synbigtable has memory leak

Delphi 10.2.3 mormot trunk 1.18

synbigtable Set every 4m to write to disk, but the memory still increases rapidly, almost the same size as the disk file

var
  DBFile: string;
  BigTable: TSynBigTableRecord;
  Rec: TSynTableData;
  ColProper: TSynTableFieldProperties;
  I: Int32;
begin
  try
    DBFile := ExtractFilePath(ParamStr(0)) + 'test.idx';
    if FileExists(DBFile) then
      DeleteFile(DBFile);

    BigTable := TSynBigTableRecord.Create(DBFile, 'test');
    BigTable.AddField('Column_1', TSynTableFieldType.tftUTF8, []);
    BigTable.AddFieldUpdate;

    Rec.Init(BigTable.Table);
    ColProper := BigTable.Table.Field[0];

    for I := 1 to 10000000 do
    begin
      Rec.SetFieldSBFValue(ColProper, ColProper.SBF(StringToUTF8('ABCDEFGHJI')));
      BigTable.RecordAdd(Rec);
      if BigTable.CurrentInMemoryDataSize > 4 * 1024 * 1024 then
        BigTable.UpdateToFile(True);
    end;
end.

Offline

#2 2020-11-12 12:53:17

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 11,692
Website

Re: Synbigtable has memory leak

Don't use SynBigTable. It is not used on production since years.

You need to call BigTable.Free once you are done with the instance.
About memory consumption, which memory indicator do you use? When the data is on disk, it is mapped on memory so in fact it may appear in the project virtual memory, but it is not allocated by GetMem/FastMM4.
So it is not a memory leak.

Offline

#3 2020-11-12 18:50:21

zhao maosen
Member
Registered: 2020-07-20
Posts: 4

Re: Synbigtable has memory leak

I already use SynBigTable in the product. Because I have to use it throughout the life of the application, I can't release it in time.

Using fastmm4 leak detection, a large number of leaks like this are found:

This block was allocated by thread 0x37A8, and the stack trace (return addresses) at the time was:
41FEE9 [Fastmm4.pas][FastMM4][_ZN7Fastmm411DebugGetMemEx][8737]
409454 [System.pas][System][_ZN6System7_GetMemEx][4732]
410034 [System.pas][System][_ZN6System14_NewAnsiStringEit][24699]
41045C [System.pas][System][_ZN6System17_LStrFromPCharLenERNS_11AnsiStringTILt0EEEPcit][25430]
5BA9D7 [SynTable.pas][SynTable][_ZN8Syntable9TSynTable15UpdateFieldDataEPciiRN6System11AnsiStringTILt65535EEES4_][5690]
5C0E43 [SynTable.pas][SynTable][_ZN8Syntable13TSynTableData16SetFieldSBFValueEPNS_24TSynTableFieldPropertiesEN6System11AnsiStringTILt65535EEE][7864]
5D119B [BigTableTest.dpr][BigTableTest][_ZN12Bigtabletest14initializationEv][67]
7FFFE69F7974
7FFFE78EA271

The block is currently used for an object of class: AnsiString

The allocation number is: 2906027

Current memory dump of 256 bytes starting at pointer address 7FF4F6DD5FE0:
04 09 61 00 A8 03 01 00 01 00 00 00 0D 00 00 00 0C 41 42 43 44 45 46 47 48 4A 49 4B 4C 00 E2 D8
C8 67 10 ED CF FF 80 80 80 80 80 80 80 80 80 80 00 00 00 00 00 00 00 00 C0 5E DD F6 F4 7F 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60 FE 41 00 00 00 00 00 00 00 00 00 A8 57 2C 00
E9 FE 41 00 00 00 00 00 54 94 40 00 00 00 00 00 34 00 41 00 00 00 00 00 5C 04 41 00 00 00 00 00
D7 A9 5B 00 00 00 00 00 43 0E 5C 00 00 00 00 00 9B 11 5D 00 00 00 00 00 74 79 9F E6 FF 7F 00 00
71 A2 8E E7 FF 7F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 A8 37 00 00 A8 37 00 00
23 01 42 00 00 00 00 00 84 94 40 00 00 00 00 00 59 01 41 00 00 00 00 00 F7 01 41 00 00 00 00 00
F4 27 41 00 00 00 00 00 66 3E 41 00 00 00 00 00 6A DE 5C 00 00 00 00 00 FC 11 5D 00 00 00 00 00
.  .  a  .  ? .  .  .  .  .  .  .  .  .  .  .  .  A  B  C  D  E  F  G  H  J  I  K  L  .  ? ?
? g  .  ? ?   €  €  €  €  €  €  €  €  €  €  .  .  .  .  .  .  .  .  ? ^  ? ? ?   .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  `  ? A  .  .  .  .  .  .  .  .  .  ? W  ,  .
? ? A  .  .  .  .  .  T  ? @  .  .  .  .  .  4  .  A  .  .  .  .  .  \  .  A  .  .  .  .  .
? ? [  .  .  .  .  .  C  .  \  .  .  .  .  .  ? .  ]  .  .  .  .  .  t  y  ? ?     .  .
q  ? ? ?     .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  ? 7  .  .  ? 7  .  .
#  .  B  .  .  .  .  .  ? ? @  .  .  .  .  .  Y  .  A  .  .  .  .  .  ? .  A  .  .  .  .  .
? '  A  .  .  .  .  .  f  >  A  .  .  .  .  .  j  ? \  .  .  .  .  .  ? .  ]  .  .  .  .  .

--------------------------------2020/11/12 19:26:32--------------------------------
A memory block has been leaked. The size is: 40

This block was allocated by thread 0x37A8, and the stack trace (return addresses) at the time was:
41FEE9 [Fastmm4.pas][FastMM4][_ZN7Fastmm411DebugGetMemEx][8737]
409454 [System.pas][System][_ZN6System7_GetMemEx][4732]
410034 [System.pas][System][_ZN6System14_NewAnsiStringEit][24699]
41045C [System.pas][System][_ZN6System17_LStrFromPCharLenERNS_11AnsiStringTILt0EEEPcit][25430]
5BA9D7 [SynTable.pas][SynTable][_ZN8Syntable9TSynTable15UpdateFieldDataEPciiRN6System11AnsiStringTILt65535EEES4_][5690]
5C0E43 [SynTable.pas][SynTable][_ZN8Syntable13TSynTableData16SetFieldSBFValueEPNS_24TSynTableFieldPropertiesEN6System11AnsiStringTILt65535EEE][7864]
5D119B [BigTableTest.dpr][BigTableTest][_ZN12Bigtabletest14initializationEv][67]
7FFFE69F7974
7FFFE78EA271

Offline

#4 2020-11-12 19:00:57

macfly
Member
From: Brasil
Registered: 2016-08-20
Posts: 276

Re: Synbigtable has memory leak

zhao maosen wrote:

I already use SynBigTable in the product. Because I have to use it throughout the life of the application, I can't release it in time.

But are you destroying the BigTable object before finishing the application?
At some point it must be destroyed.

This is the first step.

Offline

#5 2020-11-13 01:33:53

zhao maosen
Member
Registered: 2020-07-20
Posts: 4

Re: Synbigtable has memory leak

I mean that memory usage cannot be increased indefinitely when data is written to disk. This file can be very large.

Do you mean when writing to a certain number, release it and open writing again?

Offline

#6 2020-11-13 03:46:28

zhao maosen
Member
Registered: 2020-07-20
Posts: 4

Re: Synbigtable has memory leak

Process Explorer shows that physical memory is occupied instead of virtual memory, which is almost the same size as files written to disk.
Is there a plan to fix ?

Offline

#7 2020-11-13 11:22:28

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 11,692
Website

Re: Synbigtable has memory leak

Using memory mapping is a feature of the BigTable engine.
So it is not a bug to fix, but a feature.
If you don't have enough memory, then switch to something else, e.g. SQLite3.

Offline

Board footer

Powered by FluxBB