Sudden TSynDictionary crash

okoba · 2021-04-21 17:05:38

I faced a weird issue with TSynDictionary.
On a specific Value size and a specific count, it will crash (in SynLZ) when I want to save the content.
If I change the size of TValue to 1000 or count to 30M, it works fine.
The error seems related to buffer size as even if I save to JSON, it will happen again in Base64 instead of SynLZ.
Tried it with Lazarus 64bit (Trunk and Fixes) on Windows10 and mORMot 1 & 2.

program project1;

uses
  mormot.core.base,
  mormot.core.json;

type
  TValue = record
    T: array[1..800] of Byte;
  end;
  TValueDynArray = array of TValue;
var
  D: TSynDictionary;
  I: Integer;
  V: TValue;
begin
  D := TSynDictionary.Create(TypeInfo(TIntegerDynArray), TypeInfo(TValueDynArray));
  for I := 1 to 2500000 do
    D.Add(I, V);
  D.SaveToBinary;
  WriteLn('Done');
  ReadLn;
end.

ab · 2021-04-21 20:51:03

It is not a TSynDictionary problem.
This is a SynLZ bug if the data is of the same pattern.
For real data (V not the same), it should not happen.

okoba · 2021-04-22 06:21:07

It will happen again on:
- Set CompressAlgo to nil
- Saving to JSON
- Using random value for V
I faced this problem in a project with real values with the same exact behavior.

To be clear, I tried any combination of the above in such code and also in the context of my project with real values:

begin
  D := TSynDictionary.Create(TypeInfo(TIntegerDynArray), TypeInfo(TValueDynArray));
  D.CompressAlgo := nil;
  for I := 1 to 2319851 do
  begin
    TAesPrng.Main.FillRandom(@V, SizeOf(V));
    D.Add(I, V);
  end;
  D.SaveToJson;
  WriteLn('Done');
  ReadLn;
end.

Last edited by okoba (2021-04-22 06:28:07)

ab · 2021-04-22 11:11:37

I will look into this.

ab · 2021-04-22 14:59:11

The problem is that this kind of value type is not supported:

TValue = record
    T: array[1..800] of Byte;
  end

The FPC compiler doesn't generate any RTTI for the record so the framework is messed up.

ab · 2021-04-22 15:08:32

Investigating further, I suspect the problem may be because 2500000 * 800 is around 2GB big, so the code has troubles with the data length.
For smaller content, no problem. So it is not a RTTI issue.

Such big content is not supported yet.
On Delphi there is a limitation to 32-bit for string lengths, even on 64-bit systems.
Even if FPC uses a PtrInt/SizeInt, we won't support it since it would break Delphi compatibility.

okoba · 2021-04-22 15:21:10

Thanks for the check but it seems not a size problem. As I stated before it had no problem with 1000 byte or 3 million item. Smaller and bigger is fine, the problem is around the 2.5M of 800 bytes.

ab · 2021-04-22 16:10:10

Even if you don't see any problem, there are certainly some overflow, and the output is incorrect - I guess truncated to a 32-bit boundary.

I have just added a 800MB max size limit for TBufferWriter over a TRawByteStringStream.
https://github.com/synopse/mORMot2/comm … 686f1720bd

okoba · 2021-04-22 16:15:41

Limiting prevents sudden issues. As for solution, I checked the result for 3M or 1000 bytes and load it after save, it worked correctly. I used this code for months and no problem and passed the checks until I faced this number.
Another question is why JSOn writer has a problem? I did use an external writer with owned file stream and still problem happens in Base64.

mORMot Open Source

#1 2021-04-21 17:05:38

Sudden TSynDictionary crash

#2 2021-04-21 20:51:03

Re: Sudden TSynDictionary crash

#3 2021-04-22 06:21:07

Re: Sudden TSynDictionary crash

#4 2021-04-22 11:11:37

Re: Sudden TSynDictionary crash

#5 2021-04-22 14:59:11

Re: Sudden TSynDictionary crash

#6 2021-04-22 15:08:32

Re: Sudden TSynDictionary crash

#7 2021-04-22 15:21:10

Re: Sudden TSynDictionary crash

#8 2021-04-22 16:10:10

Re: Sudden TSynDictionary crash

#9 2021-04-22 16:15:41

Re: Sudden TSynDictionary crash

Board footer