[FIXED] ToJSON, VariantSaveJSON mem leaks for JSONs larger than ~40KB

Eugene Ilyin · 2020-04-08 01:17:03

Hi,

I have an issue with any JSON serialization for content size more than ~40KB.

Minimal code to reproduce:

program ToJSONMemoryLeak;

{$APPTYPE CONSOLE}

uses
  SynCommons;

var
  Index: Integer;
  JSON: RawUTF8;
  V: Variant;

begin
  ReportMemoryLeaksOnShutdown := True;

  // Let's build some DocVariant array of objects [{...}, {...}, ...]
  JSON := '[';
  for Index := 1 to 1400 do
    JSON := JSON + '{"name":"value","prop":false},';
  JSON[Length(JSON)] := ']';

  V := _Json(JSON);

  TDocVariantData(V).ToJSON; // Houston we have a problem
end.

On the recent 1.18.5940 (for both 32-bit and 64-bit platforms) on Delphi 10.3 Rio I have:

Unexpected Memory Leak
An unexpected memory leak has occurred. The sizes of unexpected leaked medium and large blocks are: 16424

Seems like regression in memory management or JSON serialization structures was introduced.

Last edited by Eugene Ilyin (2020-04-09 00:04:51)

macfly · 2020-04-08 01:49:14

I confirm this on Lazarus 2.0.7 + FPC 3.2.

Leak report: https://pastebin.com/dgLEYxGR

Eugene Ilyin · 2020-04-08 16:15:37

Thanks macfly,

The issue is in internal buffer management of TTextWriter.

When the data is bigger than provided external stack-allocated buffer - the TTextWriter code is switching to heap memory allocation
in SynCommons:55649

      if twoBufferIsExternal in fCustomOptions then // use heap, not stack
        exclude(fCustomOptions,twoBufferIsExternal) else
        FreeMem(fTempBuf); // with big content comes bigger buffer
      GetMem(fTempBuf,fTempBufSize);

As you see the twoBufferIsExternal flag is excluded from fCustomOptions to indicate that we must release memory later in TTextWriter.Destroy or during the future internal buffer reallocations.

But! Externally the TDocVariant.ToJSON code restores fCustomOptions from some strange backup variable (?)
As the result twoBufferIsExternal flag is brings back to fCustomOptions and all allocated by the TTextWriter heap memory is never released.

SynCommons:48215

      W.Add(']');
    end;
    W.fCustomOptions := backup;
  end else

Weird solution

Restore from this backup variable all flags except twoBufferIsExternal.

    W.fCustomOptions := backup - [twoBufferIsExternal] + W.fCustomOptions * [twoBufferIsExternal];

Better solution

Refactor TDocVariant.ToJSON code and not use the protected section of TTextWriter class by the external TDocVariant class.

As for now any JSON requests, serialization, or data transfer larger than ~40KB (~4Kb gzip) provides memory leaks on server

Last edited by Eugene Ilyin (2020-04-09 00:02:50)

ab · 2020-04-08 23:19:37

Oups!
It was something weird indeed.
Thanks a lot for debugging it!

Please check https://synopse.info/fossil/info/b8528d8633
It shouldn't leak memory any more.

Eugene Ilyin · 2020-04-09 00:02:14

Thanks ab,

All allocated memory now released property after the 1.18.5944 release.

ab · 2020-04-09 07:52:26

I have added a regression test for TDocVariantData.ToJSON/TTextWriter with huge JSON, to detect any memory leak as reported earlier.

Check https://synopse.info/fossil/info/f9522171d3

Eugene Ilyin · 2020-04-09 14:07:08

Good point!

I was thinking how to integrate this check and prevent such leaks in future and remembered, that SynSelfTests check for mem leaks on complete.

Thanks

mORMot Open Source

#1 2020-04-08 01:17:03