mORMot Open Source

tbo

Member

Registered: 2015-04-20

Posts: 337

Memory leak after Reduce()

I'm playing a bit with DocVariant functions where I can't find a test case as an example. The sample data is created as follows:

var
  item: Variant;
  list: Variant;
  reduced: Variant;
begin
  TDocVariant.New(item, [dvoIsObject]);
  TDocVariantData(list).Init([dvoIsArray]);
  for var idx: Integer := 0 to 2 do
  begin
    item.id := idx;
    item.source := StringToUtf8('source' + idx.ToString);
    item.target := StringToUtf8('target' + idx.ToString);
    TDocVariantData(list).AddItem(item);
  end;

This function works without any problem:

FileFromString(TDocVariantData(TDocVariantData(list).ReduceAsArray('source')).ToCsv, '_testData.csv');

This function creates the file without any problem. But when I close the program, a memory leak is reported:

TDocVariantData(list).Reduce(['source', 'target'], False, reduced);
FileFromString(TDocVariantData(reduced).ToCsv, '_testData.csv');

After Reduce() I would expect "reduced" Kind is dvArray, but it is dvUndefined. This memory will be lost:

An unexpected memory leak has occurred. The unexpected small block leaks are:
13 - 20 bytes: AnsiString x 2
21 - 28 bytes: AnsiString x 4, Unknown x 2
69 - 76 bytes: Unknown x 2

What am I doing wrong? The functional description does not give any clue.

With best regards
Thomas

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,303

Website

Re: Memory leak after Reduce()

I would rather use TDocVariantData variables, not Variant.
Then

list.AddItem(TDocVariant.NewObject(['id', idx, 'source', 'source'+idx.ToString, 'target', 'target'+idx.ToString));

Perhaps the problem comes from the item reused value.

Also note that the result of Reduce should be null before the call - or at least not reused.
So perhaps I would write

FileFromString(_Safe(list.Reduce(['source', 'target'], False,)^.ToCsv, '_testData.csv');

Note that _Safe()^ is better/safer than TDocVariantData().

tbo

Member

Registered: 2015-04-20

Posts: 337

Re: Memory leak after Reduce()

I would rather use TDocVariantData variables, not Variant.

Sorry Arnaud, I can't get it to work. Here the same problem as above. Function ReduceAsArray() works correctly, but after calling function Reduce(), a memory leak is reported when the program exits.

var
  item: TDocVariantData;
  list: TDocVariantData;
begin
  item.Init;
  list.Init([], dvArray);
  for var idx: Integer := 0 to 2 do
  begin
    item.I['id'] := idx;
    item.U['source'] := StringToUtf8('source' + idx.ToString);
    item.U['target'] := StringToUtf8('target' + idx.ToString);
    list.AddItem(Variant(item));
  end;

  FileFromString(_Safe(list.Reduce(['source', 'target'], False))^.ToCsv, '_testData.csv');
  // works! FileFromString(_Safe(list.ReduceAsArray('source'))^.ToCsv, '_testData.csv');
end;

Thank you for the support. Interesting detail: In my tests, mORMot became 1.8% faster in the last 3 days.

With best regards
Thomas

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,303

Website

Re: Memory leak after Reduce()

It was a one-liner.

Please try https://github.com/synopse/mORMot2/commit/21d72302
I included some associated regression tests.

tbo

Member

Registered: 2015-04-20

Posts: 337

Re: Memory leak after Reduce()

I included some associated regression tests.

Thank you for the bug fix. I saw that you used the following scheme in the test:

list.Init;
for var idx: Integer := 0 to 200000 do
begin
  item.Init;
  ...
  list.AddItem(Variant(item));
  item.Clear;
end;

Using the Init() and Clear() in the loop results in a runtime loss of almost 35%. This is significant. Is there anything against my variant?

With best regards
Thomas

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,303

Website

Re: Memory leak after Reduce()

Yes, it is as expected.
The Init + Clear pattern is just cleaner, and the paranoid way of writing such code.

For your purpose, see https://github.com/synopse/mORMot2/commit/f916683c
This new TDocVariantData.AddObject method should be the fastest way with no intermediary item variant.

And if you know how many items are about to be added, you can play with the capacity before the loop, for a small performance improvement.