mORMot Open Source

oz

Member

Registered: 2015-09-02

Posts: 98

Mem-leak issue with T*ObjArray out parameters in SOA methods + FIX

Hi Arnaud,

there is an issue with Methods returning T*ObjArray objects with current source version.
I have several SOA Services with following public look-a-like methods:

  IMyInterface=interface(ICQRSService)
    [some guid here]
    ...
    function GetData(out aDataArray: TDataObjArray): TCQRSResult;
    ...
  end;

With current sources all of those out parameters leak memory. T*ObjArray's parameters are not destroyed in AfterExecute().

in mORMot.pas, line 52022 (in constructor TInterfaceFactory.Create) there is following code:

      case ValueType of
      smvRawUTF8..smvWideString:
        Include(ValueKindAsm,vIsString);
      smvDynArray:
        if ObjArraySerializers.Find(ArgTypeInfo)<>nil then
          Include(ValueKindAsm,vIsObjArray);

ObjArraySerializers.Find(ArgTypeInfo) returns nil here. Going deeper into SynCommons.pas:

function TObjectListPropertyHashed.IndexOf(aObject: TObject): integer;
begin
  if fCount>0 then
    if fHashed then begin
      if not fHashValid then
        IntHashValid;
      result := fHash.HashFind(fHash.fHashElement(aObject,fHash.fHasher),PtrInt(aObject));
      if result>=0 then
        exit; // if found
    end else
    for result := 0 to fCount-1 do
      if IntComp(fList[result],aObject)=0 then
        exit;
  result := -1;
end;

The list is hashed (fHashed=true), fHash.HashFind returns result < 0 here which means the hash lookup failed.
Having a look at TObjectListPropertyHashed.Add shows that the list is not rehashed after adding new items. But imho it should be.

function TObjectListPropertyHashed.Add(aObject: TObject; out wasAdded: boolean): integer;
begin
  wasAdded := false;
  if self<>nil then
    if fHashed then begin
      if not fHashValid then    // <- That's the problem, fHashValid is true but we need to rehash
        IntHashValid;
      result := fHash.FindHashedForAdding(aObject,wasAdded,
        fHash.fHashElement(aObject,fHash.fHasher));
      if wasAdded then
        fList[result] := aObject;
    end else begin
      for result := 0 to fCount-1 do
        if IntComp(fList[result],aObject)=0 then
          exit;
      wasAdded := true;
      result := fHash.Add(aObject);
      if fCount>=TOBJECTLISTHASHED_START_HASHING_COUNT then
        fHashed := true;
    end
  else
    result := -1;
end;

Changing the implementation as following makes the mem leaks disappear.

function TObjectListPropertyHashed.Add(aObject: TObject; out wasAdded: boolean): integer;
begin
  wasAdded := false;
  if self<>nil then
    if fHashed then begin
      IntHashValid;  // "fHashValid:=false;" instead of "IntHashValid;" works too because the next .IndexOf() call is forced to rehash the list then.
      result := fHash.FindHashedForAdding(aObject,wasAdded,
        fHash.fHashElement(aObject,fHash.fHasher));
      if wasAdded then
        fList[result] := aObject;
    end else begin
      for result := 0 to fCount-1 do
        if IntComp(fList[result],aObject)=0 then
          exit;
      wasAdded := true;
      result := fHash.Add(aObject);
      if fCount>=TOBJECTLISTHASHED_START_HASHING_COUNT then
        fHashed := true;
    end
  else
    result := -1;
end;

Could you have a look at the fix and add it to trunk if ok?
Kind regard,
oz.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Mem-leak issue with T*ObjArray out parameters in SOA methods + FIX

The hash array should not be re-computed before each Add(), but only if needed.
FindHashedForAdding() will add the new hash in a safe and efficient way.

There was some redundant code for TOBJECTLISTHASHED_START_HASHING_COUNT  implementation, which was not needed any more, since TDynArrayHashed has now a similar mechanism.
I've made some refactoring, included a potential fix.
Please try http://synopse.info/fossil/info/1423af75e9

Thanks oz for the feedback, and detailed report!

oz

Member

Registered: 2015-09-02

Posts: 98

Re: Mem-leak issue with T*ObjArray out parameters in SOA methods + FIX

Hi Arnaud,

thanks for the quick reply, but unfortunately your modifications do not help after a short test. SOA methods with T*ObjArray OUT parameters still leak memory. I could create a small test-app to reproduce this issue if required, but you should be able to reproduce by simply calling any SOA method with a T*ObjArray OUT parameter.

Kind regards,
oz.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Mem-leak issue with T*ObjArray out parameters in SOA methods + FIX

I use TObjArray with out parameters in several projects with no problem.

oz

Member

Registered: 2015-09-02

Posts: 98

Re: Mem-leak issue with T*ObjArray out parameters in SOA methods + FIX

Hi Arnaud,

i created a sample app to reproduce the issue, but it doesn't happen over there.

But the problem is another one: after updating to current mORMot sources all my T*ObjArrays leak memory because they aren't recognized anymore.

The strange thing is:
I have a dedicated testsuite for DTO testing functionality which does not depend on server infrastructure.
If I remove the "TJSONSerializer.RegisterObjArrayForJSON([TypeInfo(TMyObjArray),TMy]);" call in those tests, then no memory is leaked.
If the call to RegisterObjArrayForJSON is not removed, then out params do leak memory.

In mORMot.pas, "procedure TServiceMethodExecute.AfterExecute;" the function "fDynArrays(i).Wrapper.Clear;" is called, but in "TDynArray.InternalSetLength" the method "GetIsObjArray" returns false for those arrays if registered via RegisterObjArrayForJSON. This means that class destructors are not called -> memory leak.

All my tests are done using latest trunk version and Delphi 7.

Hmm, i really don't undestand that behaviour right now.

RegisterObjArrayForJSON call done: mem leaks because destructors are not called.
RegisterObjArrayForJSON call not done: no leaks.

Sadly, I can't leave those calls in production
Do you have any idea what is happening here?

oz

Member

Registered: 2015-09-02

Posts: 98

Re: Mem-leak issue with T*ObjArray out parameters in SOA methods + FIX

Just to let you know: FindHashedForAdding seems to be ok.
Regarding my problem i was able to find the root of the problems. There's an issue with TJSONSerializer.RegisterObjArrayForJSON. See http://synopse.info/forum/viewtopic.php?id=3566 for more details.