Issue with TDynArrayHashed: failed Assertions/mem leaks

oz · 2016-09-22 11:21:03

Hi!
There seems to be a problem with TDynArrayHashed.

Following situation:
- There's one shared TSQLDBZEOSConnectionProperties connection to a Firebird legacy DB.
- Several legacy purpose SOA Services use this connection for direct SQL r/w access at SynDB level, no ORM is used here.
- One concrete services retrieves some ISQLDbRows and loops through that data using rows.Step()
- inside each of those loops several other ISQLDbRows are used to retrieve data using the same TSQLDBZEOSConnectionProperties instance.

Everything works fine for some loops, then EAssertionFailed is raised in TDynArrayHashed.HashAdd:

procedure TDynArrayHashed.HashAdd(const Elem; aHashCode: Cardinal; var result: integer);
var n,cap: integer;
begin
  n := Count;
  SetCount(n+1); // reserve space for a void element in array
  cap := Capacity;
  if cap*2-cap shr 3>=fHashsCount then begin
    // fHashs[] is too small -> recreate
    ReHash;
    result := HashFind(aHashCode,Elem); // fHashs[] has changed -> recompute
    assert(result<0);
  end;
  with fHashs[-result-1] do begin // HashFind returned negative index in fHashs[]
    Hash := aHashCode;
    Index := n;
  end;
  result := n;
end;

"assert(result<0);" fails.

Callstack:

TDynArrayHashed.HashAdd((kein Wert),1313814477,0)
TDynArrayHashed.FindHashedForAdding((kein Wert),True,1313814477)
TDynArrayHashed.AddAndMakeUniqueName('EMAIL')
TSQLDBZEOSStatement.ExecutePrepared
TSQLDBConnectionProperties.Execute('SELECT cd.Email from ContactData CD where cd.ContactDataID IN (SELECT lcp.ContactDataID from LinkCustomerPerson lcp WHERE lcp.CustomerID=?)',(...),nil,False)
...

Compiling with assertions disabled results in obviously working code but leaves memory leaks for every further call.
FastMM4 report for the first leak:

--------------------------------2016/9/22 12:22:13--------------------------------
A memory block has been leaked. The size is: 2244

This block was allocated by thread 0x18B8, and the stack trace (return addresses) at the time was:
403101 [system.pas][System][@ReallocMem][2550]
40752C [system.pas][System][DynArraySetLength][16060]
40760E [system.pas][System][@DynArraySetLength][16108]
461AF2 [SynCommons.pas][SynCommons][TDynArrayHashed.ReHash][44383]
460E2B [SynCommons.pas][SynCommons][TDynArrayHashed.Scan][43871]
460F5A [SynCommons.pas][SynCommons][TDynArrayHashed.HashAdd][43905]
461073 [SynCommons.pas][SynCommons][TDynArrayHashed.FindHashedForAdding][43943]
4610D1 [SynCommons.pas][SynCommons][TDynArrayHashed.AddAndMakeUniqueName][43955]
AD761E [ZDbcResultSetMetadata.pas][ZDbcResultSetMetadata][TZAbstractResultSetMetadata.GetColumnType][474]
B9CF92 [SynDBZeos.pas][SynDBZeos][TSQLDBZEOSStatement.ExecutePrepared][1113]
AA7233 [SynDB.pas][SynDB][TSQLDBConnectionPropertiesThreadSafe.ThreadSafeConnection][6438]

The block is currently used for an object of class: Unknown

The allocation number is: 127937

Current memory dump of 256 bytes starting at pointer address 7EBA8590:
CC 37 4F 4E 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Ì  7  O  N  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .

As far as I can say right now everything seems to be ok with my code. Afair it is save to share one TSQLDBZEOSConnectionProperties instance. It might be hard to create an app to reproduce the issue, but maybe all that informations above could help to identify if there is an issue TDynArrayHashed.

Kind regards,
oz.

mingda · 2016-09-22 13:15:02

what's your version , in 2915 TDynArrayHashed has a fix, see http://synopse.info/fossil/info/ddc1dc3 … bb6a4a827a

oz · 2016-09-22 13:22:21

mingda wrote:

what's your version , in 2915 TDynArrayHashed has a fix, see http://synopse.info/fossil/info/ddc1dc3 … bb6a4a827a

I'm using current 1.18.2983 (leaf [0505dd5792]) from today.

oz · 2016-10-17 07:09:01

There's an issue again with TDynArrayHashed.HashAdd() which i've discovered in production.
My Setup:
A Firebird 2.5 DB is beeing accessed from an SOA service directly through SynDBZeos.
Everything works as expected for the first time. Then at the 2nd/3rd call to the same method problems start.
Assertion "assert(result<0);" fails.

procedure TDynArrayHashed.HashAdd(const Elem; aHashCode: Cardinal; var result: integer);
var n,cap: integer;
begin
  n := Count;
  SetCount(n+1); // reserve space for a void element in array
  cap := Capacity;
  if cap*2-cap shr 3>=fHashsCount then
  {$ifdef UNDIRECTDYNARRAY}with InternalDynArray do{$endif} begin
    // fHashs[] is too small -> recreate
    if fCountP<>nil then
      dec(fCountP^); // don't rehash the latest entry (which may not be set)
    ReHash;
    if fCountP<>nil then
      inc(fCountP^);
    result := HashFind(aHashCode,Elem); // fHashs[] has changed -> recompute
    assert(result<0);
  end;
  with fHashs[-result-1] do begin // HashFind returned negative index in fHashs[]
    Hash := aHashCode;
    Index := n;
  end;
  result := n;
end;

Callstack:

TDynArrayHashed.HashAdd((kein Wert),14531870,0)
TDynArrayHashed.FindHashedForAdding((kein Wert),True,14531870)
TDynArrayHashed.AddAndMakeUniqueName('CommunicationID')
TSQLDBZEOSStatement.ExecutePrepared
TSQLDBConnectionProperties.Execute('select "CommunicationID",...

This is quite a serious bug for me because it happens in production environment. Arnaud, could you have a look at TDynArrayHashed please?

Last edited by oz (2016-10-17 07:10:16)

oz · 2016-10-17 08:00:32

I just helped myself with introducing an empty try...except...end block to TDynArrayHashed.HashAdd() for "eating" the exception. Things seem to work, my logs are full of "Assertion failed" exceptions, but at least the system on my customer's server is up again. But this is not the solution obviously.

ab · 2016-10-17 12:49:16

Please check http://synopse.info/fossil/info/05f616d381

oz · 2016-10-17 13:24:18

ab wrote:

Please check http://synopse.info/fossil/info/05f616d381

Issue is fixed with this version, thanks a lot!

ab · 2016-10-17 13:25:58

Nice to your that it works now.

This part of the code is a bit hard to maintain, since it is very optimized for speed more than readability, and used in several diverse usecases by the framework...
Hope it is stabilized now.

Sorry for the issue, especially on production.

oz · 2016-10-17 14:55:25

No worries, you're doing a great job maintaining this huge and fantastic project!
I'm the one who is to blame for releasing a build of our software without testing it enough
Gladly it was good enough to "eat" the exception without any further harm in this situation.
I tried to fix the root of this issue by myself, but had to give up after 10 minutes because a quick fix was required to get the system up again asap. I'm into mORMot for about a year now but there are parts who are still difficult to get into.
I've already pushed a new version for this customer with your fix included.

mORMot Open Source

#1 2016-09-22 11:21:03

Issue with TDynArrayHashed: failed Assertions/mem leaks

#2 2016-09-22 13:15:02

Re: Issue with TDynArrayHashed: failed Assertions/mem leaks

#3 2016-09-22 13:22:21

Re: Issue with TDynArrayHashed: failed Assertions/mem leaks

#4 2016-10-17 07:09:01

Re: Issue with TDynArrayHashed: failed Assertions/mem leaks

#5 2016-10-17 08:00:32

Re: Issue with TDynArrayHashed: failed Assertions/mem leaks

#6 2016-10-17 12:49:16

Re: Issue with TDynArrayHashed: failed Assertions/mem leaks

#7 2016-10-17 13:24:18

Re: Issue with TDynArrayHashed: failed Assertions/mem leaks

#8 2016-10-17 13:25:58

Re: Issue with TDynArrayHashed: failed Assertions/mem leaks

#9 2016-10-17 14:55:25

Re: Issue with TDynArrayHashed: failed Assertions/mem leaks

Board footer