Pages: 1

#1 2016-06-03 10:13:41

chapa
Member
Registered: 2012-04-30
Posts: 117

TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

Hi ab,

Running latest mORMot, XE7 64bit build and FastMM4991 FullDebugMode enabled.

Getting:

FastMM has detected an error during a FreeMem operation. The block footer has been corrupted. 

The block size is: 376

This block was allocated by thread 0x38C8, and the stack trace (return addresses) at the time was:
425A28 [FastMM4.pas][FastMM4][_ZN7Fastmm411DebugGetMemEx][8737]
4092AD [System][_ZN6System11_ReallocMemERPvx]
415529 [System][_ZN6System17DynArraySetLengthERPvS0_xPx]
415741 [System][_ZN6System18_DynArraySetLengthERPvS0_x]
E70EA6 [mORMotMongoDB.pas][mORMotMongoDB][_ZN13Mormotmongodb22TSQLRestStorageMongoDB13GetJSONValuesEN6System11AnsiStringTILt65535EEENS1_12DynamicArrayINS2_ILt65001EEEEEPN6Mormot15TJSONSerializerE][988]
E745AD [mORMotMongoDB.pas][mORMotMongoDB][_ZN13Mormotmongodb22TSQLRestStorageMongoDB10EngineListEN6System11AnsiStringTILt65001EEEbPx][1302]
9AA8E5 [mORMot.pas][mORMot][_ZN6Mormot14TSQLRestServer10EngineListEN6System11AnsiStringTILt65001EEEbPx][40319]
9874E4 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest11ExecuteListEPKNS_15TSQLRecordClassEiN6System11AnsiStringTILt65001EEE][33845]
983153 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest16MultiFieldValuesENS_15TSQLRecordClassEN6System11AnsiStringTILt65001EEES4_][32944]
98385E [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest8RetrieveEN6System11AnsiStringTILt65001EEEPNS_10TSQLRecordES3_][33004]
9848F4 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest8RetrieveEN6System11AnsiStringTILt65001EEEPKNS1_7TVarRecEiS6_iPNS_10TSQLRecordES3_][33217]

The block is currently used for an object of class: Unknown

The allocation number is: 439210

The current thread ID is 0x38C8, and the stack trace (return addresses) leading to this error is:
425C08 [FastMM4.pas][FastMM4][_ZN7Fastmm412DebugFreeMemEPv][8857]
409235 [System][_ZN6System8_FreeMemEPv]
41595D [System][_ZN6System14_DynArrayClearERPvS0_]
E7172D [mORMotMongoDB.pas][mORMotMongoDB][_ZN13Mormotmongodb22TSQLRestStorageMongoDB13GetJSONValuesEN6System11AnsiStringTILt65535EEENS1_12DynamicArrayINS2_ILt65001EEEEEPN6Mormot15TJSONSerializerE][1037]
E745AD [mORMotMongoDB.pas][mORMotMongoDB][_ZN13Mormotmongodb22TSQLRestStorageMongoDB10EngineListEN6System11AnsiStringTILt65001EEEbPx][1302]
9AA8E5 [mORMot.pas][mORMot][_ZN6Mormot14TSQLRestServer10EngineListEN6System11AnsiStringTILt65001EEEbPx][40319]
9874E4 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest11ExecuteListEPKNS_15TSQLRecordClassEiN6System11AnsiStringTILt65001EEE][33845]
983153 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest16MultiFieldValuesENS_15TSQLRecordClassEN6System11AnsiStringTILt65001EEES4_][32944]
98385E [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest8RetrieveEN6System11AnsiStringTILt65001EEEPNS_10TSQLRecordES3_][33004]
9848F4 [mORMot.pas][mORMot][_ZN6Mormot8TSQLRest8RetrieveEN6System11AnsiStringTILt65001EEEPKNS1_7TVarRecEiS6_iPNS_10TSQLRecordES3_][33217]

It seems 

item: array of TBSONElement

got corrupted, when 

item[itemcount].FromNext(row.Item.Data.DocList)

is iterated at some point

manually setting SetLength(item, 0) at the end or waiting "end;" compiler generated code is leading to memory corruption according FastMM4

Last edited by chapa (2016-06-03 10:16:12)

Offline

#2 2016-06-03 10:43:32

chapa
Member
Registered: 2012-04-30
Posts: 117

Re: TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

I guess the code

      while item[itemcount].FromNext(row.Item.Data.DocList) do begin
        inc(itemcount);
        if itemcount>itemsize then begin
          inc(itemsize);
          Setlength(item,itemsize); // a field was deleted from TSQLRecord
        end;
      end;

in some cases item[itemcount] can be out of range, addressing "future" record while row.Item.Data.DocList is not at its betEOF, leading in assigning TBSONElement record properties in not assigned heap. Later increasing item array Length may lead to unpredictable memory behaviour on next iterations on same heap area.
Or something like this smile

Offline

#3 2016-06-10 07:56:00

DigDiver
Member
Registered: 2013-04-29
Posts: 137

Re: TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

With the latest mORMot, I also have strange problems with mORMotMongoDB. Sometimes, I get AV in module  mORMot.pas in the destructor TSQLTable.Destroy, when I make a stress test of my service that uses MongoDB using JMeter. When I revert mORMotMongoDB.pas to the version from May 4, strange errors are gone.

Offline

#4 2016-06-10 10:57:56

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,620
Website

Re: TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

Please try http://synopse.info/fossil/info/2a162ee2a7

I've included another fix for TSQLRestStorageMongoDB.GetJSONValues instability.

Offline

#5 2016-06-10 12:50:12

DigDiver
Member
Registered: 2013-04-29
Posts: 137

Re: TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

It does not help.

Event Thread ID Time
First chance exception 0xC0000005 ACCESS_VIOLATION occurred at 0x00405C19, write of address 0x000004FE at 0x00405C19 (in D:\Project XE10\InboxMonitor\Win32\Debug\inboxmonitor.exe) 3108  15:29:42:455
0x00405C19 System::SysFreeMem + 0x81 in GETMEM.INC line 2996 in inboxmonitor.exe  3108  15:29:42:455
0x00406FF6 System::FreeMem + 0xA in System.pas line 4655 in inboxmonitor.exe  3108  15:29:42:455
0x0040DE68 System::DynArrayClear + 0x40 in System.pas line 34689 in inboxmonitor.exe  3108  15:29:42:455
0x0040D0F6 System::FinalizeArray + 0xFA in System.pas line 31964 in inboxmonitor.exe  3108  15:29:42:455
0x0040CFD5 System::FinalizeRecord + 0x29 in System.pas line 31638 in inboxmonitor.exe 3108  15:29:42:455
0x004098C3 System::TObject::CleanupInstance + 0x17 in System.pas line 16399 in inboxmonitor.exe 3108  15:29:42:455
0x004096C2 System::TObject::FreeInstance + 0xA in System.pas line 16193 in inboxmonitor.exe 3108  15:29:42:455
0x00409F01 System::ClassDestroy + 0x5 in System.pas line 17543 in inboxmonitor.exe  3108  15:29:42:455
0x006ABFA0 mORMot::TSQLTable::~TSQLTable + 0x3C in mORMot.pas line 25219 in inboxmonitor.exe  3108  15:29:42:455
0x0040984B System::TObject::Free + 0xB in System.pas line 16263 in inboxmonitor.exe 3108  15:29:42:455
0x006B606A mORMot::TSQLRecordFill::UnMap + 0xE6 in mORMot.pas line 29365 in inboxmonitor.exe  3108  15:29:42:455
0x006B5B73 mORMot::TSQLRecordFill::~TSQLRecordFill + 0x27 in mORMot.pas line 29261 in inboxmonitor.exe  3108  15:29:42:455
0x0040984B System::TObject::Free + 0xB in System.pas line 16263 in inboxmonitor.exe 3108  15:29:42:455
0x006B9A01 mORMot::TSQLRecord::~TSQLRecord + 0xB5 in mORMot.pas line 30346 in inboxmonitor.exe  3108  15:29:42:455
0x008F9CB1 User_Model::TGlockUser::~TGlockUser + 0x35 in User_Model.pas line 228 in inboxmonitor.exe  3108  15:29:42:455
0x0040984B System::TObject::Free + 0xB in System.pas line 16263 in inboxmonitor.exe 3108  15:29:42:455
0x009B2802 FBLServer_mongo::TBounceServer::CheckAPiUser + 0x14A in FBLServer_mongo.pas line 2336 in inboxmonitor.exe  3108  15:29:42:456
 First chance exception 0x0EEDFADE Delphi exception occurred at 0x75995B68 (class: EBSONException, message: "Incorrect supplied BSON document content")  8696  15:29:46:415
      0x75995B68 RaiseException + 0x48 in KernelBase.dll  8696  15:29:46:415
      0x00992C00 SynMongoDB::BSONParseLength + 0x48 in SynMongoDB.pas line 2917 in inboxmonitor.exe 8696  15:29:46:415
      0x00992FD6 SynMongoDB::BSONToJSON + 0x1A in SynMongoDB.pas line 3002 in inboxmonitor.exe  8696  15:29:46:415
      0x0099A8EA SynMongoDB::TMongoConnection::GetReply + 0x18E in SynMongoDB.pas line 5147 in inboxmonitor.exe 8696  15:29:46:415
      0x0099A627 SynMongoDB::TMongoConnection::GetCursor + 0x3B in SynMongoDB.pas line 5113 in inboxmonitor.exe 8696  15:29:46:415
      0x00999D06 SynMongoDB::TMongoConnection::GetRepliesAndFree + 0xC6 in SynMongoDB.pas line 4993 in inboxmonitor.exe 8696  15:29:46:415
      0x00999A5A SynMongoDB::TMongoConnection::GetBSONAndFree + 0x56 in SynMongoDB.pas line 4946 in inboxmonitor.exe  8696  15:29:46:415
      0x0099F4AD SynMongoDB::TMongoCollection::FindBSON + 0x4D in SynMongoDB.pas line 5894 in inboxmonitor.exe  8696  15:29:46:415
      0x009A592B mORMotMongoDB::TSQLRestStorageMongoDB::EngineList + 0x303 in mORMotMongoDB.pas line 1294 in inboxmonitor.exe 8696  15:29:46:415
      0x006E0144 mORMot::TSQLRestServer::EngineList + 0x8C in mORMot.pas line 40522 in inboxmonitor.exe 8696  15:29:46:416
      0x006C6D4A mORMot::TSQLRest::ExecuteList + 0x36 in mORMot.pas line 33981 in inboxmonitor.exe  8696  15:29:46:416
      0x006C3E48 mORMot::TSQLRest::MultiFieldValues + 0x64 in mORMot.pas line 33081 in inboxmonitor.exe 8696  15:29:46:416
      0x006B9C4B mORMot::TSQLRecord::TSQLRecord + 0x43 in mORMot.pas line 30380 in inboxmonitor.exe 8696  15:29:46:416
      0x009B4ADD FBLServer_mongo::TBounceServer::GetStats + 0x921 in FBLServer_mongo.pas line 2758 in inboxmonitor.exe  8696  15:29:46:416

one more:

First chance exception 0xC0000005 ACCESS_VIOLATION occurred at 0x00405C19, write of address 0x30543931 at 0x00405C19 (in D:\Project XE10\InboxMonitor\Win32\Debug\inboxmonitor.exe) 2940  15:29:46:391
      0x00405C19 System::SysFreeMem + 0x81 in GETMEM.INC line 2996 in inboxmonitor.exe  2940  15:29:46:391
      0x00406FF6 System::FreeMem + 0xA in System.pas line 4655 in inboxmonitor.exe  2940  15:29:46:391
      0x0040B681 System::LStrClr + 0x21 in System.pas line 24722 in inboxmonitor.exe  2940  15:29:46:391
      0x0099A23D SynMongoDB::TMongoConnection::Send + 0x175 in SynMongoDB.pas line 5072 in inboxmonitor.exe 2940  15:29:46:391
      0x0099A7E8 SynMongoDB::TMongoConnection::GetReply + 0x8C in SynMongoDB.pas line 5132 in inboxmonitor.exe  2940  15:29:46:391
      0x0099A627 SynMongoDB::TMongoConnection::GetCursor + 0x3B in SynMongoDB.pas line 5113 in inboxmonitor.exe 2940  15:29:46:391
      0x00999D06 SynMongoDB::TMongoConnection::GetRepliesAndFree + 0xC6 in SynMongoDB.pas line 4993 in inboxmonitor.exe 2940  15:29:46:391
      0x00999A5A SynMongoDB::TMongoConnection::GetBSONAndFree + 0x56 in SynMongoDB.pas line 4946 in inboxmonitor.exe  2940  15:29:46:391
      0x0099F4AD SynMongoDB::TMongoCollection::FindBSON + 0x4D in SynMongoDB.pas line 5894 in inboxmonitor.exe  2940  15:29:46:391
      0x009A592B mORMotMongoDB::TSQLRestStorageMongoDB::EngineList + 0x303 in mORMotMongoDB.pas line 1294 in inboxmonitor.exe 2940  15:29:46:391
      0x006E0144 mORMot::TSQLRestServer::EngineList + 0x8C in mORMot.pas line 40522 in inboxmonitor.exe 2940  15:29:46:391
      0x006C6DBF mORMot::TSQLRest::ExecuteJson + 0x23 in mORMot.pas line 33989 in inboxmonitor.exe  2940  15:29:46:391
      0x009B1642 FBLServer_mongo::TBounceServer::DownloadList + 0x13D6 in FBLServer_mongo.pas line 2153 in inboxmonitor.exe

Offline

#6 2016-06-13 14:17:45

DigDiver
Member
Registered: 2013-04-29
Posts: 137

Re: TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

Is there any progress?

Offline

#7 2016-06-13 14:54:11

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,620
Website

Re: TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

I was not able to reproduce the issue.

Without reproducible code, it is hard to find the cause!
wink

If there is an issue from the TSQLTableJson, it is not tied to the MongoDB storage, I'm afraid.

Offline

#8 2016-06-14 11:38:16

DigDiver
Member
Registered: 2013-04-29
Posts: 137

Re: TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

Try with this demo project.
With mORMotMongoDB-old.pas all works. But with the original mORMotMongoDB.pas the memory corrupt errors occurs.
To test I'm use JMeter Bounce API-test.jmx in Thread Group I'm set Number of threads(users) to 30 and Loop count to 10
https://www.dropbox.com/s/e6zez582qhjts … o.zip?dl=0

Offline

#9 2016-06-16 10:40:32

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,620
Website

Re: TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

I was able to reproduce the issue.

Fixed by http://synopse.info/fossil/info/85dbfbf833

Sorry for the delay!

Offline

#10 2016-06-17 12:22:50

chapa
Member
Registered: 2012-04-30
Posts: 117

Re: TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

Thanks, ab, it works for me

Offline

#11 2016-06-20 11:27:18

DigDiver
Member
Registered: 2013-04-29
Posts: 137

Re: TSQLRestStorageMongoDB.GetJSONValues Memory corrupt

Thanks, ab, it works for me too.

Offline

Pages: 1

Board footer

Powered by FluxBB