Memory managment in mORMot question

ASiwon · 2015-03-12 13:01:15

Hello,

I found in mORMots code this piece of code:

  for i := GarbageCollectorFreeAndNilList.Count-1 downto 0 do // LIFO
  try
    if PObject(GarbageCollectorFreeAndNilList.List[i])^<>nil then
      FreeAndNil(PObject(GarbageCollectorFreeAndNilList.List[i])^);
  except
    on E: Exception do
      ; // just ignore exceptions in client code destructors
  end;

It is in SynCommons.GarbageCollectorFree. Can you explain what kind of errors could exist in client code destructors? In my application in this piece Access Violation exception is raised for every model element. But in fact I have no idea where is the problem in my code.

ab · 2015-03-12 13:15:37

There is no issue with the framework code itself, AFAIK.
We test the framework with FastMM4 in full debug mode .

But if the user code is buggy (e.g. release the instance twice), this except block is to ensure that any such buggy code won't stop the release of other instances.
So I guess you are strongly release some class instance, which should not.
Try to run FastMM4 in full debug mode in your app to find the error root cause.

ASiwon · 2015-03-13 09:52:56

Hello,

I'm using FastMM4 for testing my applications too. And now if I'm trying to test my application which is using mORMot then depending on configuration I have memory leaks or exceptions in GarbageCollectorFree procedure. I think that it is not framework related question, because all samples are working correctly with no FastMM4 reports but I'm writing because I don't see anything wrong in my code. I don't see anything wrong but it doesn't mean that code is correct.

Is it possible that this is applications architecture related problem? In my case exception in GarbageCollectorFree occurs only for items with TSQLRecordProperties class instances object. Exactly three times (my simple model has only 3 classes) The main different from samples is that model classes are defined in dynamically loaded package (using SysUtils.LoadPackage method) So GarbageCollectorFree is called from System.FinalizeUnits procedure when packages were unloaded. I think it could works correctly when the classes would be removed manually from model when package is unloaded. Is it possible to manually remove a class from model? Unfortunatelly I can't find RemoveTable or DeleteTable in TSQLModel methods list.

ab · 2015-03-13 10:10:23

The problem may indeed come from the fact that there are two GarbageCollectorFree calls... one in the main program, and another in the package.

In fact, even RemoveTable/DeleteTable would not be sufficient, since the global "garbage collector" list is filled from RTTI information and JSON serialization.
Perhaps TJSONSerializer.RegisterClassForJSON(aTable) is the problem.
What does full debug mode say to you? What is the failing class type which is released twice?

For packages and libraries, we may have to maintain a shared list of global memory structures...
Not only for GarbageCollectorFreeAndNil, but also perhaps for TSynAnsiConvert, and so on...
A bit tricky to implement (but possible).

What if we just get rid of GarbageCollectorFree in the SynCommons finalization unit when a package is unloaded?
This would definitively leak a little bit of memory, not only a few bytes, I guess...
It may be a temporary workaround, certainly better than the Access Violations themselves...

ASiwon · 2015-03-13 13:10:51

The problem may indeed come from the fact that there are two GarbageCollectorFree calls... one in the main program, and another in the package.

There is only one GarbageCollectorFree call - from main program. It was tested using debugger. I'm using bpl packages, not dll libraries. MORMot is also compiled in separate package. Delphi compiler adds some code which prevents to calling more than once initialization or finalization units sections while packages are loaded or unloaded.

In fact, even RemoveTable/DeleteTable would not be sufficient, since the global "garbage collector" list is filled from RTTI information and JSON serialization.
Perhaps TJSONSerializer.RegisterClassForJSON(aTable) is the problem.
What does full debug mode say to you? What is the failing class type which is released twice?

The problem is that Access Violation exception is occured not for FreeAndNil call but in the line with condition:

if PObject(GarbageCollectorFreeAndNilList.List[i])^<>nil then

and this is not problem with GarbageCollectorFreeAndNilList variable. This variable has not nil value. For whole list which has 14 items problem occurs only for items: 9, 10, 11. Items before and after those items are freed correctly. Assembler code for this line looks like:

SynCommons.pas.43108: if PObject(GarbageCollectorFreeAndNilList.List[i])^<>nil then
008D9F8B A1B4F7AB00       mov eax,[$00abf7b4]
008D9F90 8B4004           mov eax,[eax+$04]
008D9F93 8B55FC           mov edx,[ebp-$04]
008D9F96 8B0490           mov eax,[eax+edx*4]
008D9F99 833800           cmp dword ptr [eax],$00
008D9F9C 7413             jz $008d9fb1

and AV is raised on line: cmp dword ptr [eax],$00

Of course exceptions are ignored and FastMM4 shows report:

This application has leaked memory. The small block leaks are (excluding expected leaks registered by pointer):

1 - 84 bytes: TSQLPropInfoRTTIInt64 x 1, TSQLPropInfoRTTIDateTime x 1, TSQLPropInfoRTTITID x 3, TSQLPropInfoRTTIInt32 x 9, TSQLPropInfoRTTIRawBlob x 2, TSQLPropInfoRTTIRawUTF8 x 4, TSQLPropInfoRTTIUnicode x 8, TSQLPropInfoRTTIEnum x 6, TSQLPropInfoList x 3, TSQLRecordFill x 33, UnicodeString x 5685, AnsiString x 45, Unknown x 4
85 - 292 bytes: AnsiString x 1, Unknown x 949
293 - 516 bytes: TSQLRecordProperties x 3, AnsiString x 2, Unknown x 33
Note: Memory leak detail is logged to a text file in the same folder as this application. To disable this memory leak check, undefine "EnableMemoryLeakReporting".

and detailed report for TSQLRecordProperties looks like:

--------------------------------2015/3/13 12:42:45--------------------------------
A memory block has been leaked. The size is: 516

This block was allocated by thread 0x1468, and the stack trace (return addresses) at the time was:
5005995A [System.pas][System][System.@GetMem][4189]
5005EA1B [System.pas][System][System.TObject.NewInstance][15174]
5005F23E [System.pas][System][System.@ClassCreate][16486]
A92268 [mORMot.pas][mORMot][TSQLRecordProperties.$bctr$qqrp17System.TMetaClass][36823]
739336 [GetRawStackTrace]
402B4E [FastMM4][FastMM4.CalculateHeaderCheckSum]
739336 [GetRawStackTrace]
739336 [GetRawStackTrace]
A6B294 [mORMot.pas][mORMot][TSQLModel.GetTableIndexExisting$qqrp17System.TMetaClass][25276]
403BA0 [FastMM4][FastMM4.DebugGetMem]
A67EB1 [mORMot.pas][mORMot][PropsCreate$qqrp17System.TMetaClass][24379]
5005EA24 [System.pas][System][System.TObject.NewInstance][15174]
A696FC [mORMot.pas][mORMot][TSQLModelRecordProperties.$bctr$qqrp16Mormot.TSQLModelp17System.TMetaClass28Mormot.TSQLRecordVirtualKind][24866]
A6A1C6 [mORMot.pas][mORMot][TSQLModel.SetTableProps$qqri][25018]
402B4E [FastMM4][FastMM4.CalculateHeaderCheckSum]
402B5D [FastMM4][FastMM4.UpdateHeaderAndFooterCheckSums]
403E54 [FastMM4][FastMM4.DebugReallocMem]
403E66 [FastMM4][FastMM4.DebugReallocMem]
50059997 [System.pas][System][System.@ReallocMem][4311]
50064245 [System.pas][System][System.DynArraySetLength][32317]
A6A8AC [mORMot.pas][mORMot][TSQLModel.AddTable$qqrp17System.TMetaClasspi][25099]
5BB27CB 
108942B [System.Generics.Collections.pas][GxRESTServerClasses][GxRESTServerClasses.{System.Generics.Collections}TList<GxRESTServerIntf.IGxModelRegister>.GetItem][680]
10873BA [GxRESTServerClasses.pas][GxRESTServerClasses][GxRESTServerClasses.TGxModelItems.ModelPrepare][229]
A0DBB0 [SynLog.pas][SynLog][TSynLog.Log$qqr22Syncommons.TSynLogInfox20System.UnicodeStringp14System.TObject][2860]
1088B06 [GxRESTServerClasses.pas][GxRESTServerClasses][GxRESTServerClasses.TGxRESTServerCreator.CreateServer][491]
50065B94 [System.pas][System][System.@IntfCopy][34063]
50065B83 [System.pas][System][System.@IntfClear][34016]
1090701 [GxRESTServerStdFactory.pas][GxRESTServerStdFactory][GxRESTServerStdFactory.TGxRESTServerFactory.Build][114]
1090A99 [GxRESTServerStdFactory.pas][GxRESTServerStdFactory][GxRESTServerStdFactory.TGxRESTServerFactory.BuildAndRun][125]
406A71 [uBkSerwerMain.pas][uBkSerwerMain][uBkSerwerMain.TfBkServerMain.actServiceStartExecute][64]
5016F382 [System.Classes.pas][System.Classes][System.Classes.TBasicAction.Execute][16354]
5043C27C [Vcl.ActnList.pas][Vcl.ActnList][Vcl.ActnList.TCustomAction.Execute][284]
5016F224 [System.Classes.pas][System.Classes][System.Classes.TBasicActionLink.Execute][16270]
5045303B [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TControl.Click][7338]
50477C7F [Vcl.StdCtrls.pas][Vcl.StdCtrls][Vcl.StdCtrls.TCustomButton.Click][5313]
504787F9 [Vcl.StdCtrls.pas][Vcl.StdCtrls][Vcl.StdCtrls.TCustomButton.CNCommand][5774]
50452AFC [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TControl.WndProc][7224]
72A5DF35 [Unknown function at HIMAGELIST_QueryInterface]
77057415 [Unknown function at RtlpNtSetValueKey]
76FB9AC4 [RtlDeactivateActivationContextUnsafeFast]
76FB9A10 [RtlActivateActivationContextUnsafeFast]
76FB9AC4 [RtlDeactivateActivationContextUnsafeFast]
74ED8E71 [CallNextHookEx]
74ED90D1 [Unknown function at CallNextHookEx]
74ED913E [Unknown function at CallNextHookEx]
74ED90E8 [Unknown function at CallNextHookEx]
74ED9115 [Unknown function at CallNextHookEx]
74ED8FCE [Unknown function at CallNextHookEx]
74ED913E [Unknown function at CallNextHookEx]
504575BA [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.WndProc][10039]
76AA20FE [Unknown function at GlobalFindAtomW]
50477929 [Vcl.StdCtrls.pas][Vcl.StdCtrls][Vcl.StdCtrls.TButtonControl.WndProc][5150]
50452737 [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TControl.Perform][7002]
50457720 [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.DoControlMsg][10108]
504581B0 [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.WMCommand][10383]
5057A9E5 [Vcl.Forms.pas][Vcl.Forms][Vcl.Forms.TCustomForm.WMCommand][6154]
50452AFC [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TControl.WndProc][7224]
50452AFC [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TControl.WndProc][7224]
6680CA49 [Unknown function at GetThemeTransitionDuration]
74EE88CF [Unknown function at GetMonitorInfoW]
74EE88E7 [Unknown function at GetMonitorInfoW]
74EE8850 [GetMonitorInfoW]
74EE885E [GetMonitorInfoW]
6680CC36 [Unknown function at GetThemeTransitionDuration]
74EDAD27 [Unknown function at PostMessageW]
504575BA [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.WndProc][10039]
5005FA89 [System.pas][System][System.TMonitor.TryEnter][17109]
5005F61D [System.pas][System][System.TMonitor.Enter][16808]
5005F4C9 [System.pas][System][System.TMonitor.CheckOwningThread][16730]
5005F79F [System.pas][System][System.TMonitor.Exit][16911]
505777FE [Vcl.Forms.pas][Vcl.Forms][Vcl.Forms.TCustomForm.WndProc][4388]
50456BFB [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.MainWndProc][9751]
50456C12 [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.MainWndProc][9754]
50456BFB [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.MainWndProc][9751]
50170092 [System.Classes.pas][System.Classes][System.Classes.StdWndProc][16860]
74ED8E71 [CallNextHookEx]
74ED90D1 [Unknown function at CallNextHookEx]
74ED8FCE [Unknown function at CallNextHookEx]
74ED932C [Unknown function at CallNextHookEx]
74F07924 [Unknown function at EndTask]
74ED9529 [Unknown function at CallNextHookEx]
76FAC6CC [ZwCallbackReturn]
74EDE4A9 [SendMessageW]
72A8139B [Unknown function at AddMRUStringW]
72A814E2 [Unknown function at AddMRUStringW]
72A81432 [Unknown function at AddMRUStringW]
74ED8E71 [CallNextHookEx]
74ED90D1 [Unknown function at CallNextHookEx]
74ED913E [Unknown function at CallNextHookEx]
74ED90E8 [Unknown function at CallNextHookEx]
5016EDE0 [System.Classes.pas][System.Classes][System.Classes.TComponent.UpdateAction][16030]
5057C6A2 [Vcl.Forms.pas][Vcl.Forms][Vcl.Forms.ProcessUpdate][7244]
5005EE17 [System.pas][System][System.@IsClass][15664]
5057C718 [Vcl.Forms.pas][Vcl.Forms][Vcl.Forms.TraverseClients3][7261]
76FB9A10 [RtlActivateActivationContextUnsafeFast]
5057C77F [Vcl.Forms.pas][Vcl.Forms][Vcl.Forms.TCustomForm.CMActionUpdate][7283]
74ED8E71 [CallNextHookEx]
74ED90D1 [Unknown function at CallNextHookEx]
74ED8FCE [Unknown function at CallNextHookEx]
74EDDDD5 [CallWindowProcW]
504576CB [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.DefaultHandler][10080]
504534A7 [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TControl.WMLButtonUp][7473]
504575BA [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.WndProc][10039]
50452AFC [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TControl.WndProc][7224]
505777FE [Vcl.Forms.pas][Vcl.Forms][Vcl.Forms.TCustomForm.WndProc][4388]
74ED913E [Unknown function at CallNextHookEx]
74ED90E8 [Unknown function at CallNextHookEx]
50452737 [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TControl.Perform][7002]
50582B0A [Vcl.Forms.pas][Vcl.Forms][Vcl.Forms.TApplication.DispatchAction][11494]
505801E8 [Vcl.Forms.pas][Vcl.Forms][Vcl.Forms.TApplication.WndProc][9857]
5005F79F [System.pas][System][System.TMonitor.Exit][16911]
5005F7F0 [System.pas][System][System.TMonitor.Exit][16933]
50434B58 [Vcl.Graphics.pas][Vcl.Graphics][Vcl.Graphics.FreeMemoryContexts][7047]
50456BFB [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.MainWndProc][9751]
76FB9AC4 [RtlDeactivateActivationContextUnsafeFast]
76FB9AC4 [RtlDeactivateActivationContextUnsafeFast]
74EDAC68 [SetTimer]
74EE1415 [GetCapture]
50456DF0 [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.IsControlMouseMsg][9807]
74ED8E71 [CallNextHookEx]
504575BA [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.WndProc][10039]
74ED913E [Unknown function at CallNextHookEx]
74A9FBF6 [Unknown function at CtfImeDestroyInputContext]
50477929 [Vcl.StdCtrls.pas][Vcl.StdCtrls][Vcl.StdCtrls.TButtonControl.WndProc][5150]
50456BFB [Vcl.Controls.pas][Vcl.Controls][Vcl.Controls.TWinControl.MainWndProc][9751]
50170092 [System.Classes.pas][System.Classes][System.Classes.StdWndProc][16860]
74ED8E71 [CallNextHookEx]
74ED90D1 [Unknown function at CallNextHookEx]
74ED8FCE [Unknown function at CallNextHookEx]
76FB09D6 [KiUserCallbackDispatcher]
74EDA66F [Unknown function at GetMessageW]
74EDA6E0 [DispatchMessageW]
50580ECF [Vcl.Forms.pas][Vcl.Forms][Vcl.Forms.TApplication.HandleMessage][10318]
5058120A [Vcl.Forms.pas][Vcl.Forms][Vcl.Forms.TApplication.Run][10456]
408512 [BkSerwerApp][BkSerwerApp.BkSerwerApp][14]
76AA7C04 [BaseThreadInitThunk]
76FCB5AF [Unknown function at RtlInitializeExceptionChain]
76FCB57A [Unknown function at RtlInitializeExceptionChain]

The block is currently used for an object of class: TSQLRecordProperties

The allocation number is: 2437

What if we just get rid of GarbageCollectorFree in the SynCommons finalization unit when a package is unloaded?
This would definitively leak a little bit of memory, not only a few bytes, I guess...
It may be a temporary workaround, certainly better than the Access Violations themselves...

Maybe it is a solution, but it will makes memory leaks tests a little harder. Memory leaks in this case is not a problem - this code is called only once - on application turn off but it will be reported everytime and this is not good. Maybe better solution is add possibility to call GarbageCollectorFree manually? I can call it just before package unloading. Packages unloading it is the one of the last things which I do before application turn off and it will be no difference - from finalization section or manually. I made small test - if I call GarbageCollectorFree procedure before package unloading then everything works correctly - no memory leaks, no exceptions are raised. There should be only no call to the mORMot code after calling GarbageCollectorFree. For example calling log procedures are also prohibited. The only change I made in SynCommons code is additional condition to prevent call GarbageCollectorFree twice (manually and from finalization section) and header for procedure in interface section. After changes procedure looks like:

procedure GarbageCollectorFree;
var i: integer;
begin
  if GarbageCollectorFreeing then Exit;

  GarbageCollectorFreeing := true;
  for i := GarbageCollector.Count-1 downto 0 do // last in, first out
  try
    GarbageCollector.Delete(i); // will call GarbageCollector[i].Free
  except
    on E: Exception do
      ; // just ignore exceptions in client code destructors
  end;
  for i := GarbageCollectorFreeAndNilList.Count-1 downto 0 do // LIFO
  try
    if PObject(GarbageCollectorFreeAndNilList.List[i])^<>nil then
      FreeAndNil(PObject(GarbageCollectorFreeAndNilList.List[i])^);
  except
    on E: Exception do
      ; // just ignore exceptions in client code destructors
  end;
  FreeAndNil(GarbageCollectorFreeAndNilList);
end;

ASiwon · 2015-03-14 10:00:10

Can you add to the mORMot source code, changes to the GarbageCollectorFree procedure which I have described in previous post?

ASiwon · 2015-03-16 07:34:24

Hello,

please let me know if you would like to make changes in mORMot code or not. This is very important to me and I have very little time to decide what I should to do. If you don't want to make changes then Ok - I will change my copy of sources. But I just need to know: should I wait for change or not.

ab · 2015-03-16 08:38:08

Since we do not use packages here (just for components gathering), we are not package expert...

In the meanwhile, I added the ability to call GarbageCollectorFree manually.
See http://synopse.info/fossil/info/aeca67282809

ASiwon · 2015-03-16 10:46:44

Thank you very much. This helped me a lot!

mORMot Open Source

#1 2015-03-12 13:01:15

Memory managment in mORMot question

#2 2015-03-12 13:15:37

Re: Memory managment in mORMot question

#3 2015-03-13 09:52:56

Re: Memory managment in mORMot question

#4 2015-03-13 10:10:23

Re: Memory managment in mORMot question

#5 2015-03-13 13:10:51

Re: Memory managment in mORMot question

#6 2015-03-14 10:00:10

Re: Memory managment in mORMot question

#7 2015-03-16 07:34:24

Re: Memory managment in mORMot question

#8 2015-03-16 08:38:08

Re: Memory managment in mORMot question

#9 2015-03-16 10:46:44

Re: Memory managment in mORMot question

Board footer