Variant in SQLRecord MemoryLeak?

Shadownildo · 2019-06-25 12:41:54

Hi There,

We are running some perfomace tests in the ours servers, With mongoDB and SQLITE, and we find out that , if we use a type Variant,in the SQLRecord for a Published Field, there is a Gigantic Memory Leak, that we can't get rid of, it's not the Object , and i don't know where it can be

procedure TSaleService.Test(SaleInfo: TJSON; out result: Boolean);
var
  aSale: TSQLRecordSale;
begin
  try
    result := false;
    aSale := TSQLRecordSale.Create(fServer, 0);
    aSale.FillFrom(SaleInfo);
    result := true;
    fServer.Commit(CONST_AUTHENTICATION_NOT_USED, false);
  finally
    FreeMemAndNil(aSale);
  end;
end;

with this Implemantation that Works, but have a Memory Leak, not in the Object, but in a Unknow Object (it's what FastMM4 tell us), but like i said, running some tests with Variants and RawUTF8 , When we use a Variant Field, and run the perfomace test, that is like 1000+ registers , the Memory of the Server go to 1.700.000+ and don't go away
in those tests, we make 10.000 "sales", with 100 "Itens" each sale in the aggregate, is sorta 1.000.000 Registers.

Image of a Failed Test

So, we try to use RawUTF8 to "Avoid" this Memory Leak, but even so, with RawUTF8 stills a little Leak that don't go away.

Image of a Successful Test

Memory Leak Infos ( We not use 10.000 sales for this image, because take to Long to FastMM4 give me the report about the leaks, so those Reports are based in a 100 "Sales" With One Item In the Aggregate )

As you can se with the image Above, With Variants or RawUTF8 Still have a Leak, but the difference between the Leak is Gigantic, but, working with Variant is a good thing, because in Mongo (In Other types of Database(SQLITE,Postgres,etc) is all the same), i can work with the data in a better way than String(RawUTF8) but if i need to escape this Leak... i prefer to work with Strings, BUT, I Can't have this Leak in the Server, RawUTF8 Leak Less, but in a long time Online, will Give me Out Of Memory and this can't happen.... and i'm really curious, Why there is this Memory Leak, and why i can't get rid of , even if i Clear the field with Unassigned, or Cleaning the Object with ClearObject() or Freeing him, i make a Small Project to work only with Variant's, and Creating nested Objects, and i don't have a Leak, but when come's to SQLRecord, this happen, anyone could help me ?

Sorry about the Bad English
Best Regards
Shadownildo

Last edited by Shadownildo (2019-06-25 12:42:49)

macfly · 2019-06-25 14:10:18

This leak should not be related to the variants.

It must be something that has been created but not destroyed, and it occurs as a consequence.
There are several objects that appear in the report.

What is this TJSON type?
And FreeMemAndNill is your own method? JCL maybe? What he does?

Shadownildo · 2019-06-25 14:17:58

macfly wrote:

This leak should not be related to the variants.
It must be something that has been created but not destroyed, and it occurs as a consequence.
There are several objects that appear in the report.
What is this TJSON type?
And FreeMemAndNill is your own method? JCL maybe? What he does?

Thanks for the Answer!

This amount of leaks only occurs when i change the fields RawUTF8 to Variant or Variant to RawUTF8,

The TJSON Type is a RawUTF8

   TJSON = type RawUTF8;

and Yes FreeMemAndNill is a Own Method,but already tested with Free and FreeAndNil, Same results.

procedure FreeMemAndNil(var ptr; size: Integer = -1);
var
  p: Pointer;
begin
  p := Pointer(ptr);
  if p <> nil then
  begin
    if size > -1 then
      FreeMem(p, size)
    else
      FreeMem(p);

    Pointer(ptr) := nil;
  end;
end;

Best Regards,
Shadownildo

Shadownildo · 2019-06-25 14:24:25

macfly wrote:

It must be something that has been created but not destroyed, and it occurs as a consequence.
There are several objects that appear in the report.

Sorry , i forgot to Answer that.

The others Objects appears because i'm testing the perfomace in a Local TSQLRestServerFullMemory Server, so i'm not destroying the Connection as should be, because, i Can't destroy a server Connection after using only one service, so FastMM4 understand as Leak

macfly · 2019-06-25 14:55:47

What's inside the TesteInsercao method?

And (not related to leak), You are retrieving an object that probably does not exist, and then overriding the data. This is one more step that has no effect.

 aSale := TSQLRecordSale.Create(fServer, 0); <-- this issues a Retrieve from DB
 aSale.FillFrom(SaleInfo);

Would be better:

 aSale := TSQLRecordSale.Create;
 aSale.FillFrom(SaleInfo);

or simply:

 aSale := TSQLRecordSale.CreateFrom(SaleInfo);

And fServer.Commit... Should be in this method?

Shadownildo · 2019-06-25 19:18:13

macfly wrote:

What's inside the TesteInsercao method?
And (not related to leak), You are retrieving an object that probably does not exist, and then overriding the data. This is one more step that has no effect.
 aSale := TSQLRecordSale.Create(fServer, 0); <-- this issues a Retrieve from DB
 aSale.FillFrom(SaleInfo);
Would be better:
 aSale := TSQLRecordSale.Create;
 aSale.FillFrom(SaleInfo);
or simply:
 aSale := TSQLRecordSale.CreateFrom(SaleInfo);
And fServer.Commit... Should be in this method?

Thanks for the Tip!, already make the changes you suggest, with CreateFrom()!

and about the Teste Insercao Method , it has basicaly a Loop that Create a TSale Object, populate Him, and the Nested Objects Inside, and Call The insertion Service

 procedure TTestPerformanceVenda.TesteInsercao;
var
  sale: TVenda;
  RestServer: TSQLRestServerFullMemory;
  i: integer;
  RestClient: TSQLRestClientURI;
  cmd: IDomVendaCommand;
  test : Boolean;
begin
  RestServer := TSQLRestServerFullMemory.CreateWithOwnModel([TSQLRecordSale]);
  RestServer.ServiceDefine(TSaleService, [ISaleService], SicClientDriven);
  RestClient := TSQLRestClientURIDll.Create(TSQLModel.Create(RestServer.Model), @URIRequest);
  try
    RestClient.Model.Owner := RestClient;
    RestClient.ServiceDefineClientDriven(ISaleService, FSaleService);
    try
      for i := 0 to 4 do
      begin
        sale := TSale.Create;
        populateSale(i, sale); // a var Parameter
        try
           FSaleService.Teste(ObjectToJSON(sale),test);
           Check(test = true);
        finally
        ClearObject(Sale,true);
        FreeAndNil(Sale);
        end;
      end;
    finally 
      FVendaService:= nil;
      RestClient := nil;
      RestClient.Free;
    end;
  finally  
    //RestServer.Free;
  end;
end;

Shadownildo · 2019-06-25 19:24:42

macfly wrote:

And fServer.Commit... Should be in this method?

And again i forgot to answer one , haha,
Why Shouldn't be ?
it's inside a TInjectableObjectRest Class Object, and have access to the info i need, or i mistaken ?

macfly · 2019-06-25 20:01:27

I think you're not persisting the data.
And as you are using Result: = True, the result will always be true even if it is not persisting.
In other words, this test does not test the insertion but tests only a possible exception inside this method.

See:

procedure TSaleService.Test(SaleInfo: TJSON; out result: Boolean);
var
  aSale: TSQLRecordSale;
begin
  try
    result := false; <-- Not needed
    aSale := TSQLRecordSale.Create(fServer, 0);
    aSale.FillFrom(SaleInfo);
    result := true; <-- do not force the result
    fServer.Commit(CONST_AUTHENTICATION_NOT_USED, false); <-- What does it do?
  finally
    FreeMemAndNil(aSale);
  end;
end;

Suggestion:

procedure TSaleService.Test(SaleInfo: TJSON; out result: Boolean);
var
  aSale: TSQLRecordSale;
begin
  try
    aSale := TSQLRecordSale.CreateFrom(SaleInfo);
    Result := fServer.Add(aSale, True) > 0; //<<- Add method persists de object and return TID of record added. 
  finally
    aSale.Free;
  end;
end;

And I think that one of your leak is here.

procedure TTestPerformanceVenda.TesteInsercao;
var
..
begin
  try
    try
    ...
    finally 
      FVendaService:= nil;
      RestClient := nil; <-- You are setting to nil then...
      RestClient.Free; <-- ...this take effect?
    end;
  finally  
    //RestServer.Free;
  end;
end;

Other point, is that you're playing with the Model owner. This is dangerous...
I Suggest:

procedure TTestPerformanceVenda.TesteInsercao;
var
  AModel : TSQLModel; //<-- a var to hold the models
begin
  AModel := TSQLModel.Create([TSQLRecordSale],...); // <-- Create the shared models
  RestServer := TSQLRestServerFullMemory.Create(AModel); //<<-- uses same models
  ...
  RestClient := TSQLRestClientURIDll.Create(AModel), @URIRequest); //<<-- uses same models
  try
    RestClient.Model.Owner := RestClient; //<-- remove this
    try
      ....
    finally 
    end;
  finally  
     RestClient.Free; //Free all
     RestServer.Free; //Free all
     AModel.Free; //Free all
  end;
end;

Shadownildo · 2019-06-25 20:22:49

It's really my bad, when i share the code, i change the Language for better Understandiment, and i erase the fServer.Add() by mistaken,Sorry, but i like more the way you implemented, thanks for the Tips again!

    fServer.Add(aSale, true);
    result:= true;
    fServer.Commit(CONST_AUTHENTICATION_NOT_USED, false);

I know the Danger with Model, in the main server all the preparations are like you suggest.

macfly · 2019-06-25 20:56:50

You do not need this in Teste method:
fServer.Commit(CONST_AUTHENTICATION_NOT_USED, false);

I not see in your code where the transaction was started.

A commit must be used in conjunction with TransactionBegin.

It would make sense to use this in your TesteInsercao method.

 RestServer.TransactionBegin(...);
  for I := 0 to 4 do
  begin
      ....
      FSaleService.Teste(ObjectToJSON(sale),test);
      ....
  end;
  RestServer.Commit(...);
 // Or a RestServer.Rollback in case of errors
 // See  TransactionBegin documentation

Last edited by macfly (2019-06-25 21:02:06)

Shadownildo · 2019-06-26 12:12:54

macfly wrote:

You do not need this in Teste method:
fServer.Commit(CONST_AUTHENTICATION_NOT_USED, false);
I not see in your code where the transaction was started.
A commit must be used in conjunction with TransactionBegin.
It would make sense to use this in your TesteInsercao method.
 RestServer.TransactionBegin(...);
  for I := 0 to 4 do
  begin
      ....
      FSaleService.Teste(ObjectToJSON(sale),test);
      ....
  end;
  RestServer.Commit(...);
 // Or a RestServer.Rollback in case of errors
 // See  TransactionBegin documentation

Thanks for the Help!

Already make these Changes!, And the Leaks Decreased significantly, but the main Leak still here , this test was made with 10 Sales with 10 itens each and The type was Variant.

Main Leak

Shadownildo · 2019-06-26 12:49:15

And just to clarefy, All those others leaks, are because i'm not Destroying the Rest server, Because in a real scenario i will not destroy the Server after using a Service, only when i close the server by myself, and if i make this in the service Side

    result := false;
    if fServer.TransactionBegin(TSQLRecordSale, CONST_AUTHENTICATION_NOT_USED) then
    begin
      try
      ....
      except
      ....
      end;
    end;
  finally
    ClearObject(aSale);
    FreeMemAndNil(aSale);
    aSale.Free;
  end;
end;

And in the Client Side i Destroy the server like you suggest

      FVendaService := nil;
      RestClient.Free;
      RestServer.Free;
      Amodel.Free;

I will Not have a Leak , But, if i Comment the RestServer.Free, i will have this.

Leak

And like i said, In a real scenario, i Can't have this Those leaks in Red after using the service..

macfly · 2019-06-26 14:23:27

On the client side it is not necessary to create a server instance. Only the client itself needs to be instantiated.

There is probably something that was created but not destroyed, these leaks are consequence.

My suggestion would be to change the tests to release the ResteServer, because the purpose of the tests is also to detect the leaks and in this case you are generating one deliberately.

Perhaps you may have to change the design of your tests to do this.

Shadownildo · 2019-06-26 14:30:58

Quick Update, make some adaptions in the perfomace project and now i'm using the Real server to make the tests and, i think i get rid of the leaks, Using ClearObject() in the SQLRecord, make a test with 12.000.000 of registers , Simultaneously, and All Okay!, and the Memory in the server don't go more than 100.000kb/s for me this is a big WIN, i think this memory is a Buffer, i don't know if there is a way to ger rid of this, Thanks for the help macfly!
but taking advantage of the conversation, With the suggestion you make, i Can't make Simultaneously insertions , because of the

if  FServer.TransactionBegin(TSQLRecordSale) then
      begin
        .....
      end;

I search for some answers in the documentation, and is recommended to use BATCH, or use TransactionBegin on Client-side, but i don't really like to do this kind of Transaction Control in the Client, you got any suggestions how to do in the Server-Side?

  try
    .....
      try
      .......
      except
        ......
  finally
    ClearObject(aSale);
    aSale.Free;

12.000.000 Registers

Shadownildo · 2019-06-26 20:27:00

Another Quick Update, but the Question above still remains...

More changes, and now the server don't go more than 30.000 Kb/s , don't matter How many Registers i'm sending, last test we put 18.000.000, yes 18 Millions Registers simultaneously, This is REALLY good, BUT, for this, we are using SicClientDriven, and our Worries are about the CrossPlataform Issue, like , if using JavaScript to make a Request in our Server, The "Destruction" of this Service, have to be in the Client Side , Because of the SicClientDriven, but there is no way to do this there, So i'm wondering, The server verify this "Destruction" by Verifying the Bridge Between Client-Server, and when the request of the JavaScript Application "Ends" , it understand that have to destroy the instance?

Shadownildo · 2019-06-28 13:55:32

Another Update !

Already Solved all the Question's Above ! And just for curiosity because i think is awesome , Now Using Parallel Process and Threads, with Batch, We are Inserting 10.000.000 Millions Registers, in 6 SECONDS , with a 30.000 Kb/s Server in Memory, This Speed is just WOW and with this memory is Just WOWW , Thanks For the Help !

ab · 2019-06-28 19:10:29

Happy news!

macfly · 2019-06-28 20:29:42

Double WOW is always good :cool

mORMot Open Source

#1 2019-06-25 12:41:54

Variant in SQLRecord MemoryLeak?

#2 2019-06-25 14:10:18

Re: Variant in SQLRecord MemoryLeak?

#3 2019-06-25 14:17:58

Re: Variant in SQLRecord MemoryLeak?

#4 2019-06-25 14:24:25

Re: Variant in SQLRecord MemoryLeak?

#5 2019-06-25 14:55:47

Re: Variant in SQLRecord MemoryLeak?

#6 2019-06-25 19:18:13

Re: Variant in SQLRecord MemoryLeak?

#7 2019-06-25 19:24:42

Re: Variant in SQLRecord MemoryLeak?

#8 2019-06-25 20:01:27

Re: Variant in SQLRecord MemoryLeak?

#9 2019-06-25 20:22:49

Re: Variant in SQLRecord MemoryLeak?

#10 2019-06-25 20:56:50

Re: Variant in SQLRecord MemoryLeak?

#11 2019-06-26 12:12:54

Re: Variant in SQLRecord MemoryLeak?

#12 2019-06-26 12:49:15

Re: Variant in SQLRecord MemoryLeak?

#13 2019-06-26 14:23:27

Re: Variant in SQLRecord MemoryLeak?

#14 2019-06-26 14:30:58

Re: Variant in SQLRecord MemoryLeak?

#15 2019-06-26 20:27:00

Re: Variant in SQLRecord MemoryLeak?

#16 2019-06-28 13:55:32

Re: Variant in SQLRecord MemoryLeak?

#17 2019-06-28 19:10:29

Re: Variant in SQLRecord MemoryLeak?

#18 2019-06-28 20:29:42

Re: Variant in SQLRecord MemoryLeak?

Board footer