mORMot Open Source

foncci

Member

Registered: 2013-11-15

Posts: 53

Memory Leak if exception is raised

ab,

There is huge memory leak if an exception is raised inside an interface method service. This is simple, just add an exception at the end of a method service and call it 1000 times and you will see what happens in the task manager. To make it simpler, if there is an exception inside the method, the server never release the open connection and of course, in 2 hours the server runs out of memory. I have to remarks that this happens only with sicClientDriven, if you use sicPerThread it doesn't happend because it only creates one connection in the server.
I discover it because I'm getting the exception if the fake interface call and after 2 hours all services are using the whole memory. If you want I can send you a video with this.
I'm using mORMot to connect more than 10 branches and replicate data between them. It is a lot of data and it is replicated in real-time. All my method services are enclose in try-except so those exceptions are not mine. The is a bug deep inside mORMot. If you want I would like to do paid session with you to fix this. I'm in a lot of troubles because I replaced Rem Objects with mORMot and now we are collapsed.

function TServiceIntegrador.Consulta(aComando : RawJSON; var aNav : RawJSON) : RawJSON;
var NavRec : TConsultaNav;
begin
  try
    if TServerIntegrador.LogVerbose=1 then
      TLogProc.WriteLogMemo(format('Consulta(In) = Comando=%s', [aComando]));

    NavRec.FromJSON(aNav);
    Connect;
    Result := TdmDataHandle.ExportData(DB, StringToUTF8(aComando), NavRec);
    aNav := NavRec.ToJSON;

    if TServerIntegrador.LogVerbose=1 then
      TLogProc.WriteLogMemo(format('Consulta(Out) = %s', [Result]));
  except
    on e:exception do TLogProc.WriteLogMemo('TServiceIntegrador.Consulta->'+e.Message);
  end;

  raise Exception.Create('Error raro desde el server');   <<<<<<<<<<<<<<<<<<<<<<<<<<< My exception
end;

The real problem is the random iService.xxxx fake interface exception that is generated inside mORMot code and with this memory leaks is leaking all the memory.

Last edited by foncci (2014-03-30 03:53:20)

foncci

Member

Registered: 2013-11-15

Posts: 53

Re: Memory Leak if exception is raised

This is the log I get for the client exception TThreadSendMessages.Execute Invalid fake IServiceIntegrador.Consulta interface call: :
and nothing more. Digging in mORMot code I see that it should return {Error:500 xxx} too but I just receive white spaces instead of the error code, so I think that is a bug too.
I have compiled it with XE2 and XE5 and we get the same exception. This time I have made the server sicPerThread.
By the way, this was made using useHttpSocket because I'm trying everyone option to see if the error goes away. The exception is similar if you are using useHTTPAPI.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Server >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>.
D:\MAGNUS\TestIntegrador.exe 1.0.0.0 (2014-03-29 22:37:56)
Host=SER_FLP User=SOFTEL CPU=4*9-6-7685 OS=9.2=6.0.6002 Wow64=1 Freq=14318180
Environment variables=ALLUSERSPROFILE=C:\ProgramData	APPDATA=C:\Users\SOFTEL\AppData\Roaming	CLIENTNAME=SERVIDOR	CommonProgramFiles=C:\Program Files (x86)\Common Files	CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files	CommonProgramW6432=C:\Program Files\Common Files	COMPUTERNAME=SER_FLP	ComSpec=C:\Windows\system32\cmd.exe	DFSTRACINGON=FALSE	FP_NO_HOST_CHECK=NO	HOMEDRIVE=C:	HOMEPATH=\Users\SOFTEL	LOCALAPPDATA=C:\Users\SOFTEL\AppData\Local	LOGONSERVER=\\SER_FLP	NUMBER_OF_PROCESSORS=4	OS=Windows_NT	Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\Windows System Resource Manager\bin;;C:\Windows\idmu\common;C:\Program Files\Firebird\Firebird_2_5\bin	PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC	PROCESSOR_ARCHITECTURE=x86	PROCESSOR_ARCHITEW6432=AMD64	PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 30 Stepping 5, GenuineIntel	PROCESSOR_LEVEL=6	PROCESSOR_REVISION=1e05	ProgramData=C:\ProgramData	ProgramFiles=C:\Program Files (x86)	ProgramFiles(x86)=C:\Program Files (x86)	ProgramW6432=C:\Program Files	PSModulePath=C:\Windows\system32\WindowsPowerShell\v1.0\Modules\	PUBLIC=C:\Users\Public	SESSIONNAME=RDP-Tcp#1	SystemDrive=C:	SystemRoot=C:\Windows	TEMP=C:\Users\SOFTEL\AppData\Local\Temp\3	TMP=C:\Users\SOFTEL\AppData\Local\Temp\3	TRACE_FORMAT_SEARCH_PATH=\\winseqfe\release\Windows6.0\lh_sp2rtm\6002.18005.090410-1830\amd64fre\symbols.pri\TraceFormat	USERDOMAIN=SER_FLP	USERNAME=SOFTEL	USERPROFILE=C:\Users\SOFTEL	windir=C:\Windows
TSQLLog 1.18 2014-03-30T02:04:33

20140330 02043343 EXC   ECrtSocket ("SndLow -10054") at 003A98DF SynCrtSock.TCrtSocket.SndLow (2278)  stack trace 003AAF78 SynCrtSock.THttpServerResp.Execute (3103) 000CB484 System.Classes.Classes.ThreadProc (14561) 0000A03E System.ThreadWrapper (23409) 
20140330 02051847 EXC   ECrtSocket ("SndLow -10054") at 003A98DF SynCrtSock.TCrtSocket.SndLow (2278)  stack trace 003AAF78 SynCrtSock.THttpServerResp.Execute (3103) 000CB484 System.Classes.Classes.ThreadProc (14561) 0000A03E System.ThreadWrapper (23409) 

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Client >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>.
20140330 02020809 EXC   EOSError ("System Error.  Code: 12002.\r\n") at 0002D31F System.SysUtils.SysUtils.RaiseLastOSError (24225)  stack trace 0000A134 System.@NewAnsiString (23704) 0000AB60 System.@LStrCatN (26990) 003A2AEF mORMot.TServiceFactoryClient.InternalInvoke (35145) 00005A62 System.@GetMem (4321) 0000A134 System.@NewAnsiString (23704) 003A2978 mORMot.TServiceFactoryClient.Invoke (35115) 00398BBC mORMot.InternalProcess (32403) 
20140330 02020809 EXC   EInterfaceFactoryException ("Invalid fake IServiceIntegrador.Consulta interface call: : ") at 0039894A mORMot.RaiseError (32344)  stack trace 000CB484 System.Classes.Classes.ThreadProc (14561) 0000A03E System.ThreadWrapper (23409) 

Just in case you need it, this is the same exception using useHTTPAPI

20140330 05101203 EXC   EHttpApiServer ("HttpSendHttpResponse failed: Se ha intentado una operación en una conexión de red inexistente (1229)") at 003AD6B3 SynCrtSock.EHttpApiServer.RaiseOnError (4669)  stack trace 000CB484 System.Classes.Classes.ThreadProc (14561) 0000A03E System.ThreadWrapper (23409) 
20140330 05105508 EXC   EHttpApiServer ("HttpSendHttpResponse failed: Se ha intentado una operación en una conexión de red inexistente (1229)") at 003AD6B3 SynCrtSock.EHttpApiServer.RaiseOnError (4669)  stack trace 000CB484 System.Classes.Classes.ThreadProc (14561) 0000A03E System.ThreadWrapper (23409) 

Last edited by foncci (2014-03-30 11:06:18)

foncci

Member

Registered: 2013-11-15

Posts: 53

Re: Memory Leak if exception is raised

After a deep debug I think I know what is happening:
I'm doing 5 select queries that returns a around 20 000 rows in a method of an Interface based service. The process takes almost 20 seconds to complete. The data returned is converted to json and returned in the Result variable of the method.
First I change the function to a procedure with a var param of RawJSON type to see if this solve the problem. Still the same exception.
Second, I reduced the amount of data returned to the half but I was still getting the exception.
Third doing a cycle I start requesting all the information in packages of 10 rows only, now IT WORKS. The method executes in less than half a second.
I have a timeout like this:
     fClient.WinAPI.ReceiveTimeout := 120000; // 4 Min
     fClient.KeepAliveMS := 120000;
so I have 4 minutes timeout and the whole process never took more than 20 seconds. Is Timeout working?

Conclusion:
The client is not honoring the TimeOut that is set in the connection or according to the exception log generated and posted before this there is a bug in the string handling routines when you send huge strings. So I'm asking now because I think I read that strings are "infinite" in size. Is there any size limit in RawJSON or RawUTF8 types?
Third and my most concerning question, I dig inside mORMot code and I found the method that deals with timeout in the server. In the debugger it works fine but I debug in my laptop locally and everything runs faster locally. Is there any chance that  the Client closes the connection because the thread that is suppose to response to its keep alive packets is working doing some queries? If so, how can I do these queries without blocking the thread and why is that thread blocked because it should be a thread dedicated to run the method.
The memory leak is obvious now, the connection is closed by the client but the server is still processing those queries, the client tries again and again a new connection with more queries and in half an hour the service collapses.

Any help will be appreciated. I have expend days with this issue.

Regards,

Al

Last edited by foncci (2014-03-30 11:25:24)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: Memory Leak if exception is raised

AFAIK if the client fails with a time out, it closed its connection.
Then on the server, when the process is finished, it fails to send back the result over the broken connection.

There is no size limit in RawJSON/RawUTF8 types, but the free memory.
All those content RawJSON is recycled when the server finishes to process the query, even in case of broken connection.
It should not leak any memory.

How is your service created?
How many threads did you define in your thread pool?
You should define useHTTPAPI, and not useHttpSocket, on a production server.

In all cases, 20 seconds for 20 000 rows of data is very slow.
For instance, with our integrated SQLite3 engine, it takes around 60 ms to get 20 000 rows of data.
Why is it so slow?
IMHO a service should never take 20 seconds to process any request. There is clearly something wrong in your DB or SQL statement (missing index?).

I suspect you should use paging, i.e. never return 20 000 rows of data, but just some part of it (100 rows).
Do not use a "naive" paging with LIMIT / OFFSET, but a paging over an indexed value (e.g. the ID, or the Name, with an ORDER BY clause), then remember the latest returned last value, then use a where clause > last value.
See http://stackoverflow.com/questions/1446 … of-records

foncci

Member

Registered: 2013-11-15

Posts: 53

Re: Memory Leak if exception is raised

I know 20s is slow but I do some data processing inside it and connection latency is an issue too. The method process Invoices, Invoices Details, Payments, Customers and Products. I mean 20 000 invoices but when you sum all data it is more than 200 000 records. The time is pretty good and the indexes are well made. The result json is big, it is more than 5mb, now taking only 10 records is the way I found to make it work. But in any case why the client closes the connection? This is the real question here. If the timeout is set to 4 minutes and the processing time is only 20s why the connection is close? I think we have a bug in the KeepAlive or ReceiveTimeOut. If the connection last for more than a minute all my problems are solved.
About leaking memory I understand what you said, it takes time to release all those memory but there is always a new client connection so it will reserve more memory than it release and that why the service collapses.
I have always used useHTTPAPI, I try useHttpSocket just for testing to see if with it the connection is not closed but it is. I have been with this problem for a couple of weeks now so I'm trying every details to find the solution.

Service creation:

    CompressShaAesSetKey(cMyShaKey);
    aModel := TSQLModel.Create([], cServiceName);
    aServer := TSQLRestServerFullMemory.Create(aModel, '', false, true);
    aServer.ServiceRegister(TServiceIntegrador, [TypeInfo(IServiceIntegrador)], cServiceImplementation).AllowAll; // .AllowAllByName(['User']);
    aHTTPServer := TSQLHttpServer.Create(aPort, [aServer],'+',useHttpApiRegisteringURI, 32, secSynShaAes);

Client creation

  CompressShaAesSetKey(cMyShaKey);
  fModel := TSQLModel.Create([], cServiceName);
  fClient := TSQLHttpClient.Create(aServer, aPort, fModel);
  fClient.Compression := [hcSynLZ, hcDeflate, hcSynShaAes];
  if not fClient.ServerTimeStampSynchronize
    then raise ESynConnError.Create('Remote service not available on server');

  if (not fClient.SetUser(aUser, aPass))  or
     (not fClient.ServiceRegister([TypeInfo(IServiceIntegrador)], cServiceImplementation))
    then raise Exception.Create('Remote service not available on server. 1');

  if DebugHook<>0 then
   begin // 100 min Solo por debugging process
     fClient.WinAPI.ReceiveTimeout := 6000000;
   end
  else
   begin
     fClient.WinAPI.ReceiveTimeout := 120000; // 4 Min
     fClient.KeepAliveMS := 120000;
   end;
   // TWinHttpAPI.Create tiene los TIMEOUTS

  fService := fClient.Service<IServiceIntegrador>;

Contract

  IServiceIntegrador = interface(IInvokable)
    ['{DAC7E463-BBEC-4FC2-A7B2-B91E338D667C}']
    Procedure UpdSQL(aTipo : integer; aComando, aData : RawJSON; out aError : RawJson);
    Procedure UpdSQLFromSlave(aComando, aData : RawJSON);
    Procedure Consulta(aComando : RawJSON; out aData : RawJSON; var aNav : RawJSON);
    Procedure ExecAnySQLArr(aScript : RawJSON);
    Procedure ExecAnySQL(jComando : RawJSON);
    function  Echo(const s : TConsultaNav) : TConsultaNav;
    //function  EchoWithException(s : RawJSON) : RawJSON;
  end;

You never told me why client connection is closed in that short time. Usually in less than 10 seconds the client closes the connection and that is the root of all my problems. By the way the message that is generated is quite confusing. It never say "connection closed" or something like that, it is always "...fake interface call:" without any status code like with the rest of exceptions.
If you are still online we can do a TeamViewer session and I can show you more details.

Thanks for your time. I appreciate it.

Last edited by foncci (2014-03-30 13:19:12)

foncci

Member

Registered: 2013-11-15

Posts: 53

Re: Memory Leak if exception is raised

BTW I just read your suggestion from StackOverflow, I do the same:
select * from ventas where status<>21 {Moved to main office} order by id rows 10
Of course this is created dynamically and next I read all dependent tables in the same way. The order is made by a json string like this:
{TABLE:"VENTAS", WHERE:"STATUS<>21"}
As I said before the 20s+ is due to the Status and Dates fields updates and some other calculation for reports.
This is a module supported by mORMot that is capable of integrate tables among different databases without needing of triggers and it is working 24/7.
Thanks again for mORMot and any news about the timeout will be appreciated.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: Memory Leak if exception is raised

About the client timeouts, it is set in TWinHTTP.InternalRequest() as defined by http://msdn.microsoft.com/en-us/library … p/aa384116

Try to change the default value defined by HTTP_DEFAULT_RECEIVETIMEOUT = 30000 in SynCrtSock.pas.

Perhaps we have an issue in TWinHTTP.InternalRequest().
Setting the timeouts after the connection is open is perhaps not taken in account by the API.

You can also try to use for the client the TSQLHttpClientWinSock class instead of the default TSQLHttpClient = TSQLHttpClientWinHTTP.
I suspect that TSQLHttpClientWinSock timeouts may work better, since they are not at all implemented the same as for WinHTTP.

foncci

Member

Registered: 2013-11-15

Posts: 53

Re: Memory Leak if exception is raised

I already debugged TWinHTTP.InternalRequest 2 days ago and it gets the ReceiveTimeOut I set, if it uses the value or not is another story. I assumed that it was OK but now I agree with you, We do have an issue with that method. I cannot go inside it because it is an external function but for sure it is not setting the timeout and it is a huge problem. Now, lets guess that we have an issue with TWinHTTP.InternalRequest(), I tried useHttpSocket with exactly the same results, then we have an issue with TWinINet.InternalRequest too. We have something else going wrong in here.

I changed HTTP_DEFAULT_RECEIVETIMEOUT = 240000; and now it works fine. So the issue is as you said before <"Setting the timeouts after the connection is open is perhaps not taken in account by the API.">
Maybe the best idea is to add a parameter to the constructor to control this and update the SAD with this behavior. It has cost me 4 days this week including the weekend and days last month when I was confused about the exception message. By the way, a more intuitive error message should be used. The idea is to save somebody's else time.

Will you be able to go deeper on it. I need to do more time consuming process in the server side and TimeOuts is a must for me. I have a solution now but something more elegant should be nice.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: Memory Leak if exception is raised

OK.

We have just added SendTimeout and ReceiveTimeout optional parameters (in ms) to TWinHttpAPI constructors (instead of published properties which were not taken in account by the API) and to TSQLHttpClientWinHTTP / TSQLHttpClientWinINet constructors.
See http://synopse.info/fossil/info/51f977e95f

foncci

Member

Registered: 2013-11-15

Posts: 53

Re: Memory Leak if exception is raised

Thanks, those are great news. I will try it tonight.