mORMot Open Source

There is a memory leak caused When WrapperMethod is called, upon visiting the wrapper page to generate mORMotClient code.

The memory leak is specifically caused by the method function TWrapperContext.ContextFromMethods(int: TInterfaceFactory): variant;
Where "Result" can already be initialized by a previous call of this very method, resulting into calling init again on the same TDocVariantData, which results into AnsiString memory leaks.

I don't know with 100% certainty, if this is caused by our specific mustache files or other configurations, but I'm quite certain this is not caused by my code.

Now you could say, (As I initially thought), how come result is assigned before you even enter the method? Well it seems like the previous TDocVariantData result instance created by that function, seems to remain there, I suppose as "garbage".
Similar bihaviour can be seen in SynCommons.pas, in the function:
function _ObjFast(const NameValuePairs: array of const): variant;

There you can see that the first thing that is done is, is checking if result needs to be cleared prior to getting initialized again. So exact same concept I think, however in case of the method "ContextFromMethods" (Which is called in loops in quite some places), there is no such precaution available.
In ContextFromMethods, in order to initialize the resulting TDocVariantData, the method "TDocVariant.NewFast" is used, which on its turn calls InitFast, which doesn't do such a check such as in _ObjFast.

Calling VarClear(Result) before NewFast is called inside the method "ContextFromMethods", seems to solve the issue. But I don't know If it's dangerous to call it, meaning that I don't know with certainty, that initialized result data is somehow referenced somewhere else, and can be used.

So, it would be nice if someone else can look at this, confirm this, and possible also think for a (better) solution.

Kind Regards.

Hi,

I think there is a bug in mORMotSQLite3 TSQLRestServerDB.TransactionBegin.

In that method, whenever inherited TransactionBegin(aTable,SessionID) is called, and the result is true,
that means that TSQLRest.fTransactionActiveSession is set to the SessionID of that transaction.

So the transaction session is successfully allocated to this very transaction.
then there is the following code:

try
  DB.TransactionBegin; // will call DB.Lock
except
  on ESQLite3Exception do
      result := false;
end;

So if DB.TransactionBegin were to throw an exception, the exception is caught returning a False, which indicates that running the transaction failed.
However, returning a False is not enough! Because earlier inherited TransactionBegin(aTable,SessionID) succeeded, TSQLRest.fTransactionActiveSession is still set to SessionID, and is NOT reset.
Meaning that whenever this situation occurs (that DB.TransactionBegin throws an exception), fTransactionActiveSession Remains > 0, meaning that NO other transaction can be start, because
inherited TransactionBegin(aTable,SessionID) (which is the first thing that is called whenever you call TransactionBegin on mORMotSQLite3) will always return false as a result.
It ONLY returns true, when fTransactionActiveSession=0.

So EVERY other TransactionBegin call on mORMotSQLite3 will fail if even once the DB.TransactionBegin call in the highlighted code above fails.

Correct me if I'm wrong but that's my conclusion.
Because if we reach the call DB.TransactionBegin, it means that inherited TransactionBegin(aTable,SessionID), meaning that fTransactionActiveSession and fTransactionTable are both allocated to that very call.
meaning that if this transaction fails, inherited rollback needs to be called to release back fTransactionTable and especially fTransactionActiveSession, because that blocks any other attempt to do a transaction on that database.

This issue can be avoided by doing this:

try
  TransactionBegin
  ...
  Commit
finally
  Rollback
end;

So by basically moving the TransactionBegin INSIDE the Try finally block, that even if TransactionBegin were to fail, and even if there is no actual transaction active, it still calls Rollback, which on its turn Calls the TSQLRest.Rollback in mORMot.pas, which resets fTransactionActiveSession back to 0.
This is a weird solution and shouldn't be necessary, because whenever something goes wrong in DB.TransactionBegin, it still Calls Execute('ROLLBACK TRANSACTION;'); So Me doing an extra rollback in that scenario shouldn't be necessary. And procedure wise doesn't make sense to call Rollback if BeginTransaction failed. Unless it was meant to be used like that which i doubt.
Also this code is not really safe, If TransactionBegin were to fail for other reasons, unless you always use a different sessionID's for each other transaction call you do, especially when done on other threads.

Nonetheless I think this would be a possible solution that would make more sense:
So the exact same code as above, but changed as following: (Note: this is not necessarily the solution I suggest, is just to give an idea)

try
  DB.TransactionBegin; // will call DB.Lock
except
  on ESQLite3Exception do
  begin
    result := false;
    try
      // resetting fTransactionActive flag
      inherited RollBack;
    Except
      //
    end;
  end;
end;

Once again, because we know fTransactionActiveSession is allocated to that specific call because of the succeeding inherited call, I think it should be safe to do an inherited Rollback call that releases back fTransactionActiveSession.
Once again, correct me if I'm wrong, or feel free to suggest better solutions if possible.

Now, of course there is a reason why I suggest this fix, because I have an issue as following:
Imagine 2 consecutively running threads (2 clients) that both tries to do a transaction at the same time. Useful to note that I always pass GetCurrentThreadId as SessionID.
Now this is the exact scenario that happens:

> Thread 1: BeginTransaction Enter
> Thread 1: BeginTransaction Exit (success)
> Thread 1: Commit Enter
> Thread 1: Execute('COMMIT TRANSACTION') (NOTE: It might be possible that this line takes longer, or is called after Thread 2 has started, but the important part is that it has proceeded enough so that at the point when Thread 2 calls Execute('ROLLBACK TRANSACTION;'), it throws an exception)
> Thread 1: fTransactionActiveSession := 0
> Thread 2: BeginTransaction Enter (Initial check succeeds, because inherited BeginTransaction returns true because fTransactionActiveSession = 0) (IMPORTANT NOTE!!: At this point Thread 1 has NOT set fTransactionActive back to False yet in SynSQLite3 in TSQLDataBase)
> Thread 2: TSQLDataBase.TransactionBegin ENTER
> Thread 2: TSQLDataBase.TransactionBegin Throws Exception: Because fTransactionActive is still true from Thread 1, so instead this code is called by Thread 2: Execute('ROLLBACK TRANSACTION;'); which throws an exception because there is no Running transaction as Thread 1 already called Execute('COMMIT TRANSACTION')
                  the exception: ESQLite3Exception {"ErrorCode":1,"SQLite3ErrorCode":2,"Message":"Error SQLITE_ERROR (1) using 3.8.7.4 - 'cannot rollback - no transaction is active' extended_errcode=1"}
> Thread 2: Sets fTransactionActive to False in TSQLDataBase.TransactionBegin
> Thread 2: BeginTransaction Exit (while fTransactionActiveSession is still set to Thread 2)
> Thread 1: Commit Exit. (doesn't matter whether or not Thread 2 or 1 exits first)

So as you can see in that example, fTransactionActive (in SynSQLite3 in TSQLDataBase) is set to false too late after fTransactionActiveSession (in mORMot in TSQLRest) is set to 0 by Thread 1, causing TransactionBegin of Thread 2 to proceed further than it should be, causing it to throw an exception and leaving the transaction locked because it doesn't reset fTransactionActiveSession after an exception was thrown.
fTransactionActive (in SynSQLite3 in TSQLDataBase) is in this case the culprit, but that doesn't mean that TransactionBegin should be less safe, because other code is in the wrong. Nevertheless also it does seem to me like fTransactionActive might not be thread safe, and am not sure if that might be related to this issue.

Hope my explanation makes sense and isn't too complicated, can add more details if necessary. Or explain further.

NOTE: The exact order of the calls between the two threads might not be 100% correct, but it's more or less in that order, hopefully making my issue clear.

Hi,

I have a server that has multiple instances of TSQLRestServerDB running on it. For each user we have a separate databse that has his own data and do his own stuff.
We do backups on these servers once every night using the BackupBackground call. Making a local copies of the database files.

This works great, everything is fine.

However, lately the users increased, reaching approximately 200 and some of them are intensive users who have lots of data, which have databases up to 200MB. Problem is, because I created a 32bit server application (due to certain libraries I use), I am limited to roughly 4GB of RAM.
Normally I have enough resources to handle all of this on the system this server is running, but the BackupBackground significantly increases the memory usage and is not released. And the amount of memory used up depends on the actual size of the database. It's not perfectly linear, but there is a clearly visible correlation.

If this was only for the moment when the backup was created, no problem, but because this is done for roughly 200 databases. It increases the memory usage A LOT. And can only solve this by stopping the servers. I initially thought this had to do with the cache of the database, but that wasn't the case. That was limited to the default 10MB , and clearing it didn't have any effect. Using different parameters on BackupBackground didn't change anything either, read through the comments for that method and similar methods in the code and couldn't find much.

So my question is, what is using this memory? Why is this persistent? And how can I avoid this?

Thanks upfront.

Hi,

I have developed an agenda using Delphi with a server and client, where I use ORM objects (TSQLRecords) as each appointment.
There is an automatic refresh system that refreshes the agenda every minute. But that is a full refresh, where the complete date range is re-retrieved that is currently visible on the client side.
That is rather overkill I want to refresh only if there are any changes done on the specific table where these appointment records are stored.

The only thing that I was able to find that came close to it was the audit system, which isn't very practical for this application, very resource heavy and is overkill. I only have to see IF ANY changes took place on said records table since Time X.
So is there any built in system in the ORM to check/track this easily through client side so that it knows it has to do a refresh / retrieve the changes on that table?
Or if not, how can I best implement this?

Thanks upfront