#51 Re: mORMot 1 » Newbie, stuck at the 1st hurdle » 2016-10-05 11:52:15

oz

What's the return value of "Server.Add(Customer, true);"? Not "0"?

#52 Re: mORMot 1 » Master/Slave replications - tips :) » 2016-10-05 11:26:35

oz

After dealing with replication/synchronization scenarios for 20+ years now, there is one thing i could say and advice to anyone from my experiences:
This kind of functionality is very very domain specific and depends hell of a lot on your use case. If you have to support such offline read/write services in your application, then don't start implementing this functionality in your DB or ORM layer. Do it in your business- or application layers. Don't think in tables, think in objects.
Replication at DB/ORM layer is painfull based on my experiences. Of course you can do that, but sooner or later you will face nasty little problems which will require you to do dirty code hacking, polluting any domain model.
Furthermore mORMot is not a very good candidate for such offline replication services at DB/ORM layers because of it's numerical ID design imho. But from my point of view this is not a limitation because:
1. It's a not a good idea to start building sync/replication services at DB/ORM layers.
2. mORMot is absolutely great about it's DDD capatibilities, and thats the rigth place to start building sync services at all.

I see the possibility to create some kind of generic object based sync/replication services for various scenarios with the help of mORMot's DDD toolbox, but thats a huge and difficult task to do.

@hnb:
I have some questions about your solution. Your model is 1 Master, N Slaves, right? If so, then how do you deal with ID conflicts? Do you use some kind of Master/Slave ID Converter, do you use fixed size ID ranges (e.g: Slave 1: 0-1m, Slave 2: 1m-2m,...), or do you use some kind of custom id handling based on guids or other non numerical field types?

#53 Re: mORMot 1 » Issue with TJSONSerializer.RegisterObjArrayForJSON +DEMO » 2016-10-05 10:15:55

oz

I've created a ticket for this issue http://synopse.info/fossil/tktview/954f … 2786231833 . Demo code is attached in ticket description and can be copy&pasted to an empty .dpr file.
Arnaud, did you have time to run the demo? It's quite a serious low level issue i guess...

#54 Re: mORMot 1 » Mem-leak issue with T*ObjArray out parameters in SOA methods + FIX » 2016-09-29 09:51:21

oz

Just to let you know: FindHashedForAdding seems to be ok.
Regarding my problem i was able to find the root of the problems. There's an issue with TJSONSerializer.RegisterObjArrayForJSON. See http://synopse.info/forum/viewtopic.php?id=3566 for more details.

#55 mORMot 1 » Issue with TJSONSerializer.RegisterObjArrayForJSON +DEMO » 2016-09-29 09:46:40

oz
Replies: 3

Hi Arnaud,

there is an issue with TJSONSerializer.RegisterObjArrayForJSON. It looks like if there is a limitation to RegisterObjArrayForJSON. When adding the 126th T*ObjArray<->T* pair, all T*ObjArray params start leaking memory on server side. My last post in http://synopse.info/forum/viewtopic.php?id=3559 describes what is happening then.

I've attached a demo program to reproduce the issue, you can download from: http://www.megafileupload.com/g1ez/MemLeakTest.zip

Compile and run the program, everything works as expected.
Enable conditional define {$DEFINE SHOWBUG} and .GetAll(TTestObjArray) leaks memory. 1000 instances of TTest are not destroyed, because TJSONSerrializer is broken.

It took me some time to find out what's happening here, i hope my test helps.

Kind regards,
oz.

#56 Re: mORMot 1 » Mem-leak issue with T*ObjArray out parameters in SOA methods + FIX » 2016-09-28 10:11:14

oz

Hi Arnaud,

i created a sample app to reproduce the issue, but it doesn't happen over there.

But the problem is another one: after updating to current mORMot sources all my T*ObjArrays leak memory because they aren't recognized anymore.

The strange thing is:
I have a dedicated testsuite for DTO testing functionality which does not depend on server infrastructure.
If I remove the "TJSONSerializer.RegisterObjArrayForJSON([TypeInfo(TMyObjArray),TMy]);" call in those tests, then no memory is leaked.
If the call to RegisterObjArrayForJSON is not removed, then out params do leak memory.

In mORMot.pas, "procedure TServiceMethodExecute.AfterExecute;" the function "fDynArrays(i).Wrapper.Clear;" is called, but in "TDynArray.InternalSetLength" the method "GetIsObjArray" returns false for those arrays if registered via RegisterObjArrayForJSON. This means that class destructors are not called -> memory leak.

All my tests are done using latest trunk version and Delphi 7.

Hmm, i really don't undestand that behaviour right now.

RegisterObjArrayForJSON call done: mem leaks because destructors are not called.
RegisterObjArrayForJSON call not done: no leaks.

Sadly, I can't leave those calls in production wink
Do you have any idea what is happening here?

#57 Re: mORMot 1 » SignUp on HTTP » 2016-09-27 16:08:11

oz

Maybe I got you wrong, but you said the IV is generated by the server, so the server already knows about the IV. Doesn't it? Why are you encrypting that thing at all? Is it about transport layer security?
Maybe you could explain in more detail...

#58 Re: mORMot 1 » Mem-leak issue with T*ObjArray out parameters in SOA methods + FIX » 2016-09-27 15:52:02

oz

Hi Arnaud,

thanks for the quick reply, but unfortunately your modifications do not help after a short test. SOA methods with T*ObjArray OUT parameters still leak memory. I could create a small test-app to reproduce this issue if required, but you should be able to reproduce by simply calling any SOA method with a T*ObjArray OUT parameter.

Kind regards,
oz.

#59 mORMot 1 » Mem-leak issue with T*ObjArray out parameters in SOA methods + FIX » 2016-09-27 11:18:54

oz
Replies: 5

Hi Arnaud,

there is an issue with Methods returning T*ObjArray objects with current source version.
I have several SOA Services with following public look-a-like methods:

  IMyInterface=interface(ICQRSService)
    [some guid here]
    ...
    function GetData(out aDataArray: TDataObjArray): TCQRSResult;
    ...
  end;

With current sources all of those out parameters leak memory. T*ObjArray's parameters are not destroyed in AfterExecute().

in mORMot.pas, line 52022 (in constructor TInterfaceFactory.Create) there is following code:

      case ValueType of
      smvRawUTF8..smvWideString:
        Include(ValueKindAsm,vIsString);
      smvDynArray:
        if ObjArraySerializers.Find(ArgTypeInfo)<>nil then
          Include(ValueKindAsm,vIsObjArray);

ObjArraySerializers.Find(ArgTypeInfo) returns nil here. Going deeper into SynCommons.pas:

function TObjectListPropertyHashed.IndexOf(aObject: TObject): integer;
begin
  if fCount>0 then
    if fHashed then begin
      if not fHashValid then
        IntHashValid;
      result := fHash.HashFind(fHash.fHashElement(aObject,fHash.fHasher),PtrInt(aObject));
      if result>=0 then
        exit; // if found
    end else
    for result := 0 to fCount-1 do
      if IntComp(fList[result],aObject)=0 then
        exit;
  result := -1;
end;

The list is hashed (fHashed=true), fHash.HashFind returns result < 0 here which means the hash lookup failed.
Having a look at TObjectListPropertyHashed.Add shows that the list is not rehashed after adding new items. But imho it should be.

function TObjectListPropertyHashed.Add(aObject: TObject; out wasAdded: boolean): integer;
begin
  wasAdded := false;
  if self<>nil then
    if fHashed then begin
      if not fHashValid then    // <- That's the problem, fHashValid is true but we need to rehash
        IntHashValid;
      result := fHash.FindHashedForAdding(aObject,wasAdded,
        fHash.fHashElement(aObject,fHash.fHasher));
      if wasAdded then
        fList[result] := aObject;
    end else begin
      for result := 0 to fCount-1 do
        if IntComp(fList[result],aObject)=0 then
          exit;
      wasAdded := true;
      result := fHash.Add(aObject);
      if fCount>=TOBJECTLISTHASHED_START_HASHING_COUNT then
        fHashed := true;
    end
  else
    result := -1;
end;

Changing the implementation as following makes the mem leaks disappear.

function TObjectListPropertyHashed.Add(aObject: TObject; out wasAdded: boolean): integer;
begin
  wasAdded := false;
  if self<>nil then
    if fHashed then begin
      IntHashValid;  // "fHashValid:=false;" instead of "IntHashValid;" works too because the next .IndexOf() call is forced to rehash the list then.
      result := fHash.FindHashedForAdding(aObject,wasAdded,
        fHash.fHashElement(aObject,fHash.fHasher));
      if wasAdded then
        fList[result] := aObject;
    end else begin
      for result := 0 to fCount-1 do
        if IntComp(fList[result],aObject)=0 then
          exit;
      wasAdded := true;
      result := fHash.Add(aObject);
      if fCount>=TOBJECTLISTHASHED_START_HASHING_COUNT then
        fHashed := true;
    end
  else
    result := -1;
end;

Could you have a look at the fix and add it to trunk if ok?
Kind regard,
oz.

#60 Re: mORMot 1 » Issue with TDynArrayHashed: failed Assertions/mem leaks » 2016-09-22 13:22:21

oz
mingda wrote:

what's your version , in 2915 TDynArrayHashed has a fix, see http://synopse.info/fossil/info/ddc1dc3 … bb6a4a827a

I'm using current 1.18.2983 (leaf [0505dd5792]) from today.

#61 mORMot 1 » Issue with TDynArrayHashed: failed Assertions/mem leaks » 2016-09-22 11:21:03

oz
Replies: 8

Hi!
There seems to be a problem with TDynArrayHashed.

Following situation:
- There's one shared TSQLDBZEOSConnectionProperties connection to a Firebird legacy DB.
- Several legacy purpose SOA Services use this connection for direct SQL r/w access at SynDB level, no ORM is used here.
- One concrete services retrieves some ISQLDbRows and loops through that data using rows.Step()
- inside each of those loops several other ISQLDbRows are used to retrieve data using the same TSQLDBZEOSConnectionProperties instance.

Everything works fine for some loops, then EAssertionFailed is raised in TDynArrayHashed.HashAdd:

procedure TDynArrayHashed.HashAdd(const Elem; aHashCode: Cardinal; var result: integer);
var n,cap: integer;
begin
  n := Count;
  SetCount(n+1); // reserve space for a void element in array
  cap := Capacity;
  if cap*2-cap shr 3>=fHashsCount then begin
    // fHashs[] is too small -> recreate
    ReHash;
    result := HashFind(aHashCode,Elem); // fHashs[] has changed -> recompute
    assert(result<0);
  end;
  with fHashs[-result-1] do begin // HashFind returned negative index in fHashs[]
    Hash := aHashCode;
    Index := n;
  end;
  result := n;
end;

"assert(result<0);" fails.

Callstack:

TDynArrayHashed.HashAdd((kein Wert),1313814477,0)
TDynArrayHashed.FindHashedForAdding((kein Wert),True,1313814477)
TDynArrayHashed.AddAndMakeUniqueName('EMAIL')
TSQLDBZEOSStatement.ExecutePrepared
TSQLDBConnectionProperties.Execute('SELECT cd.Email from ContactData CD where cd.ContactDataID IN (SELECT lcp.ContactDataID from LinkCustomerPerson lcp WHERE lcp.CustomerID=?)',(...),nil,False)
...

Compiling with assertions disabled results in obviously working code but leaves memory leaks for every further call.
FastMM4 report for the first leak:

--------------------------------2016/9/22 12:22:13--------------------------------
A memory block has been leaked. The size is: 2244

This block was allocated by thread 0x18B8, and the stack trace (return addresses) at the time was:
403101 [system.pas][System][@ReallocMem][2550]
40752C [system.pas][System][DynArraySetLength][16060]
40760E [system.pas][System][@DynArraySetLength][16108]
461AF2 [SynCommons.pas][SynCommons][TDynArrayHashed.ReHash][44383]
460E2B [SynCommons.pas][SynCommons][TDynArrayHashed.Scan][43871]
460F5A [SynCommons.pas][SynCommons][TDynArrayHashed.HashAdd][43905]
461073 [SynCommons.pas][SynCommons][TDynArrayHashed.FindHashedForAdding][43943]
4610D1 [SynCommons.pas][SynCommons][TDynArrayHashed.AddAndMakeUniqueName][43955]
AD761E [ZDbcResultSetMetadata.pas][ZDbcResultSetMetadata][TZAbstractResultSetMetadata.GetColumnType][474]
B9CF92 [SynDBZeos.pas][SynDBZeos][TSQLDBZEOSStatement.ExecutePrepared][1113]
AA7233 [SynDB.pas][SynDB][TSQLDBConnectionPropertiesThreadSafe.ThreadSafeConnection][6438]

The block is currently used for an object of class: Unknown

The allocation number is: 127937

Current memory dump of 256 bytes starting at pointer address 7EBA8590:
CC 37 4F 4E 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Ì  7  O  N  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .

As far as I can say right now everything seems to be ok with my code. Afair it is save to share one TSQLDBZEOSConnectionProperties instance. It might be hard to create an app to reproduce the issue, but maybe all that informations above could help to identify if there is an issue TDynArrayHashed.

Kind regards,
oz.

#62 Re: mORMot 1 » Is there some documentation on master detail relationships in mORMot? » 2016-09-21 20:12:10

oz

Imho DDD is the only way to go for building complex applications. By Following the DDD priciples using mORMot's great SOA features you can seperate storage classes from your DTO objects. You can use T*ObjArrays in your DTO's and persist those arrays to regular RawUTF8 fields in your TSQLRecord* classes using custom field assignment functions from DTO to TSQLRecord and vice versa.

#63 Re: mORMot 1 » Disabling/switching interface implementations at runtime » 2016-06-23 09:47:25

oz

So, to come to a conclusion:
AddInterface() will be left untouched for now, introducing some kind of GlobalInterfaceResolutionLock-like locking has to be discussed.

Arnaud, are you ok with adding following 3 methods to mORMot.pas for a start?

procedure TInterfaceResolverInjected.DeleteResolver(aResolver: TInterfaceResolver);
var
  ix: integer;
begin
  ix:=ObjArrayFind(fResolvers,aResolver);
  if ix>=0 then
    ObjArrayDelete(fResolvers,ix);
end;
procedure TSQLRestServer.ServiceUndefine(const aInterfaces: array of TGUID);
begin
  (ServiceContainer as TServiceContainerServer).DeleteInterface(TInterfaceFactory.GUID2TypeInfo(aInterfaces));
end;
procedure TServiceContainer.DeleteInterface(const aInterfaces: array of PTypeInfo);
var
  iInterfaces: integer;
  iMethods: integer;
  Idx: integer;
  MethodName: RawUTF8;
  InterfaceName: RawUTF8;
  tmpInterfaceName: RawUTF8;
begin
  for iInterfaces:=Low(aInterfaces) to High(aInterfaces) do begin
    tmpInterfaceName:=aInterfaces[iInterfaces]^.Name;
    if tmpInterfaceName[1] in ['I','i'] then
      Delete(tmpInterfaceName,1,1);
    idx:=fList.IndexOf(tmpInterfaceName);
    if idx>-1 then begin
      fList.Objects[idx].Free;
      fList.Delete(idx);
    end;
    for iMethods:=High(fListInterfaceMethod) downto Low(fListInterfaceMethod) do begin
      Split(fListInterfaceMethod[iMethods].InterfaceDotMethodName,'.',InterfaceName,MethodName);
      if InterfaceName=tmpInterfaceName then
        fListInterfaceMethods.Delete(iMethods);
    end;
  end;
end;

Any call to AddInterface()/DeleteInterface() methods has to be done in a thread-safe way, which the programmer is responsible for.

#64 Re: mORMot 1 » Disabling/switching interface implementations at runtime » 2016-06-23 08:47:34

oz
Leslie7 wrote:

It works if it is done the right way. smile

The List fList references is always entirely readonly.

for any change
1. OldList:= fList  // two references  to the same list.
2. OldList is duplicated to NewList // two identical lists
3. changes are applied to NewList
4. if  InterlockedCompareExchangePointer (@fList, @NewList , @OldList) <> @OldList then
    // we know that it has been changed since we started --> Start again with 1.
   else
   // we know that it has not been changed since we started and  InterlockedCompareExchangePointer has already replaced  fList  with NewList  --> SUCCESS

Ok, now I get what you mean. But imho it still does not help here. One problem is that not only fList is involved, fListInterfaceMethod is involved too. So we have to do similiar things for fListInterfaceMethod array --> 2 atomic operations in a non-atomic "parent" method. Having a look at the current implementation it should be no problem to operate on fListInterfaceMethod if another thread is operating on fList currently. Possible problems here are more of an academic kind and should only arise if one is trying to Add/Delete the same interface from several thread at the same time.

Leslie7 wrote:

I mean, that there is a huge number of calls to find service implementation and some rare calls to DeleteInterface/AddInterface.
So we can serialize calls to DeleteInterface/AddInterface, but use lock-free algorithm to find service implementation.

Imho there's no other way then locking all calls accessing fList if we want to make the whole thing really thread safe. Have a look at function TInterfaceResolverInjected.TryResolve(aInterface: PTypeInfo; out Obj): boolean;. There's a GlobalInterfaceResolutionLock involved too when adding/deleting/getting interfaces from global storage. It's always dangerous to make Add/Delete methods thread safe, but not Get methods.
The questions is: is it really time-critical to intruduce a lock for the Resolve() method here?

#65 Re: mORMot 1 » Websocket notification from server to web client » 2016-06-22 19:11:03

oz

Imho you are trying to solve that problem on the wrong application tier. TWebSocketProtocolChat.ProcessFrame is very low level. That's the reason why you are facing that problem at all.
Taking the chat service as example, there is:

procedure TChatService.Join(const pseudo: string; const callback: IChatCallback);

Put the callback parameter together with session params you can retrieve from global threadvar ServiceContext in some kind of simple data object/record. Store that object/record to any kind of list/array instead of "fConnected: array of IChatCallback". You can identify single sessions by searching the array/list now. The found data record/object has a direct reference to the IChatCallback. You have direct access to its functions for this single client connection.

#66 Re: mORMot 1 » Disabling/switching interface implementations at runtime » 2016-06-22 18:35:00

oz

In my opinion there is no need to make AddInterface() and DeleteInterface() method thread safe at all. AddInterface() exists for a very long time right now, and there has never been any problem with it.
Calling DeleteInterface() is quite a low-level task, and you should have a good reason to do that. This feature is not intended to be called heavily anywhere in your server sources. The programmer is responsible to ensure that AddInterface() and DeleteInterface() method calls are done in thread safe functions. Personally I use those features only inside a single thread safe singleton class.

#67 Re: mORMot 1 » Disabling/switching interface implementations at runtime » 2016-06-22 16:26:30

oz
Leslie7 wrote:

Interlocked commands allow you to check if the original value is at place at the time of the exchange.   If not you can restart  the process  in a loop until your modified list can be safely exchanged. Checking for time out in the loop is  most likely not necessary for normal use case, but probably better to include because of possible stress tests.

I know about Interlocked*** commands, but that would not help in this situation imho. By calling InterlockedExchangePointer you are exactly doing what the method name tells you: you are exchanging one pointer with another one, in a thread safe, atomic way. And that's the problem. Exchanging the pointer is atomic, but the DeleteInterface/AddInterface method is still a non-atomic operation. If 2 or more threads are calling DeleteInterface() method then you will face exactly the same problems as you do without the InterlockedExchangePointer call. It does not help. InterlockedExchangePointer only ensures that exchanging the fList pointer is an atomic operation, it does not care about the data "behind" that pointer. Various threads would "overwrite" fList again and again without seeing each others changes. Comparing to the old value doesn't help you either.

#68 Re: mORMot 1 » Disabling/switching interface implementations at runtime » 2016-06-22 06:56:08

oz

Creating copies of those list does not help here. That would be "thread safe" only by the meaning of no access violations happening.
Imagine what happens if 2 threads are calling DeleteInterface at the same time, each one deleting another interface. Each call to DeleteInterface creates its own copy of the lists, deletes the interface and exchanges the pointer to that list. It will happen that one of those DeleteInterface calls would be "lost" for sure because at the time when Thread 2 is creating the private fList copy, Thread 1 hasn't exchanged the fList pointer yet.

So, if the goal is to make this function thread safe then there is no other way then locking.

I'll write down some more thoughts/advices about that whole thing later...

#69 Re: mORMot 1 » Disabling/switching interface implementations at runtime » 2016-06-21 13:51:51

oz

Well, imho it's not thread safe because access to TServiceContainer.fList and TServiceContainer.fListInterfaceMethod is done without any kind of locking involved.
I think it's not a good idea to introduce locks for "fList"/"fListInterfaceMethod" access in general. Those fields are accessed in many TServiceContainer methods. Locking every access to those fields could result in performance problems i guess. Maybe it's a good idea to add a lock for the "DeleteInterface" method call itself. This would prevent potential "index out of bounds" exceptions when calling TServiceContainer.DeleteInterface() from several thread at once.

  TServiceContainer = class(TInterfaceResolverInjected)
  protected
    fDeleteInterfaceLock: TSynLocker;
  ...
constructor TServiceContainer.Create(aRest: TSQLRest);
begin
  ...
  fDeleteInterfaceLock.Init;
end;
destructor TServiceContainer.Destroy;
var i: integer;
begin
  for i := 0 to fList.Count-1 do
    fList.Objects[i].Free;
  fList.Free;
  fDeleteInterfaceLock.Done;
  inherited;
end;
procedure TServiceContainer.DeleteInterface(const aInterfaces: array of PTypeInfo);
var
  iInterfaces: integer;
  iMethods: integer;
  Idx: integer;
  MethodName: RawUTF8;
  InterfaceName: RawUTF8;
  tmpInterfaceName: RawUTF8;
begin
  fDeleteInterfaceLock.Lock;
  try
    for iInterfaces:=Low(aInterfaces) to High(aInterfaces) do begin
      tmpInterfaceName:=aInterfaces[iInterfaces]^.Name;
      if tmpInterfaceName[1] in ['I','i'] then
        Delete(tmpInterfaceName,1,1);
      idx:=fList.IndexOf(tmpInterfaceName);
      if idx>-1 then begin
        fList.Objects[idx].Free;
        fList.Delete(idx);
      end;
      for iMethods:=High(fListInterfaceMethod) downto Low(fListInterfaceMethod) do begin
        Split(fListInterfaceMethod[iMethods].InterfaceDotMethodName,'.',InterfaceName,MethodName);
        if InterfaceName=tmpInterfaceName then
          fListInterfaceMethods.Delete(iMethods);
      end;
    end;
  finally
    fDeleteInterfaceLock.UnLock;
  end;
end;

I haven't created a testcase yet, but I see no problems working under server load. The worst thing that could happen is that a call to "Service.Resolve(IDomInterfaceToExchange)" would fail if the Interface is being deleted. I haven't checked what happens to currently allocated "IDomInterfaceToExchange" objects/references when freeing the factory itself. But as far as i understand it should have no impact on already assigned objects implementing that particular interface. They should remain assigned until you set them to NIL.

#70 Re: mORMot 1 » Disabling/switching interface implementations at runtime » 2016-06-21 11:11:08

oz

Hi Arnaud,

I forgot about this topic, but I've implemented it successfully some time ago. Could you please review the code and incorporate it into trunk!?

Following changes have to be applied to "mORMot.pas":

1. New public procedure "SeviceUndefine" in TSQLRestServer

procedure TSQLRestServer.ServiceUndefine(const aInterfaces: array of TGUID);
begin
  (ServiceContainer as TServiceContainerServer).DeleteInterface(TInterfaceFactory.GUID2TypeInfo(aInterfaces));
end;

2. New public procedure "DeleteInterface" in TServiceContainer

procedure TServiceContainer.DeleteInterface(const aInterfaces: array of PTypeInfo);
var
  iInterfaces: integer;
  iMethods: integer;
  Idx: integer;
  MethodName: RawUTF8;
  InterfaceName: RawUTF8;
  tmpInterfaceName: RawUTF8;
begin
  for iInterfaces:=Low(aInterfaces) to High(aInterfaces) do begin
    tmpInterfaceName:=aInterfaces[iInterfaces]^.Name;
    if tmpInterfaceName[1] in ['I','i'] then
      Delete(tmpInterfaceName,1,1);
    idx:=fList.IndexOf(tmpInterfaceName);
    if idx>-1 then begin
      fList.Objects[idx].Free;
      fList.Delete(idx);
    end;
    for iMethods:=High(fListInterfaceMethod) downto Low(fListInterfaceMethod) do begin
      Split(fListInterfaceMethod[iMethods].InterfaceDotMethodName,'.',InterfaceName,MethodName);
      if InterfaceName=tmpInterfaceName then
        fListInterfaceMethods.Delete(iMethods);
    end;
  end;
end;

By using those methods one is able to switch Interface-Implementations (and Factories) at runtime.

Kind regards,
oz.

#71 Re: mORMot 1 » JSON RESTful web api from DataSnap to mORMot » 2016-06-20 12:00:03

oz

You are storing the data (fSessionData) to the TSQLAuthUser object, not in session context here.
Have a look at "ServiceContext: TServiceRunningContext;" declaration in mORMot.pas to read more about session context.

#72 Re: mORMot 1 » Request for a HTTP server workflow changes » 2016-06-10 21:13:10

oz

Personally, i'm also very interested in seeing this feature beeing implemented in trunk.
Unfortunately I had no time to give that http.sys based WebSocket server a try, neither to have a look at that fork.
But it really sounds promising imho.

#74 Re: mORMot 1 » Change request: Convert "procedure ObjectToJSONFile()" into a function » 2016-06-10 13:32:56

oz

FileFromString could fail because of wrong FileName parameter, or because write access to FileName is not allowed.
More generally: an operation could always fail if file i/o is involved.

#75 mORMot 1 » Change request: Convert "procedure ObjectToJSONFile()" into a function » 2016-06-10 08:50:56

oz
Replies: 4

Hi Arnaud,

imho the method "ObjectToJSONFile" should be a function, not procedure. The current implementation is:

procedure ObjectToJSONFile(Value: TObject; const JSONFile: TFileName;
  Options: TTextWriterWriteObjectOptions);
var humanread: boolean;
    json: RawUTF8;
begin
  humanread := woHumanReadable in Options; 
  Exclude(Options,woHumanReadable);
  json := ObjectToJSON(Value,Options);
  if humanread then
    // woHumanReadable not working with custom JSON serializers, e.g. T*ObjArray
    JSONBufferReformatToFile(pointer(json),JSONFile) else
    FileFromString(json,JSONFile);
end;

"JSONBufferReformatToFile()" and "FileFromString()" do return valueable output, so that should be passed as result.
Changing that procedure into a function should have no impacts on existing code.
What do you thing, could you include following change to mORMot.pas:

function ObjectToJSONFile(Value: TObject; const JSONFile: TFileName;
  Options: TTextWriterWriteObjectOptions): boolean;
var humanread: boolean;
    json: RawUTF8;
begin
  humanread := woHumanReadable in Options; 
  Exclude(Options,woHumanReadable);
  json := ObjectToJSON(Value,Options);
  if humanread then
    // woHumanReadable not working with custom JSON serializers, e.g. T*ObjArray
    result:=JSONBufferReformatToFile(pointer(json),JSONFile) else
    result:=FileFromString(json,JSONFile);
end;

#76 Re: mORMot 1 » WinHTTP, http.sys and WebSockets » 2016-05-25 06:36:48

oz

I'm very interested about this topic too!

Could you loose some words about implementation details?
- Encryption: are https connections supported or does it use some other encryption mechanism
- Is there some kind of fallback to mORMots default implementation if http.sys websockets are not available?
- ...
Kind regards

#77 Re: mORMot 1 » Best way to free/manage threads » 2016-05-18 08:14:03

oz

Hi,
this behaviour is alteady implemented in:

TThread.FreeOnTerminate:=true;

#78 mORMot 1 » Disabling/switching interface implementations at runtime » 2016-03-24 12:24:55

oz
Replies: 16

Hi Arnaud,

i'm in need of being able to switch/disable service interface implementations without the need to restart the mORMot server. That's why I need to have counterparts for TIntefaceResolver.InjectResolver() and TSQLRestServer.ServiceDefine(). Those two methods would allow to replace/disable interface implementations at runtime.

My DeleteResolver implementation which seems to do it's job:

procedure TInterfaceResolverInjected.DeleteResolver(aResolver: TInterfaceResolver);
var
  ix: integer;
begin
  ix:=ObjArrayFind(fResolvers,aResolver);
  if ix>0 then
    ObjArrayDelete(fResolvers,ix);
end;

But I don't know how to implement TSQLRestServer.ServiceUnDefine without diving deep into the source. Could you provide some help please?

Kind regards,
oz.

#79 mORMot 1 » Issue with TDynArray.Delete() » 2016-03-21 20:15:53

oz
Replies: 1

Hi Arnaud,

there's an issue with TDynArray.Delete() if the TDynArray contains any kind of T*ObjArray objects.
TDynArray.Delete(aIndex) will not only call the object destructor at [aIndex] position...
Going down the callstack to some deeper level shows that TDynArray.InternalSetLength() calls the next object's destructor too. The array size does not decrease... at the end access violations will happen.

Testcase to reproduce the bug:

type
  TPerson=class(TSynPersistent)
  private
    fFullName: RawUTF8;
  published
    property FullName: RawUTF8 read fFullName write fFullName;
  end;
  TPersonObjArray=array of TPerson;

procedure TTestDynArray.DynArrayDelete;
var
  DA: TDynArray;
  arrPerson: TPersonObjArray;
  i: integer;
  person: TPerson;
const
  cMax=3;
begin
  TJSONSerializer.RegisterObjArrayForJSON([TypeInfo(TPersonObjArray),TPerson]);
  DA.Init(TypeInfo(TPersonObjArray), arrPerson);
  for i:=1 to cMax do
  begin
    person:=TPerson.Create;
    person.FullName:=FormatUTF8('FullName-%',[(i)]);
    DA.Add(person);
  end;
  for i:=Low(arrPerson) to High(arrPerson) do
  begin
    person:=arrPerson[(i)];
    Check(person.FullName=FormatUTF8('FullName-%',[(i+1)]));
  end;
  DA.Delete(0);
  for i:=Low(arrPerson) to High(arrPerson) do
  begin
    person:=arrPerson[(i)];
    Check(person.FullName=FormatUTF8('FullName-%',[(i+2)]));              // BOOM!!! -> access violation 
  end;
  DA.Clear;
end;

TDynArray.Delete implementation:

procedure TDynArray.Delete(aIndex: Integer);
var n, len: integer;
    P: PAnsiChar;
begin
  if fValue=nil then
    exit; // avoid GPF if void
  n := Count;
  if cardinal(aIndex)>=cardinal(n) then
    exit; // out of range
  dec(n);
  P := pointer(PtrUInt(fValue^)+PtrUInt(aIndex)*ElemSize);
  if ElemType<>nil then
    _Finalize(P,ElemType) else
    if GetIsObjArray then
      FreeAndNil(PObject(P)^);                  // <- The TPerson object is destroyed here
  if n>aIndex then begin
    len := cardinal(n-aIndex)*ElemSize;
    MoveFast(P[ElemSize],P[0],len);
    if ElemType<>nil then // avoid GPF
      FillcharFast(P[len],ElemSize,0);
  end;
  SetCount(n);                                  // <- this leads to the issue...
end;
procedure TDynArray.SetCount(aCount: integer);
.
.
.
      end else
      if aCount>0 then // aCount=0 should release memory (e.g. TDynArray.Clear)
        // size-down -> only if worth it (for faster Delete)
        if (capa<=MINIMUM_SIZE) or (capa-aCount<capa shr 3) then
          exit;
    end;
  end;
  // no external Count, array size-down or array up-grow -> realloc
  InternalSetLength(aCount);           // <- this leads to the issue...
end;

Going deeper...

procedure TDynArray.InternalSetLength(NewLength: PtrUInt);
var p: PDynArrayRec;
    pa: PAnsiChar absolute p;
    OldLength, NeededSize, minLength: PtrUInt;
    pp: pointer;
    i: integer;
begin // this method is faster than default System.DynArraySetLength() function
  // check that new array length is not just a hidden finalize
  if NewLength=0 then begin
    {$ifndef NOVARIANTS} // faster clear of custom variant uniformous array
    if ArrayType=TypeInfo(TVariantDynArray) then begin
      VariantDynArrayClear(TVariantDynArray(fValue^));
      exit;
    end;
    {$endif}
    if GetIsObjArray then
      for i := 0 to Count-1 do
        PObjectArray(fValue^)^[(i)].Free;
    _DynArrayClear(fValue^,ArrayType);
    exit;
  end;
  // retrieve old length
  p := fValue^;
  if p<>nil then begin
    dec(PtrUInt(p),Sizeof(TDynArrayRec)); // p^ = start of heap object
    OldLength := p^.length;
  end else
    OldLength := 0;
  // calculate the needed size of the resulting memory structure on heap
  NeededSize := NewLength*ElemSize+Sizeof(TDynArrayRec);
  if NeededSize>1024*1024*512 then // max allowed memory block is 512MB
    raise ERangeError.CreateFmt('TDynArray SetLength(%s,%d) size concern',
      [PShortString(@PDynArrayTypeInfo(ArrayType).NameLen)^,NewLength]);
  // if not shared (refCnt=1), resize; if shared, create copy (not thread safe)
  if (p=nil) or (p^.refCnt=1) then begin
    if NewLength<OldLength then
      if ElemType<>nil then
        _FinalizeArray(pa+NeededSize,ElemType,OldLength-NewLength) else
        if GetIsObjArray then
          for i := NewLength to OldLength-1 do
            PObjectArray(fValue^)^[(i)].Free;                // <- the TPerson object is destroyed another time. Without this call everything works as expeced.
    ReallocMem(p,neededSize);
  end else begin
    InterlockedDecrement(PInteger(@p^.refCnt)^); // FPC has refCnt: PtrInt
    GetMem(p,neededSize);
    minLength := oldLength;
    if minLength>newLength then
      minLength := newLength;
    if ElemType<>nil then begin
      pp := pa+Sizeof(TDynArrayRec);
      FillcharFast(pp^,minLength*elemSize,0);
      CopyArray(pp,fValue^,ElemType,minLength)
    end else
      MoveFast(fValue^,pa[Sizeof(TDynArrayRec)],minLength*elemSize);
  end;
  // set refCnt=1 and new length to the heap memory structure
  with p^ do begin
    refCnt := 1;
    {$ifdef FPC}
    high := newLength-1;
    {$else}
    length := newLength;
    {$endif}
  end;
  Inc(PtrUInt(p),Sizeof(p^));
  // reset new allocated elements content to zero
  if NewLength>OldLength then begin
    OldLength := OldLength*elemSize;
    FillcharFast(pa[OldLength],neededSize-OldLength-Sizeof(TDynArrayRec),0);
  end;
  fValue^ := p;
end;

As a quick fix I simply disabled the "PObjectArray(fValue^)^[(i)].Free;" call. For now everything works as expected, but I don't know about potential side effects.

bye,
oz.

#80 Re: mORMot 1 » How to free a RawByteStringToStream » 2016-01-13 08:16:25

oz
Thomas-Acia wrote:

I do this :

  temp:=TStream.Create;    
  [...]
  temp := RawByteStringToStream(content);
  [...]
  temp.Free;    

but I have the feeling that the stream is not correctly released.

You should use:

  temp := RawByteStringToStream(content);
  try
    [...]
  finally
    temp.Free;
  end;    

Just have a look at RawByteStringToStream() implementation. The stream is created by this function.

#81 Re: mORMot 1 » Add a object with blob in one POST » 2015-12-10 14:36:54

oz

On Batch-Level use following overloaded method: TSQLRestBatch.Add(Value: TSQLRecord; SendData,ForceID: boolean; const CustomFields: TSQLFieldBits; DoNotAutoComputeFields: boolean): integer;


This would send all fields, including Blobs:

ConnectionManager.DocumentClient.BatchAdd(msg.Details, True, False, ALL_FIELDS);

#82 Re: mORMot 1 » blob cache » 2015-12-09 15:12:01

oz

Your test code shows only one UpdateBlob method call. It would be helpful to see more code.
Did you try Arnauds suggestion with breakpoints? Maybe your rec object already contains wrong values.
Another good thing to do is to seperate the issue into some testcase based demo project.
Did you try other DataAccess components (zeos) too?

#83 Re: mORMot 1 » Suggestion: Protocol independent interface callbacks » 2015-12-09 14:42:23

oz
ab wrote:

Update: I've made some more refactoring, so that coupling to WebSockets would be ever more reduced.
See http://synopse.info/fossil/info/1df6154ab2

Now you would be able to enhance a THttpServer class:
- Set   fCanNotifyCallback := true in the constructor, as in TWebSocketServer;
- Override THttpServerGeneric.Callback virtual method, as in TWebSocketServerRest.

Thanks to this, either long or period pooling may be used to implement callbacks.

Arnaud...
What the hell...!
Mate, you are amazingly fast and your support is absolutely outstandig!

You've just managed to decouple things before I could even post my answer regarding your previous message. smile

Thank's a lot!

I'll have a look at your enhancements.

#84 Re: mORMot 1 » Suggestion: Protocol independent interface callbacks » 2015-12-09 14:37:37

oz
ab wrote:

AFAIK WebSockets are very proxy-friendly, unless there is packet inspection, which is very doubtful.

That's exactly the root of all evil. Some of our customers have very restrictive high-security network setups, using packet inspection/traffic analysis. I don't know which kind of firewall is running over there, but it has to be a very good one. Their firewall is able to detect the HTTP WebSocket upgrade change and kills the connection. Unfortunately this behaviour cannot be changed, because it is used to detect other software product who are doing similiar things, but are blacklisted by customer's IT dept. But even disabling packet inpection won't help, because all traffic is routed through a HTTP proxy. smile

ab wrote:

By design, the actual implementation is not WebSockets specific.
You could add your own callback mechanism in TSQLHttpServer.NotifyWebSocketsCallback.

I'll try to make it even more uncoupled.

Thanks for the hint, i'll have a look at TSQLHttpServer.NotifyWebSocketsCallback and will try to find out how that whole thing is working.

#85 mORMot 1 » Suggestion: Protocol independent interface callbacks » 2015-12-09 12:46:38

oz
Replies: 4

Hi!

I'd really like to use mORMot's interface callback push mechanism in my current mORMot based SOA DDD project. Unfortunately I can't use those interface callbacks because it's current implementation is bound to the WebSockets protocol. Using WebSockets for client/server communication is no option in my target business because of several reasons (firewall policies, proxy policies, etc ...).
Furthermore I really don't like the fact that one is loosing http.sys support when upgrading to WebSockets. Please correct me if i'm wrong about that, but afaik WebSockets are not supported by the http.sys server right now.

That's why i've implemented a message based Polling/LongPolling server-to-client push-like notification solution. The implementation is based on message objects and a publish/subscripe mechanism.
On server-side those notification message objects are stored as JSON in an in-memory table. Simplified code:

type
  TSQLPushNotification=class(TSQLRecord)
  published
    property PushToSessionID: TSessionID;  // Session ID this notification is meant for
    property JsonData: TJsonData; // The data object
  end;

On client-side there's the ability to switch between polling and long polling modes. Polling mode utilizes a background thread who simply is querying the TSQLPushNotification table every X seconds. Long polling mode uses a permanent and dedicated 2nd connection within a background thread. I could go into details here, but I'll assume you are familiar with longpolling mechanism.

Now i'm wondering if it is possible to "cut off" the WebSocket binding of interface callbacks implementation and replace them with any other kind of transmission layer, e.g. my long polling implementation. The idea is to introduce some kind of "push-abstraction-layer" who is not bound to a specific protocol.

What do you think, is it possible to do sth. like that?

Kind regards,
oz.

#86 Re: mORMot 1 » Serious bug/issue regarding WebSockets » 2015-09-23 19:13:32

oz
ab wrote:

I did not have the ESynCrypto exception.

I received a 404 error, mainly due to the fact that the callbacks are a global list.
This IMHO a wrong design of your sample.
Callbacks should be private to each instance, and released/unregistered ASAP from the client side.
You are sharing callbacks, and some are still trying to be notified even if their TMyHttpClient owner is not there any more.
IMHO it should be in each TTestRest to maintain the callback list.

Hey, it's me again wink
After re-reading your post: the usage of this is to notify all connected clients that some event has occured. So the design is based on the chat server sample.

#87 Re: mORMot 1 » Serious bug/issue regarding WebSockets » 2015-09-23 16:26:52

oz

One more thing:
Most of the times there are 404 errors prior to the exception stated above.
But just right now at the moment the exception occured without any prior 404 error in my testcase.
Maybe that information helps.
King regards,
oz.

#88 Re: mORMot 1 » Serious bug/issue regarding WebSockets » 2015-09-23 16:20:14

oz

Hi Arnaud,

first of all: thanks for your quick reply, as usual!

Yes, you are right, the 404 error can happen! But, that's not the problem. As far as I understand, it's not because a global list used to store callbacks on server side.
This global list ist only used by the server. The clients are keeping their callback instances in the private fCallback: ICallback variable. Such 404 errors can always happen because the client which should be notified, could already have disconnected. That 404 error is perfectly catched by the try...except block inside TTestRest.Add().

I'm quite sure that this is not the problem. Let me describe my thougths in more detail:

TTestRest is implementing IMyCommand in sicClientDriven mode.
This means that each connected client has its own TTestRest object living on server side.
That's why some kind of global list has to be used, otherwise each callbacks list (which is private to TTestRest then) would only contain one single entry. Access to the global List ist secured using CriticalSections, so there can't be any threading issues.

By the way: Imho the included demo "Project31ChatServer.dpr" is doing exactly the same at the end. The server-side interface is using "sicShared", which means that there is only one instance on server, shared by all clients. So, at the end, "fConnected: array of IChatCallback;" used in the demo to store the callbacks is exactly the same which is done by my testcase.
I think i'm right about that, or did I miss the point?

Arnaud, i'm quite sure that there's an issue with WebSockets implementation under heavy server load.

In my real project, those callbacks on server-side are not stored within a global list. I'm using my own custom session handling inside my business logic tier, which does not depend on mORMot's sessions. At the end, there's something like a TmySession instance for each connected client. Every TmySession has one IMyCallback reference to the callback Interface registered by the client. Everything runs perfectly fine, until running the heavy load stress test. From time to time the 404 errors occur, but that's ok, they are catched and those 404 errors can't be prevented. They just happen sometimes.

I'm pretty sure there's an issue with WebSockets, because of strange testcase behaviour.
70 out of 100 times the testcase will produce 404 errors. Ok.
29 out of 100 times the testcase runs without errors. Ok.
BUT: Sometimes, maybe 1 out of 100 times there are exceptions in:
Stacktrace:

TAESAbstract.DecryptPKCS7('PƒäÈZ#]2gzþ¥éÁŒNçSá¶î¢âM”G'#4#5'~xp`*væ©hW'#$A'†'#$1F'¥NÃgPcc@œeÁ€67$æù'#$1E'$dþ>^‘'#$1C'ª8GCOA½$'#$E',šEÇÞGŒ'#$1C'ÕfÙÔ¶°'#$1C'â'#3#$11'Ëÿm¿Œ+Ç'#$F#$B'•'#$1A'Ü'#$B'›[ÈZ¹Ö'#$C'='#$10#$10'ÿÿ'#8'6´'#$15'T¾?ÊY¼–á!@WÆÚEŒŒJ'#$F'1æ2,ušÌXJçß'#$14'P#…Šïü9érÀ?wöûŒ'#$12#$12#$12#$10'Ýw<TnëÀÓûžk²¹F|'#$C'‰ùEôÏYj!§O'#8#$E'§'#$A'[Ý«i7Sàå&(×€ý®Ëï¹jË~؁ù7•'#$F']¨'#$15'ؾ«ëÃ'#7'à×.‹­äFÝÖÖáãf„îÐ'#$A'1 â^3óuÜa(¸'#$D'˜',True)
TWebSocketProtocolBinary.AfterGetFrame((focBinary, 'PƒäÈZ#]2gzþ¥éÁŒNçSá¶î¢âM”G'#4#5'~xp`*væ©hW'#$A'†'#$1F'¥NÃgPcc@œeÁ€67$æù'#$1E'$dþ>^‘'#$1C'ª8GCOA½$'#$E',šEÇÞGŒ'#$1C'ÕfÙÔ¶°'#$1C'â'#3#$11'Ëÿm¿Œ+Ç'#$F#$B'•'#$1A'Ü'#$B'›[ÈZ¹Ö'#$C'='#$10#$10'ÿÿ'#8'6´'#$15'T¾?ÊY¼–á!@WÆÚEŒŒJ'#$F'1æ2,ušÌXJçß'#$14'P#…Šïü9érÀ?wöûŒ'#$12#$12#$12#$10'Ýw<TnëÀÓûžk²¹F|'#$C'‰ùEôÏYj!§O'#8#$E'§'#$A'[Ý«i7Sàå&(×€ý®Ëï¹jË~؁ù7•'#$F']¨'#$15'ؾ«ëÃ'#7'à×.‹­äFÝÖÖáãf„îÐ'#$A'1 â^3óuÜa(¸'#$D'˜'))
TWebSocketProcess.GetFrame((focBinary, 'PƒäÈZ#]2gzþ¥éÁŒNçSá¶î¢âM”G'#4#5'~xp`*væ©hW'#$A'†'#$1F'¥NÃgPcc@œeÁ€67$æù'#$1E'$dþ>^‘'#$1C'ª8GCOA½$'#$E',šEÇÞGŒ'#$1C'ÕfÙÔ¶°'#$1C'â'#3#$11'Ëÿm¿Œ+Ç'#$F#$B'•'#$1A'Ü'#$B'›[ÈZ¹Ö'#$C'='#$10#$10'ÿÿ'#8'6´'#$15'T¾?ÊY¼–á!@WÆÚEŒŒJ'#$F'1æ2,ušÌXJçß'#$14'P#…Šïü9érÀ?wöûŒ'#$12#$12#$12#$10'Ýw<TnëÀÓûžk²¹F|'#$C'‰ùEôÏYj!§O'#8#$E'§'#$A'[Ý«i7Sàå&(×€ý®Ëï¹jË~؁ù7•'#$F']¨'#$15'ؾ«ëÃ'#7'à×.‹­äFÝÖÖáãf„îÐ'#$A'1 â^3óuÜa(¸'#$D'˜'),1,False)
TWebSocketProcess.ProcessLoop
TWebSocketProcessClientThread.Execute
ThreadProc($7EEE12E0)
ThreadWrapper($7EF88580)

The exception is raised here:

function TAESAbstract.DecryptPKCS7(const Input: RawByteString;
  IVAtBeginning: boolean): RawByteString;
var len,iv,padding: integer;
begin
  len := length(Input);
  DecryptLen(len,iv,pointer(Input),IVAtBeginning);
  SetString(result,nil,len);
  Decrypt(@PByteArray(Input)^[iv],pointer(result),len);
  padding := ord(result[len]); // result[1..len]
  if padding>AESBlockSize then   // <-  THIS IS WHERE IT HAPPENS
    raise ESynCrypto.CreateUTF8('%.DecryptPKCS7: Invalid content',[self]);
  SetLength(result,len-padding);
end;

It's hard to reproduce this issue, because most of the times everything is ok. But sometimes, things start going wrong.

I hope you get my point. Feel free to ask any further questions!
I could rewrite the testcase if it helps, but I don't see that there's something wrong with it.

King regards,
oz.

#89 mORMot 1 » Serious bug/issue regarding WebSockets » 2015-09-23 10:50:37

oz
Replies: 5

Hi Arnaud!

I've just faced a serious bug in mORMot's WebSockets implementation.
The following testcase can reproduce the bug:

program mORMotBug;

{$I Synopse.inc} // define HASINLINE USETYPEINFO CPU32 CPU64 OWNNORMTOUPPER

{$APPTYPE CONSOLE}

{$R *.res}

uses
  FastMM4,
  SysUtils,
  Classes,
  Windows,
  SynSqlite3Static,
  SynSqlite3,
  SynCommons,
  SynLog,
  SynTests,
  mORMot,
  mORMotHttpServer,
  mORMotHttpClient,
  mORMotSqlite3,
  mORMotDDD;

const
  /// "FORCE_BUG = true;" will send WebSocket notifications to the other connected clients in IMyCommand.Add() method.
  // At some point there will happen an ESynCrypto Exception: 'TAESCFB.DecryptPKCS7: Invalid content'!
  // The Websocket bug will occur. "FORCE_BUG = false;" won't produce any issues.
  FORCE_BUG = true;

  HTTP_PORT = '80';
  WEBSOCKET_KEY = 'key';

type
  ICallback = interface(IInvokable)
    ['{B42B0BBA-AA9C-459B-8731-1C52441361C9}']
    procedure DoCallback(const aDescription: RawUTF8);
  end;

var
  gCallbackArray: array of ICallback;
  gCriticalSection: TRTLCriticalSection;

type
  TCallback = class(TInterfacedObject, ICallback)
  public
    procedure DoCallback(const aDescription: RawUTF8);
  end;


  // This is our simple Test data class. Will be mapped to TSQLRecordTest.
  TTest = class(TSynPersistent)
  private
    fDescription: RawUTF8;
  published
    property Description: RawUTF8 read fDescription write fDescription;
  end;
  TTestObjArray = array of TTest;

  // The corresponding TSQLRecord for TTest.
  TSQLRecordTest = class(TSQLRecord)
  private
    fDescription: RawUTF8;
  published
    property Description: RawUTF8 read fDescription write fDescription;
  end;

  // CQRS Query Interface fo TTest
  IMyQuery = interface(ICQRSService)
    ['{DD402806-39C2-4921-98AA-A575DD1117D6}']
    function SelectByDescription(const aDescription: RawUTF8): TCQRSResult;
    function SelectAll: TCQRSResult;
    function Get(out aAggregate: TTest): TCQRSResult;
    function GetAll(out aAggregates: TTestObjArray): TCQRSResult;
    function GetNext(out aAggregate: TTest): TCQRSResult;
    function GetCount: integer;
  end;

  // CQRS Command Interface for TTest
  IMyCommand = interface(IMyQuery)
    ['{F0E4C64C-B43A-491B-85E9-FD136843BFCB}']
    function Add(const aAggregate: TTest): TCQRSResult;
    function Update(const aUpdatedAggregate: TTest): TCQRSResult;
    function Delete: TCQRSResult;
    function DeleteAll: TCQRSResult;
    function Commit: TCQRSResult;
    function Rollback: TCQRSResult;
    function RegisterCallback(const aInterface: ICallback): TCQRSResult;
  end;

  // The infratructure REST class implementing the Query and Command Interfaces for TTest
  TTestRest = class(TDDDRepositoryRestCommand,IMyCommand,IMyQuery)
  public
    function SelectByDescription(const aDescription: RawUTF8): TCQRSResult;
    function SelectAll: TCQRSResult;
    function Get(out aAggregate: TTest): TCQRSResult;
    function GetAll(out aAggregates: TTestObjArray): TCQRSResult;
    function GetNext(out aAggregate: TTest): TCQRSResult;
    function Add(const aAggregate: TTest): TCQRSResult;
    function Update(const aUpdatedAggregate: TTest): TCQRSResult;
    function RegisterCallback(const aInterface: ICallback): TCQRSResult;
  end;

  // REST Factory for TTestRest instances
  TTestRestFactory = class(TDDDRepositoryRestFactory)
  public
    constructor Create(aRest: TSQLRest; aOwner: TDDDRepositoryRestManager=nil); reintroduce;
  end;

  // Test container
  TMyTests = class(TSynTestsLogged)
  published
    procedure MyTests;
  end;

  // Test case doing the actual work
  TMyTestCase = class(TSynTestCase)
  private
    // Rest server
    fRestServer: TSQLRestServerDB;
    // Http server
    fHttpServer: TSQLHttpServer;
    /// Will create as many Clients as specified by aClient.
    // - Each client will perform as many Requests as specified by aRequests.
    // - This function will wait for all Clients until finished.
    function ClientTest(const aClients, aRequests: integer):boolean;
  protected
    // Cleaning up the test
    procedure CleanUp; override;
  published
    // Delete any old Test database on start
    procedure DeleteOldDatabase;
    // Start the whole DDD Server (http and rest)
    procedure StartServer;
    // Test straight-forward access using 1 thread and 1 client
    procedure SingleClientTest;
    // Test concurrent access with multiple clients. This will crash!
    procedure MultiClientTest;
  end;

  // Custom TSQLHttpClient encapsulating the remote IMyCommand interface.
  TMyHttpClient=class(TSQLHttpClientWebsockets)
  private
    // Internal Model
    fModel: TSQLModel;
    // IMyCommand interface. Will be assigned inside SetUser
    fMyCommand: IMyCommand;
    fCallback: ICallback;
  public
    constructor Create(const aServer,aPort: RawUTF8); //overload;
    destructor Destroy; override;
    function SetUser(const aUserName, aPassword: RawUTF8; aHashedPassword: Boolean=false): boolean; reintroduce;
    property MyCommand: IMyCommand read fMyCommand;
  end;

  // The thread used by TMyTestCase.ClientTest
  TMyThread = class(TThread)
  private
    fHttpClient: TMyHttpClient;
    fRequestCount: integer;
    fId: integer;
    fIsError: boolean;
  protected
    procedure Execute; override;
  public
    constructor Create(const aId, aRequestCount: integer);
    destructor Destroy; override;
    property IsError: boolean read fIsError;
  end;

{ TTestRest }

function TTestRest.SelectByDescription(
  const aDescription: RawUTF8): TCQRSResult;
begin
  result := ORMSelectOne('Description=?',[aDescription],(aDescription=''));
end;

function TTestRest.SelectAll: TCQRSResult;
begin
  result := ORMSelectAll('',[]);
end;

function TTestRest.Get(out aAggregate: TTest): TCQRSResult;
begin
  result := ORMGetAggregate(aAggregate);
end;

function TTestRest.GetAll(
  out aAggregates: TTestObjArray): TCQRSResult;
begin
  result := ORMGetAllAggregates(aAggregates);
end;

function TTestRest.GetNext(out aAggregate: TTest): TCQRSResult;
begin
  result := ORMGetNextAggregate(aAggregate);
end;

function TTestRest.Add(const aAggregate: TTest): TCQRSResult;
var
  i: integer;
begin
  result := ORMAdd(aAggregate);
  if not FORCE_BUG then
    exit;
  EnterCriticalSection(gCriticalSection);
  try
    for i := high(gCallbackArray) downto 0 do // downwards for InterfaceArrayDelete()
      try
        gCallbackArray[i].DoCallback('');
      except
        InterfaceArrayDelete(gCallbackArray,i); // unsubscribe the callback on failure
      end;
  finally
    LeaveCriticalSection(gCriticalSection);
  end;
end;

function TTestRest.Update(
  const aUpdatedAggregate: TTest): TCQRSResult;
begin
  result := ORMUpdate(aUpdatedAggregate);
end;


{ TInfraRepoUserFactory }

constructor TTestRestFactory.Create(aRest: TSQLRest;
  aOwner: TDDDRepositoryRestManager);
begin
  inherited Create(IMyCommand,TTestRest,TTest,aRest,TSQLRecordTest,aOwner);
end;

function TTestRest.RegisterCallback(
  const aInterface: ICallback): TCQRSResult;
begin
  EnterCriticalSection(gCriticalSection);
  try
    InterfaceArrayAdd(gCallbackArray, aInterface);
  finally
    LeaveCriticalSection(gCriticalSection);
  end;
  result:=cqrsSuccess;
end;

{ TMyTests }

procedure TMyTests.MyTests;
begin
  AddCase([TMyTestCase]);
end;

{ TMyTestCase }

procedure TMyTestCase.CleanUp;
var
  i: integer;
begin
  EnterCriticalSection(gCriticalSection);
  try
    for i:=High(gCallbackArray) downto 0 do
      InterfaceArrayDelete(gCallbackArray, i);
  finally
    LeaveCriticalSection(gCriticalSection);
  end;
  if Assigned(fHttpServer) then
    FreeAndNil(fHttpServer);
  if Assigned(fRestServer) then
    FreeAndNil(fRestServer);
end;

procedure TMyTestCase.DeleteOldDatabase;
begin
  if FileExists(ChangeFileExt(ParamStr(0), '.db3')) then
    SysUtils.DeleteFile(ChangeFileExt(ParamStr(0), '.db3'));
  CheckNot(FileExists(ChangeFileExt(ParamStr(0), '.db3')));
end;

procedure TMyTestCase.StartServer;
begin
  fRestServer:=TSQLRestServerDB.CreateWithOwnModel([TSQLRecordTest], ChangeFileExt(ParamStr(0), '.db3'), true);
  with fRestServer do begin
      DB.Synchronous := smNormal;
      DB.LockingMode := lmExclusive;
      CreateMissingTables();
      TInterfaceFactory.RegisterInterfaces([TypeInfo(IMyQuery),TypeInfo(IMyCommand)]);
      ServiceContainer.InjectResolver([TTestRestFactory.Create(fRestServer)],true);
      ServiceDefine(TTestRest,[IMyCommand],sicClientDriven);
    end;
  fHttpServer:=TSQLHttpServer.Create(HTTP_PORT, fRestServer, '+', useBidirSocket);
  fHttpServer.WebSocketsEnable(fRestServer, WEBSOCKET_KEY);
end;

procedure TMyTestCase.MultiClientTest;
begin
  ClientTest(20,50);
end;

procedure TMyTestCase.SingleClientTest;
var
  HttpClient: TMyHttpClient;
  test: TTest;
  i: integer;
const
  MAX = 1000;
begin
  HttpClient:=TMyHttpClient.Create('localhost', HTTP_PORT);
  try
    Check(HttpClient.SetUser('Admin', 'synopse'));
    test:=TTest.Create;
    try
      for i:=0 to MAX-1 do begin
        test.Description:=FormatUTF8('test-%',[i]);
        Check(HttpClient.MyCommand.Add(test)=cqrsSuccess);
      end;
      Check(HttpClient.MyCommand.Commit=cqrsSuccess);
    finally
      test.Free;
    end;
  finally
    HttpClient.Free;
  end;
end;

function TMyTestCase.ClientTest(const aClients, aRequests: integer):boolean;
var
  i: integer;
  arrThreads: array of TMyThread;
  arrHandles: array of THandle;
  rWait: Cardinal;
begin
  result := false;
  SetLength(arrThreads, aClients);
  SetLength(arrHandles, aClients);
  for i:=Low(arrThreads) to High(arrThreads) do
  begin
    arrThreads[i]:=TMyThread.Create(i,aRequests);
    arrHandles[i]:=arrThreads[i].Handle;
    arrThreads[i].Resume;
  end;
  try
    repeat
      rWait:= WaitForMultipleObjects(aClients, @arrHandles[0], True, INFINITE);
    until rWait<>WAIT_TIMEOUT;
  finally
    for i:=Low(arrThreads) to High(arrThreads) do
    begin
      CheckNot(arrThreads[i].IsError);
      arrThreads[i].Free;
    end;
    SetLength(arrThreads, 0);
    SetLength(arrHandles, 0);
  end;
end;

{ TMyHttpClient }

constructor TMyHttpClient.Create(const aServer,aPort: RawUTF8);
begin
  fModel:=TSQLModel.Create([TSQLRecordTest]);
  fCallback:=TCallback.Create;
  inherited Create(aServer, aPort, fModel);
end;

destructor TMyHttpClient.Destroy;
begin
  fCallback:=nil;
  fMyCommand:=nil;
  inherited;
  fModel.Free;
end;

function TMyHttpClient.SetUser(const aUserName, aPassword: RawUTF8; aHashedPassword: Boolean=false): boolean;
begin
  result := inherited SetUser(aUserName, aPassword, aHashedPassword);
  WebSocketsUpgrade(WEBSOCKET_KEY);
  if result then
  begin
    ServiceDefine([IMyCommand],sicClientDriven);
    Services.Resolve(IMyCommand, fMyCommand);
  end;
  fMyCommand.RegisterCallback(fCallback);
end;


{ TMyThread }

constructor TMyThread.Create(const aID, aRequestCount: integer);
begin
  inherited Create(true);
  fRequestCount:=aRequestCount;
  fId:=aId;
  fIsError:=false;
  fHttpClient := TMyHttpClient.Create('localhost', HTTP_PORT);
  fHttpClient.SetUser('Admin', 'synopse');
end;

destructor TMyThread.Destroy;
begin
  fHttpClient.Free;
  inherited;
end;

procedure TMyThread.Execute;
var
  i: integer;
  test: TTest;
  success: boolean;
begin
  test:=TTest.Create;
  try
    success:=true;
    for i:=0 to fRequestCount-1 do begin
      test.Description:=FormatUTF8('test-%-%',[fID, i]);
      success:=success and (fHttpClient.MyCommand.Add(test)=cqrsSuccess);
      if not success then
        break;
    end;
    if success then
      success:=fHttpClient.MyCommand.Commit=cqrsSuccess;
    if not success then
    begin
      fIsError:=true;
      raise Exception.Create('Something went wrong!');
    end;
  finally
    test.Free;
  end;
end;

{ TCallback }

procedure TCallback.DoCallback(const aDescription: RawUTF8);
begin
  //writeln('callback!');
end;

begin
  InitializeCriticalSection(gCriticalSection);
  with TMyTests.Create('mORMot DDD Test') do
  try
    Run;
  finally
    Free;
  end;
  WriteLn(#13#10'Done - Press ENTER to Exit');
  ReadLn;
  DeleteCriticalSection(gCriticalSection);
end.

The issue happens under heavy server load.
At some point there will happen an ESynCrypto Exception: 'TAESCFB.DecryptPKCS7: Invalid content' while sending WebSocket notifications inside IMyCommand.Add() !
If you set "FORCE_BUG = false;" then there won't be any issues. This option will disable WebSocket notifications inside IMyCommand.Add method.
King regards,
oz.

#90 Re: mORMot 1 » Possibly a Serious bug in mORMot's DDD functionality! » 2015-09-08 19:25:01

oz

Thank you for integrating my tests into the framework!

Yeah, Multi-thread could be hard, but mastered at the end wink

Well, when talking about multi threading... wink

In my current project, there will be soon the need to run defined jobs at planned times (TDateTime) inside the server process.
Those jobs are well defined happenings in my domain, but leaving the domain language and going technically:
At the end, a "job" will be a function called by a thread and do something like:

procedure ThisJobWillBeRunByAThreadpoolThreadAtDateTime;
var
  cmdA: IMyCommandA;  // a DDD CQRS interface
  cmdB: IMyCommandB; // another DDD CQRS interface
  fAuthUserCmd: IDomUkiAuthUserCommand;
begin
  Factory.Rest.Services.Resolve(IMyCommandA, cmdA);
  Factory.Rest.Services.Resolve(IMyCommandB, cmdB);
  if cmdA.Foo then
    if cmdB.Bar;
end;

Is there some kind of abstract Threadpool included in the framework which i could use for creating such a feature?

#91 Re: mORMot 1 » Possibly a Serious bug in mORMot's DDD functionality! » 2015-09-08 15:43:00

oz

Great! Thanks a lot for your support and this fantastic framework in general! smile

#92 Re: mORMot 1 » Possibly a Serious bug in mORMot's DDD functionality! » 2015-09-08 15:00:56

oz

After further investigation:
It looks like if this is a client side problem. When using only one TSQLHttpClient instance per client exe-process it seems to work just fine. I've made a Test running the standalone server and 10 client.exe processes. Each client instance using only 1 TSQLHttpClient. Each of the 10 clients did insert 50000 records without any problem.

#93 Re: mORMot 1 » Possibly a Serious bug in mORMot's DDD functionality! » 2015-09-08 10:51:08

oz
danielkuettner wrote:

I would first split server and client project and then test again.

I've done the split for debugging purpoises, but the problem remains.
Server source:

program mORMotBugServer;

{$I Synopse.inc} // define HASINLINE USETYPEINFO CPU32 CPU64 OWNNORMTOUPPER

{$APPTYPE CONSOLE}

{$R *.res}

uses
  FastMM4,
  SysUtils,
  Classes,
  Windows,
  SynSqlite3Static,
  SynSqlite3,
  SynCommons,
  SynLog,
  SynTests,
  mORMot,
  mORMotHttpServer,
  mORMotHttpClient,
  mORMotSqlite3,
  mORMotDDD;

const
  HTTP_PORT = '80';

type
  // This is our simple Test data class. Will be mapped to TSQLRecordTest.
  TTest = class(TSynPersistent)
  private
    fDescription: RawUTF8;
  published
    property Description: RawUTF8 read fDescription write fDescription;
  end;
  TTestObjArray = array of TTest;

  // The corresponding TSQLRecord for TTest.
  TSQLRecordTest = class(TSQLRecord)
  private
    fDescription: RawUTF8;
  published
    property Description: RawUTF8 read fDescription write fDescription;
  end;

  // CQRS Query Interface fo TTest
  IMyQuery = interface(ICQRSService)
    ['{DD402806-39C2-4921-98AA-A575DD1117D6}']
    function SelectByDescription(const aDescription: RawUTF8): TCQRSResult;
    function SelectAll: TCQRSResult;
    function Get(out aAggregate: TTest): TCQRSResult;
    function GetAll(out aAggregates: TTestObjArray): TCQRSResult;
    function GetNext(out aAggregate: TTest): TCQRSResult;
    function GetCount: integer;
  end;

  // CQRS Command Interface for TTest
  IMyCommand = interface(IMyQuery)
    ['{F0E4C64C-B43A-491B-85E9-FD136843BFCB}']
    function Add(const aAggregate: TTest): TCQRSResult;
    function Update(const aUpdatedAggregate: TTest): TCQRSResult;
    function Delete: TCQRSResult;
    function DeleteAll: TCQRSResult;
    function Commit: TCQRSResult;
    function Rollback: TCQRSResult;
  end;

  // The infratructure REST class implementing the Query and Command Interfaces for TTest
  TTestRest = class(TDDDRepositoryRestCommand,IMyCommand,IMyQuery)
  public
    function SelectByDescription(const aDescription: RawUTF8): TCQRSResult;
    function SelectAll: TCQRSResult;
    function Get(out aAggregate: TTest): TCQRSResult;
    function GetAll(out aAggregates: TTestObjArray): TCQRSResult;
    function GetNext(out aAggregate: TTest): TCQRSResult;
    function Add(const aAggregate: TTest): TCQRSResult;
    function Update(const aUpdatedAggregate: TTest): TCQRSResult;
  end;

  // REST Factory for TTestRest instances
  TTestRestFactory = class(TDDDRepositoryRestFactory)
  public
    constructor Create(aRest: TSQLRest; aOwner: TDDDRepositoryRestManager=nil); reintroduce;
  end;

{ TTestRest }

function TTestRest.SelectByDescription(
  const aDescription: RawUTF8): TCQRSResult;
begin
  result := ORMSelectOne('Description=?',[aDescription],(aDescription=''));
end;

function TTestRest.SelectAll: TCQRSResult;
begin
  result := ORMSelectAll('',[]);
end;

function TTestRest.Get(out aAggregate: TTest): TCQRSResult;
begin
  result := ORMGetAggregate(aAggregate);
end;

function TTestRest.GetAll(
  out aAggregates: TTestObjArray): TCQRSResult;
begin
  result := ORMGetAllAggregates(aAggregates);
end;

function TTestRest.GetNext(out aAggregate: TTest): TCQRSResult;
begin
  result := ORMGetNextAggregate(aAggregate);
end;

function TTestRest.Add(const aAggregate: TTest): TCQRSResult;
begin
  result := ORMAdd(aAggregate);
end;

function TTestRest.Update(
  const aUpdatedAggregate: TTest): TCQRSResult;
begin
  result := ORMUpdate(aUpdatedAggregate);
end;


{ TInfraRepoUserFactory }

constructor TTestRestFactory.Create(aRest: TSQLRest;
  aOwner: TDDDRepositoryRestManager);
begin
  inherited Create(IMyCommand,TTestRest,TTest,aRest,TSQLRecordTest,aOwner);
end;


var
  fRestServer: TSQLRestServerDB;
  fHttpServer: TSQLHttpServer;
begin
  if FileExists(ChangeFileExt(ParamStr(0), '.db3')) then
    SysUtils.DeleteFile(ChangeFileExt(ParamStr(0), '.db3'));
  fRestServer:=TSQLRestServerDB.CreateWithOwnModel([TSQLRecordTest], ChangeFileExt(ParamStr(0), '.db3'), true);
  try
    with fRestServer do begin
      DB.Synchronous := smNormal;
      DB.LockingMode := lmExclusive;
      CreateMissingTables();
      TInterfaceFactory.RegisterInterfaces([TypeInfo(IMyQuery),TypeInfo(IMyCommand)]);
      ServiceContainer.InjectResolver([TTestRestFactory.Create(fRestServer)],true);
      ServiceDefine(TTestRest,[IMyCommand],sicClientDriven);
    end;
    fHttpServer:=TSQLHttpServer.Create(HTTP_PORT, fRestServer, '+', useHttpApiRegisteringURI);
    try
      WriteLn(#13#10'Server running - Press ENTER to Exit');
      ReadLn;
    finally
      fHttpServer.Free;
    end;
  finally
    fRestServer.Free;
  end;
end.

Clients source:

program mORMotBugClients;

{$I Synopse.inc} // define HASINLINE USETYPEINFO CPU32 CPU64 OWNNORMTOUPPER

{$APPTYPE CONSOLE}

{$R *.res}

uses
  FastMM4,
  SysUtils,
  Classes,
  Windows,
  SynSqlite3Static,
  SynSqlite3,
  SynCommons,
  SynLog,
  SynTests,
  mORMot,
  mORMotHttpServer,
  mORMotHttpClient,
  mORMotSqlite3,
  mORMotDDD;

const
  HTTP_PORT = '80';

type
  // This is our simple Test data class. Will be mapped to TSQLRecordTest.
  TTest = class(TSynPersistent)
  private
    fDescription: RawUTF8;
  published
    property Description: RawUTF8 read fDescription write fDescription;
  end;
  TTestObjArray = array of TTest;

  // The corresponding TSQLRecord for TTest.
  TSQLRecordTest = class(TSQLRecord)
  private
    fDescription: RawUTF8;
  published
    property Description: RawUTF8 read fDescription write fDescription;
  end;

  // CQRS Query Interface fo TTest
  IMyQuery = interface(ICQRSService)
    ['{DD402806-39C2-4921-98AA-A575DD1117D6}']
    function SelectByDescription(const aDescription: RawUTF8): TCQRSResult;
    function SelectAll: TCQRSResult;
    function Get(out aAggregate: TTest): TCQRSResult;
    function GetAll(out aAggregates: TTestObjArray): TCQRSResult;
    function GetNext(out aAggregate: TTest): TCQRSResult;
    function GetCount: integer;
  end;

  // CQRS Command Interface for TTest
  IMyCommand = interface(IMyQuery)
    ['{F0E4C64C-B43A-491B-85E9-FD136843BFCB}']
    function Add(const aAggregate: TTest): TCQRSResult;
    function Update(const aUpdatedAggregate: TTest): TCQRSResult;
    function Delete: TCQRSResult;
    function DeleteAll: TCQRSResult;
    function Commit: TCQRSResult;
    function Rollback: TCQRSResult;
  end;

  // Test container
  TMyTests = class(TSynTestsLogged)
  published
    procedure MyTests;
  end;

  // Test case doing the actual work
  TMyTestCase = class(TSynTestCase)
  private
    /// Will create as many Clients as specified by aClient.
    // - Each client will perform as many Requests as specified by aRequests.
    // - This function will wait for all Clients until finished.
    function ClientTest(const aClients, aRequests: integer):boolean;
  published
    // Test straight-forward access using 1 thread and 1 client
    procedure SingleClientTest;
    // Test concurrent access with multiple clients. This will crash!
    procedure MultiClientTest;
  end;

  // Custom TSQLHttpClient encapsulating the remote IMyCommand interface.
  TMyHttpClient=class(TSQLHttpClient)
  private
    // Internal Model
    fModel: TSQLModel;
    // IMyCommand interface. Will be assigned inside SetUser
    fMyCommand: IMyCommand;
  public
    constructor Create(const aServer,aPort: RawUTF8); //overload;
    destructor Destroy; override;
    function SetUser(const aUserName, aPassword: RawUTF8; aHashedPassword: Boolean=false): boolean; reintroduce;
    property MyCommand: IMyCommand read fMyCommand;
  end;

  // The thread used by TMyTestCase.ClientTest
  TMyThread = class(TThread)
  private
    fHttpClient: TMyHttpClient;
    fRequestCount: integer;
    fId: integer;
    fIsError: boolean;
  protected
    procedure Execute; override;
  public
    constructor Create(const aId, aRequestCount: integer);
    destructor Destroy; override;
    property IsError: boolean read fIsError;
  end;


{ TMyTests }

procedure TMyTests.MyTests;
begin
  AddCase([TMyTestCase]);
end;

{ TMyTestCase }

procedure TMyTestCase.MultiClientTest;
begin
  ClientTest(20,50);
end;

procedure TMyTestCase.SingleClientTest;
var
  HttpClient: TMyHttpClient;
  test: TTest;
  i: integer;
const
  MAX = 1000;
begin
  HttpClient:=TMyHttpClient.Create('localhost', HTTP_PORT);
  try
    Check(HttpClient.SetUser('Admin', 'synopse'));
    test:=TTest.Create;
    try
      for i:=0 to MAX-1 do begin
        test.Description:=FormatUTF8('test-%',[i]);
        Check(HttpClient.MyCommand.Add(test)=cqrsSuccess);
      end;
      Check(HttpClient.MyCommand.Commit=cqrsSuccess);
    finally
      test.Free;
    end;
  finally
    HttpClient.Free;
  end;
end;

function TMyTestCase.ClientTest(const aClients, aRequests: integer):boolean;
var
  i: integer;
  arrThreads: array of TMyThread;
  arrHandles: array of THandle;
  rWait: Cardinal;
begin
  result := false;
  SetLength(arrThreads, aClients);
  SetLength(arrHandles, aClients);
  for i:=Low(arrThreads) to High(arrThreads) do
  begin
    arrThreads[i]:=TMyThread.Create(i,aRequests);
    arrHandles[i]:=arrThreads[i].Handle;
    arrThreads[i].Resume;
  end;
  try
    repeat
      rWait:= WaitForMultipleObjects(aClients, @arrHandles[0], True, INFINITE);
    until rWait<>WAIT_TIMEOUT;
  finally
    for i:=Low(arrThreads) to High(arrThreads) do
    begin
      CheckNot(arrThreads[i].IsError);
      arrThreads[i].Free;
    end;
    SetLength(arrThreads, 0);
    SetLength(arrHandles, 0);
  end;
end;

{ TMyHttpClient }

constructor TMyHttpClient.Create(const aServer,aPort: RawUTF8);
begin
  fModel:=TSQLModel.Create([TSQLRecordTest]);
  inherited Create(aServer, aPort, fModel);
end;

destructor TMyHttpClient.Destroy;
begin
  fMyCommand:=nil;
  inherited;
  fModel.Free;
end;

function TMyHttpClient.SetUser(const aUserName, aPassword: RawUTF8; aHashedPassword: Boolean=false): boolean;
begin
  result := inherited SetUser(aUserName, aPassword, aHashedPassword);
  if result then
  begin
    TInterfaceFactory.RegisterInterfaces([TypeInfo(IMyQuery),TypeInfo(IMyCommand)]);
    ServiceDefine([IMyCommand],sicClientDriven);
    Services.Resolve(IMyCommand, fMyCommand);
  end;
end;


{ TMyThread }

constructor TMyThread.Create(const aID, aRequestCount: integer);
begin
  inherited Create(true);
  fRequestCount:=aRequestCount;
  fId:=aId;
  fIsError:=false;
  fHttpClient := TMyHttpClient.Create('localhost', HTTP_PORT);
  fHttpClient.SetUser('Admin', 'synopse');
end;

destructor TMyThread.Destroy;
begin
  fHttpClient.Free;
  inherited;
end;

procedure TMyThread.Execute;
var
  i: integer;
  test: TTest;
  success: boolean;
begin
  test:=TTest.Create;
  try
    success:=true;
    for i:=0 to fRequestCount-1 do begin
      test.Description:=FormatUTF8('test-%-%',[fID, i]);
      success:=success and (fHttpClient.MyCommand.Add(test)=cqrsSuccess);
      if not success then
        break;
    end;
    if success then
      success:=fHttpClient.MyCommand.Commit=cqrsSuccess;
    if not success then
    begin
      fIsError:=true;
      raise Exception.Create('Something went wrong!');
    end;
  finally
    test.Free;
  end;
end;

begin
  with TMyTests.Create('mORMot DDD Test') do
  try
    Run;
  finally
    Free;
  end;
  WriteLn(#13#10'Done - Press ENTER to Exit');
  ReadLn;
end.

#94 Re: mORMot 1 » Possibly a Serious bug in mORMot's DDD functionality! » 2015-09-08 10:24:17

oz
danielkuettner wrote:

I would first split server and client project and then test again.

I've already thought about that, but:
-It would make the automated test suite much more complicated. A seperate server process has to be compiled, started, managed and stopped by the test. Server exceptions will happen in another process, ...
-The mORMot framework should be able to pass such tests. I don't see any reasons why this shouldn't work inside a single process. There already are other tests dealing with concurrent access to the server in the test suite. These test do pass, so should this one. mORMot DDD Layer doesn't introduce any kind of magic.. it's based on well known mORMot bricks, the composition of those bricks simply should work.

I think you made this proposal because hosting multiple clients instances inside server process could/will slow down response times. But the same thing -slow down because of heavy load- could happen in production use too. mORMot is a fantastic framework with great performance and should be able to handle such usage without any problem.

#95 Re: mORMot 1 » Possibly a Serious bug in mORMot's DDD functionality! » 2015-09-08 09:48:23

oz

It's me again smile
I've just read the link you provided. I think I get the point, but i don't see any connection between this problem and the one from the other thread. There is no "playing around" with transactions in the test project. It is based on mORMot's standard DDD implementation, no custom transaction or batch processing is done here.

#96 Re: mORMot 1 » Possibly a Serious bug in mORMot's DDD functionality! » 2015-09-08 09:34:30

oz

Hi Arnaud,
thanks for your quick reply. I've just downloaded and installed the latest sqlite3.obj and sqlite3fts.obj files, but it didn't help. The problem still exists. I'll check your link in a minute...

#97 mORMot 1 » Possibly a Serious bug in mORMot's DDD functionality! » 2015-09-08 08:30:09

oz
Replies: 11

Hi Arnaud,
I'm currently working on a mORMot based server using it's DDD (doman driven design) functionality.
While creating a TestCase for concurrent client access to the server, i've faced a serious problem.
I created a Test project for reproducing the bug by extracting all the relevant parts of my sources. The Test is attached as a single, compileable .DPR file.
Delphi 7 is used for compiling.

The problem is in TMyTestCase.MultiClientTest.
When using a single client instance, everything works just fine. But as soon as there are at least 2 client instances, things start getting wrong. At some point a "ESQLite3Exception ("Error SQLITE_ERROR (1) using 3.8.8.1 - 'cannot rollback - no transaction is active' extended_errcode=1")" is raised when calling TTestRest.Commit.

Just run the attached program and see what is happing.

program mORMotBug;

{$I Synopse.inc} // define HASINLINE USETYPEINFO CPU32 CPU64 OWNNORMTOUPPER

{$APPTYPE CONSOLE}

{$R *.res}

uses
  FastMM4,
  SysUtils,
  Classes,
  Windows,
  SynSqlite3Static,
  SynSqlite3,
  SynCommons,
  SynLog,
  SynTests,
  mORMot,
  mORMotHttpServer,
  mORMotHttpClient,
  mORMotSqlite3,
  mORMotDDD;

const
  HTTP_PORT = '80';

type
  // This is our simple Test data class. Will be mapped to TSQLRecordTest.
  TTest = class(TSynPersistent)
  private
    fDescription: RawUTF8;
  published
    property Description: RawUTF8 read fDescription write fDescription;
  end;
  TTestObjArray = array of TTest;

  // The corresponding TSQLRecord for TTest.
  TSQLRecordTest = class(TSQLRecord)
  private
    fDescription: RawUTF8;
  published
    property Description: RawUTF8 read fDescription write fDescription;
  end;

  // CQRS Query Interface fo TTest
  IMyQuery = interface(ICQRSService)
    ['{DD402806-39C2-4921-98AA-A575DD1117D6}']
    function SelectByDescription(const aDescription: RawUTF8): TCQRSResult;
    function SelectAll: TCQRSResult;
    function Get(out aAggregate: TTest): TCQRSResult;
    function GetAll(out aAggregates: TTestObjArray): TCQRSResult;
    function GetNext(out aAggregate: TTest): TCQRSResult;
    function GetCount: integer;
  end;

  // CQRS Command Interface for TTest
  IMyCommand = interface(IMyQuery)
    ['{F0E4C64C-B43A-491B-85E9-FD136843BFCB}']
    function Add(const aAggregate: TTest): TCQRSResult;
    function Update(const aUpdatedAggregate: TTest): TCQRSResult;
    function Delete: TCQRSResult;
    function DeleteAll: TCQRSResult;
    function Commit: TCQRSResult;
    function Rollback: TCQRSResult;
  end;

  // The infratructure REST class implementing the Query and Command Interfaces for TTest
  TTestRest = class(TDDDRepositoryRestCommand,IMyCommand,IMyQuery)
  public
    function SelectByDescription(const aDescription: RawUTF8): TCQRSResult;
    function SelectAll: TCQRSResult;
    function Get(out aAggregate: TTest): TCQRSResult;
    function GetAll(out aAggregates: TTestObjArray): TCQRSResult;
    function GetNext(out aAggregate: TTest): TCQRSResult;
    function Add(const aAggregate: TTest): TCQRSResult;
    function Update(const aUpdatedAggregate: TTest): TCQRSResult;
  end;

  // REST Factory for TTestRest instances
  TTestRestFactory = class(TDDDRepositoryRestFactory)
  public
    constructor Create(aRest: TSQLRest; aOwner: TDDDRepositoryRestManager=nil); reintroduce;
  end;

  // Test container
  TMyTests = class(TSynTestsLogged)
  published
    procedure MyTests;
  end;

  // Test case doing the actual work
  TMyTestCase = class(TSynTestCase)
  private
    // Rest server
    fRestServer: TSQLRestServerDB;
    // Http server
    fHttpServer: TSQLHttpServer;
    /// Will create as many Clients as specified by aClient.
    // - Each client will perform as many Requests as specified by aRequests.
    // - This function will wait for all Clients until finished.
    function ClientTest(const aClients, aRequests: integer):boolean;
  protected
    // Cleaning up the test
    procedure CleanUp; override;
  published
    // Delete any old Test database on start
    procedure DeleteOldDatabase;
    // Start the whole DDD Server (http and rest)
    procedure StartServer;
    // Test straight-forward access using 1 thread and 1 client
    procedure SingleClientTest;
    // Test concurrent access with multiple clients. This will crash!
    procedure MultiClientTest;
  end;

  // Custom TSQLHttpClient encapsulating the remote IMyCommand interface.
  TMyHttpClient=class(TSQLHttpClient)
  private
    // Internal Model
    fModel: TSQLModel;
    // IMyCommand interface. Will be assigned inside SetUser
    fMyCommand: IMyCommand;
  public
    constructor Create(const aServer,aPort: RawUTF8); //overload;
    destructor Destroy; override;
    function SetUser(const aUserName, aPassword: RawUTF8; aHashedPassword: Boolean=false): boolean; reintroduce;
    property MyCommand: IMyCommand read fMyCommand;
  end;

  // The thread used by TMyTestCase.ClientTest
  TMyThread = class(TThread)
  private
    fHttpClient: TMyHttpClient;
    fRequestCount: integer;
    fId: integer;
    fIsError: boolean;
  protected
    procedure Execute; override;
  public
    constructor Create(const aId, aRequestCount: integer);
    destructor Destroy; override;
    property IsError: boolean read fIsError;
  end;

{ TTestRest }

function TTestRest.SelectByDescription(
  const aDescription: RawUTF8): TCQRSResult;
begin
  result := ORMSelectOne('Description=?',[aDescription],(aDescription=''));
end;

function TTestRest.SelectAll: TCQRSResult;
begin
  result := ORMSelectAll('',[]);
end;

function TTestRest.Get(out aAggregate: TTest): TCQRSResult;
begin
  result := ORMGetAggregate(aAggregate);
end;

function TTestRest.GetAll(
  out aAggregates: TTestObjArray): TCQRSResult;
begin
  result := ORMGetAllAggregates(aAggregates);
end;

function TTestRest.GetNext(out aAggregate: TTest): TCQRSResult;
begin
  result := ORMGetNextAggregate(aAggregate);
end;

function TTestRest.Add(const aAggregate: TTest): TCQRSResult;
begin
  result := ORMAdd(aAggregate);
end;

function TTestRest.Update(
  const aUpdatedAggregate: TTest): TCQRSResult;
begin
  result := ORMUpdate(aUpdatedAggregate);
end;


{ TInfraRepoUserFactory }

constructor TTestRestFactory.Create(aRest: TSQLRest;
  aOwner: TDDDRepositoryRestManager);
begin
  inherited Create(IMyCommand,TTestRest,TTest,aRest,TSQLRecordTest,aOwner);
end;

{ TMyTests }

procedure TMyTests.MyTests;
begin
  AddCase([TMyTestCase]);
end;

{ TMyTestCase }

procedure TMyTestCase.CleanUp;
begin
  if Assigned(fHttpServer) then
    FreeAndNil(fHttpServer);
  if Assigned(fRestServer) then
    FreeAndNil(fRestServer);
end;

procedure TMyTestCase.DeleteOldDatabase;
begin
  if FileExists(ChangeFileExt(ParamStr(0), '.db3')) then
    SysUtils.DeleteFile(ChangeFileExt(ParamStr(0), '.db3'));
  CheckNot(FileExists(ChangeFileExt(ParamStr(0), '.db3')));
end;

procedure TMyTestCase.StartServer;
begin
  fRestServer:=TSQLRestServerDB.CreateWithOwnModel([TSQLRecordTest], ChangeFileExt(ParamStr(0), '.db3'), true);
  with fRestServer do begin
      DB.Synchronous := smNormal;
      DB.LockingMode := lmExclusive;
      CreateMissingTables();
      TInterfaceFactory.RegisterInterfaces([TypeInfo(IMyQuery),TypeInfo(IMyCommand)]);
      ServiceContainer.InjectResolver([TTestRestFactory.Create(fRestServer)],true);
      ServiceDefine(TTestRest,[IMyCommand],sicClientDriven);
    end;
  fHttpServer:=TSQLHttpServer.Create(HTTP_PORT, fRestServer, '+', useHttpApiRegisteringURI);
end;

procedure TMyTestCase.MultiClientTest;
begin
  ClientTest(20,50);
end;

procedure TMyTestCase.SingleClientTest;
var
  HttpClient: TMyHttpClient;
  test: TTest;
  i: integer;
const
  MAX = 1000;
begin
  HttpClient:=TMyHttpClient.Create('localhost', HTTP_PORT);
  try
    Check(HttpClient.SetUser('Admin', 'synopse'));
    test:=TTest.Create;
    try
      for i:=0 to MAX-1 do begin
        test.Description:=FormatUTF8('test-%',[i]);
        Check(HttpClient.MyCommand.Add(test)=cqrsSuccess);
      end;
      Check(HttpClient.MyCommand.Commit=cqrsSuccess);
    finally
      test.Free;
    end;
  finally
    HttpClient.Free;
  end;
end;

function TMyTestCase.ClientTest(const aClients, aRequests: integer):boolean;
var
  i: integer;
  arrThreads: array of TMyThread;
  arrHandles: array of THandle;
  rWait: Cardinal;
begin
  result := false;
  SetLength(arrThreads, aClients);
  SetLength(arrHandles, aClients);
  for i:=Low(arrThreads) to High(arrThreads) do
  begin
    arrThreads[i]:=TMyThread.Create(i,aRequests);
    arrHandles[i]:=arrThreads[i].Handle;
    arrThreads[i].Resume;
  end;
  try
    repeat
      rWait:= WaitForMultipleObjects(aClients, @arrHandles[0], True, INFINITE);
    until rWait<>WAIT_TIMEOUT;
  finally
    for i:=Low(arrThreads) to High(arrThreads) do
    begin
      CheckNot(arrThreads[i].IsError);
      arrThreads[i].Free;
    end;
    SetLength(arrThreads, 0);
    SetLength(arrHandles, 0);
  end;
end;

{ TMyHttpClient }

constructor TMyHttpClient.Create(const aServer,aPort: RawUTF8);
begin
  fModel:=TSQLModel.Create([TSQLRecordTest]);
  inherited Create(aServer, aPort, fModel);
end;

destructor TMyHttpClient.Destroy;
begin
  fMyCommand:=nil;
  inherited;
  fModel.Free;
end;

function TMyHttpClient.SetUser(const aUserName, aPassword: RawUTF8; aHashedPassword: Boolean=false): boolean;
begin
  result := inherited SetUser(aUserName, aPassword, aHashedPassword);
  if result then
  begin
    ServiceDefine([IMyCommand],sicClientDriven);
    Services.Resolve(IMyCommand, fMyCommand);
  end;
end;


{ TMyThread }

constructor TMyThread.Create(const aID, aRequestCount: integer);
begin
  inherited Create(true);
  fRequestCount:=aRequestCount;
  fId:=aId;
  fIsError:=false;
  fHttpClient := TMyHttpClient.Create('localhost', HTTP_PORT);
  fHttpClient.SetUser('Admin', 'synopse');
end;

destructor TMyThread.Destroy;
begin
  fHttpClient.Free;
  inherited;
end;

procedure TMyThread.Execute;
var
  i: integer;
  test: TTest;
  success: boolean;
begin
  test:=TTest.Create;
  try
    success:=true;
    for i:=0 to fRequestCount-1 do begin
      test.Description:=FormatUTF8('test-%-%',[fID, i]);
      success:=success and (fHttpClient.MyCommand.Add(test)=cqrsSuccess);
      if not success then
        break;
    end;
    if success then
      success:=fHttpClient.MyCommand.Commit=cqrsSuccess;
    if not success then
    begin
      fIsError:=true;
      raise Exception.Create('Something went wrong!');
    end;
  finally
    test.Free;
  end;
end;

begin
//  TSynLogTestLog := TSQLLog; // share the same log file with whole mORMot
  TSQLLog.Family.Level := LOG_STACKTRACE; // log errors by default
  TSQLLog.Family.Level := []; // NO log by default (ignore expected ERROR 400)
  with TMyTests.Create('mORMot DDD Test') do
  try
    Run;
  finally
    Free;
  end;
  WriteLn(#13#10'Done - Press ENTER to Exit');
  ReadLn;
end.

Now, the question is: is there something wrong with my test source, or is this a problem in mORMot's DDD functionality?

I'm looking forward to hearing from you!

Board footer

Powered by FluxBB