Posts by DigDiver

DigDiver · mORMot 1

Does anybody implement correct works with mongoDB replicaset?

The problem is that all insert/update TMongoCollection operations calls Database.Client.Connections[0] where index "0" is a primary server. When the primary server is down/reboots the primary server in replicaset will be changed. After that all insert/update operations will fail with the error:

..."connectionId":6,"err":"Not primary while writing to database.ServerStat","code":189,"codeName":"PrimarySteppedDown"...

As a solution, maybe we should run the command "ismaster" to resort Database.Client.Connections periodically or when error occurred?

{
    "hosts" : [ 
        "devdb1:27017", 
        "devdb2:27017", 
        "devdb3:27017"
    ],
    "setName" : "devgapps",
    "setVersion" : 3,
    "ismaster" : false,
    "secondary" : true,
    "primary" : "devdb1:27017",
    "me" : "devdb3:27017",
    "lastWrite" : {
        "opTime" : {
            "ts" : Timestamp(1547122432, 1),
            "t" : NumberLong(1)
        },
        "lastWriteDate" : ISODate("2019-01-10T12:13:52.000Z"),
        "majorityOpTime" : {
            "ts" : Timestamp(1547122432, 1),
            "t" : NumberLong(1)
        },
        "majorityWriteDate" : ISODate("2019-01-10T12:13:52.000Z")
    },
    "maxBsonObjectSize" : 16777216,
    "maxMessageSizeBytes" : 48000000,
    "maxWriteBatchSize" : 100000,
    "localTime" : ISODate("2019-01-10T12:13:54.403Z"),
    "logicalSessionTimeoutMinutes" : 30,
    "minWireVersion" : 0,
    "maxWireVersion" : 6,
    "readOnly" : false,
    "ok" : 1.0,
    "operationTime" : Timestamp(1547122432, 1)
}

One of the problems is internal host names

       "devdb1:27017", 
        "devdb2:27017", 
        "devdb3:27017"

but this may be resolved by adding aliases to TMongoClient.Create('devdb1@192.168.0.1',...'

DigDiver · mORMot 1

Sometime I need to specify the collation for aggregate operation.

For example, change numericOrdering flag that determines whether to compare numeric strings as numbers or as strings.

{locale: "simple", numericOrdering: true}

function TMongoCollection.AggregateCallFromJson(const pipelineJSON: RawUTF8;
  var reply,res: variant; const collation: RawUTF8 = ''): boolean;
begin // see http://docs.mongodb.org/manual/reference/command/aggregate
  if fDatabase.Client.ServerBuildInfoNumber<2020000 then
    raise EMongoException.Create('Aggregation needs MongoDB 2.2 or later');
  if fDatabase.Client.ServerBuildInfoNumber>=3060000 then begin
    // db.runCommand({aggregate:"test",pipeline:[{$group:{_id:null,max:{$max:"$_id"}}}],cursor:{}})
    if collation <> '' then
     Database.RunCommand(BSONVariant('{aggregate:"%",pipeline:[%],cursor:{}, collation:%}',[Name,pipelineJSON, collation],[]),reply)
    else
     Database.RunCommand(BSONVariant('{aggregate:"%",pipeline:[%],cursor:{}}',[Name,pipelineJSON],[]),reply);
    // {"cursor":{"firstBatch":[{"_id":null,"max":1510}],"id":0,"ns":"db.test"},"ok":1}
    res := reply.cursor;
    if not VarIsNull(res) then
      res := res.firstBatch;
  end else begin
    // db.runCommand({aggregate:"test",pipeline:[{$group:{_id:null,max:{$max:"$_id"}}}]})
    Database.RunCommand(BSONVariant('{aggregate:"%",pipeline:[%]}',[Name,pipelineJSON],[]),reply);
    // { "result" : [ { "_id" : null, "max" : 1250 } ], "ok" : 1 }
    res := reply.result;
  end;
  result := not VarIsNull(res);
end;

The code above is working for me.

DigDiver · mORMot 1

Property Text has type shortstring

procedure TJSONSerializer.WriteObject(Value: TObject; Options: TTextWriterWriteObjectOptions);
...
  procedure WriteProp(P: PPropInfo);
...
    Kind := P^.PropType^.Kind;  <- for property "Text" Kind is tkString. And tkString is not handled in the code below
    case Kind of

DigDiver · mORMot 1

In the previous version the StatsAsJson method returns:

    "LastTime": {
      "MicroSec": 2644,
      "Text": "2.64ms"
    },
    "MinimalTime": {
      "MicroSec": 7,
      "Text": "7us"
    },
    "AverageTime": {
      "MicroSec": 18995,
      "Text": "18.99ms"
    },

In the new version (missing "Text"):

    "LastTime": {
      "MicroSec": 58
    },
    "MinimalTime": {
      "MicroSec": 11
    },
    "AverageTime": {
      "MicroSec": 1054
    },
    "MaximalTime": {
      "MicroSec": 121127
    },...

DigDiver · mORMot 1

ab wrote:

IIRC j2oIgnoreUnknownProperty has nothing to do with TSQLRecord ORM fields: it is about TDocVariant serialization.
What is the exact problem/exception raised with such an old DB?
I supposed old fields are just ignored by TSQLRecord.Fill.

The TSQLRecord.Fill does not ignored unknown fields and deserialization stops on unknown fields.

DigDiver · mORMot 1

Nobody needs?

DigDiver · mORMot 1

In some cases, it would by nice to have an overload function TSQLRecord.FillOne(Options: TJSONToObjectOptions=[]);

The problem is that I try to connect to the mongoDB server with a "very old" documents. At the moment not all fields in the collection are actual, but since the collection stores old values, I need to create too many dummy fields in my TSQLRecord class,
so adding [j2oIgnoreUnknownProperty] to FillOne call would solve the problem. What do you think?

DigDiver · mORMot 1

Hello to All,

How can I keep a server's statics in the form of a time series? Current stat I can get via TSQLRestServer.StatsAsJson method.

Does anyone have an example of use of TSynMonitorUsageRest? Or I need to use something else for this purpose?

DigDiver · mORMot 1

Thanks, fix works fine.

DigDiver · mORMot 1

Hi, in the new version after the TTextWriter.Text is executed and then new data is added, the old data will be broken. Is it normal?

 Writer := TTextWriter.CreateOwnedStream();

 Writer.AddString('One-');
 Writer.AddString('Two-');
 Writer.AddString('Free-');

 Writer.FlushFinal;

 t1:= Writer.Text; // One-Two-Free-

 Writer.AddString('Ones more');

 Writer.FlushFinal;

 t2 := Writer.Text; // #0#0#0#0#0#0#0#0#0#0#0#0#0'Ones more'

DigDiver · mORMot 1

The TSystemTime record in Winapi.Windows defained as:

  _SYSTEMTIME = record
    wYear: Word;
    wMonth: Word;
    wDayOfWeek: Word;
    wDay: Word;
    wHour: Word;
    wMinute: Word;
    wSecond: Word;
    wMilliseconds: Word;
  end;

The TSynSystemTime defained as:

  TSynSystemTime = {$ifdef ISDELPHI2006ANDUP}record{$else}object{$endif}
    Year, Month, Day, DayOfWeek,
    Hour, Minute, Second, MilliSecond: word;

See the wrong order of Day and DayOfWeek in TSynSystemTime declaration

So function FromNowUTC returns 3/5/2018 11:17:25 AM instead of 3/23/2018 11:17:25 AM

procedure TSynSystemTime.FromNowUTC;
begin
  {$ifdef MSWINDOWS}
  GetSystemTime(TSystemTime(self)); // this API is fast enough for our purpose
  {$else}
  GetNowUTCSystem(TSystemTime(self));
  {$endif}
end;

DigDiver · mORMot 1

I found topic https://synopse.info/forum/viewtopic.php?id=953,
but unable to find implementation of THttpApiWebSocketServer.

DigDiver · mORMot 1

Scalable HTTP/S and TCP client sockets for the cloud
https://github.com/grijjy/DelphiScalableClientSockets

DigDiver · mORMot 1

Each method is already executed in its own thread, from the http server thread pool, so I don't see why creating a new thread may help here

There is no need to create a new thread. If you pass a reference to a class, then create a thread inherited from TSynThread instead of plain TSynThread, by default create TSynThread as it is.

So if you want to re-use some existing resource, create a pool of resources.

Now it is implemented in this way. Only in this case the another thread pool is used.

DigDiver · mORMot 1

I suggest adding the ability to create threadPool for TSQLHttpServer with custom ThreadClass.

Here is an example of pseudocode:

  Type
   TAPIServer = class(TSQLRestServerFullMemory)
  ...
    published  procedure ServerMethod(Ctxt: TSQLRestServerURIContext); 
   ...
   
  procedure TAPIServer.ServerMethod(Ctxt: TSQLRestServerURIContext);
  var
   Obj1 : TSomeObject;
   Obj2 : TOtherObject;
   ...
  begin
   Obj1 := TSomeObject.Create;
   Obj2 := TOtherObject.Create;
   try
    // Heavy creation
    ...
    // Some calculations
    ...
    // Network functions


    Ctxt.Returns(AnyValue); 

   finally
    Obj1.free;
    Obj2.free;
   end;
  end;

In this example, for each call ServerMethod some amount of objects is created and destroyed.
Objects initialization can be costly both in time and in CPU, eg compiling a large number of regular expressions,
creating hash tables and so on. And all this is destroyed after the method is executed.

You can of course create a separate class, and call its methods through CriticalSection, but
with long calculations it will greatly slow down the server.

How to improve:

type TThreadClass = class of TSynThread;
 ..
TSQLHttpServer.Create(const aPort: AnsiString; aServer: TSQLRestServer;
      const aDomainName: AnsiString='+';
      aHttpServerKind: TSQLHttpServerOptions=HTTP_DEFAULT_MODE; aRestAccessRights: PSQLAccessRights=nil;
      ServerThreadPoolCount: Integer=32; aHttpServerSecurity: TSQLHttpServerSecurity=secNone;
      const aAdditionalURL: AnsiString=''; const aQueueName: SynUnicode=''; ThreadClass: TThreadClass);

The TSQLHttpServer constructor is called with a class reference ThreadClass.
When TSQLHttpServer is cloned, it creates threads with our class.

type 
 TMyThreadClass = class(TSynThread)      
 ..
 end;

 consturctor TMyThreadClass.Create()
 var
   Obj1 : TSomeObject;
   Obj2 : TOtherObject;
   ...
  begin
   Obj1 := TSomeObject.Create;
   Obj2 := TOtherObject.Create;
   // Heavy creation
  end; 
..

 HttpServer  :=  TSQLHttpServer.Create(Port, [APIServer], '+',  useHttpApi, 32, false, TMyThreadClass);

In the ServerMethod we get the pointer on Thread, which currently serves THttpServerRequest
and call the thread method:

 procedure TAPIServer.ServerMethod(Ctxt: TSQLRestServerURIContext);
 var
  MyTh : TMyThreadClass;
 begin
  MyTh := Ctxt.Thread as TMyThreadClass;
  MyTh.calculation(Ctxt.inputUtf8['value1']);
  Ctxt.Returns(MyTh.CalculatedValue);
 end;

In this case, we noticeably speed up the processing of requests and reduce the load on the memory and CPU.

What do you think?

DigDiver · mORMot 1

What is the best way to forward a request to other server? I have server method. When a request is arrives, I need to do something with data and if necessary forward the original request (all headers, body) to the third party server.

DigDiver · mORMot 1

The problem is that j2oIgnoreUnknownProperty option do not work in the record de-serialize.

DigDiver · mORMot 1

Is anyone aware of the existence of the SMTP server built on the IOCP basis? I tried different versions of SMTP servers (ICS, INDY), but they have low performance. In some conditions the ICS hangs and stops receiving the messages.
INDY uses one thread per connection - it takes too much memory.
The web part of the project uses mORMot and works very well.
But I also need to process incoming email messages.

DigDiver · mORMot 1

procedure TSystemUse.BackgroundExecute
...
difftot := (fSysPrevKernel-skrn)+(fSysPrevUser-susr); // In my case the difftot < 0 
...
   if difftot>0 then begin                                         //  when I try to use:  if difftot<>0 then begin    CPU usage is correct              
                Kernel := diffkrn*100/difftot;
                User := diffusr*100/difftot;
              end else begin
                Kernel := 0;
                User := 0;
              end;

DigDiver · mORMot 1

I found that call.RestAccessRights is always NIL when I try to use any method via TSQLRestClientRedirect. Maybe I need to make any adjustment in ExistingServer?

DigDiver · mORMot 1

Thanks, ab, it works for me too.

DigDiver · mORMot 1

Client := TSQLRestClientRedirect.Create(ExistingServer);

Client.SetUser() does not work, because Ctxt.Call^.RestAccessRights is nil in:

procedure TSQLRestServer.SessionCreate(var User: TSQLAuthUser;
  Ctxt: TSQLRestServerURIContext; out Session: TAuthSession);
var i: integer;
begin
  Session := nil;
  if (reOneSessionPerUser in Ctxt.Call^.RestAccessRights^.AllowRemoteExecute) and
     (fSessions<>nil) then

DigDiver · mORMot 1

Try with this demo project.
With mORMotMongoDB-old.pas all works. But with the original mORMotMongoDB.pas the memory corrupt errors occurs.
To test I'm use JMeter Bounce API-test.jmx in Thread Group I'm set Number of threads(users) to 30 and Loop count to 10
https://www.dropbox.com/s/e6zez582qhjts … o.zip?dl=0

DigDiver · mORMot 1

Is there any progress?

DigDiver · mORMot 1

It does not help.

Event Thread ID Time
First chance exception 0xC0000005 ACCESS_VIOLATION occurred at 0x00405C19, write of address 0x000004FE at 0x00405C19 (in D:\Project XE10\InboxMonitor\Win32\Debug\inboxmonitor.exe) 3108  15:29:42:455
0x00405C19 System::SysFreeMem + 0x81 in GETMEM.INC line 2996 in inboxmonitor.exe  3108  15:29:42:455
0x00406FF6 System::FreeMem + 0xA in System.pas line 4655 in inboxmonitor.exe  3108  15:29:42:455
0x0040DE68 System::DynArrayClear + 0x40 in System.pas line 34689 in inboxmonitor.exe  3108  15:29:42:455
0x0040D0F6 System::FinalizeArray + 0xFA in System.pas line 31964 in inboxmonitor.exe  3108  15:29:42:455
0x0040CFD5 System::FinalizeRecord + 0x29 in System.pas line 31638 in inboxmonitor.exe 3108  15:29:42:455
0x004098C3 System::TObject::CleanupInstance + 0x17 in System.pas line 16399 in inboxmonitor.exe 3108  15:29:42:455
0x004096C2 System::TObject::FreeInstance + 0xA in System.pas line 16193 in inboxmonitor.exe 3108  15:29:42:455
0x00409F01 System::ClassDestroy + 0x5 in System.pas line 17543 in inboxmonitor.exe  3108  15:29:42:455
0x006ABFA0 mORMot::TSQLTable::~TSQLTable + 0x3C in mORMot.pas line 25219 in inboxmonitor.exe  3108  15:29:42:455
0x0040984B System::TObject::Free + 0xB in System.pas line 16263 in inboxmonitor.exe 3108  15:29:42:455
0x006B606A mORMot::TSQLRecordFill::UnMap + 0xE6 in mORMot.pas line 29365 in inboxmonitor.exe  3108  15:29:42:455
0x006B5B73 mORMot::TSQLRecordFill::~TSQLRecordFill + 0x27 in mORMot.pas line 29261 in inboxmonitor.exe  3108  15:29:42:455
0x0040984B System::TObject::Free + 0xB in System.pas line 16263 in inboxmonitor.exe 3108  15:29:42:455
0x006B9A01 mORMot::TSQLRecord::~TSQLRecord + 0xB5 in mORMot.pas line 30346 in inboxmonitor.exe  3108  15:29:42:455
0x008F9CB1 User_Model::TGlockUser::~TGlockUser + 0x35 in User_Model.pas line 228 in inboxmonitor.exe  3108  15:29:42:455
0x0040984B System::TObject::Free + 0xB in System.pas line 16263 in inboxmonitor.exe 3108  15:29:42:455
0x009B2802 FBLServer_mongo::TBounceServer::CheckAPiUser + 0x14A in FBLServer_mongo.pas line 2336 in inboxmonitor.exe  3108  15:29:42:456

 First chance exception 0x0EEDFADE Delphi exception occurred at 0x75995B68 (class: EBSONException, message: "Incorrect supplied BSON document content")  8696  15:29:46:415
      0x75995B68 RaiseException + 0x48 in KernelBase.dll  8696  15:29:46:415
      0x00992C00 SynMongoDB::BSONParseLength + 0x48 in SynMongoDB.pas line 2917 in inboxmonitor.exe 8696  15:29:46:415
      0x00992FD6 SynMongoDB::BSONToJSON + 0x1A in SynMongoDB.pas line 3002 in inboxmonitor.exe  8696  15:29:46:415
      0x0099A8EA SynMongoDB::TMongoConnection::GetReply + 0x18E in SynMongoDB.pas line 5147 in inboxmonitor.exe 8696  15:29:46:415
      0x0099A627 SynMongoDB::TMongoConnection::GetCursor + 0x3B in SynMongoDB.pas line 5113 in inboxmonitor.exe 8696  15:29:46:415
      0x00999D06 SynMongoDB::TMongoConnection::GetRepliesAndFree + 0xC6 in SynMongoDB.pas line 4993 in inboxmonitor.exe 8696  15:29:46:415
      0x00999A5A SynMongoDB::TMongoConnection::GetBSONAndFree + 0x56 in SynMongoDB.pas line 4946 in inboxmonitor.exe  8696  15:29:46:415
      0x0099F4AD SynMongoDB::TMongoCollection::FindBSON + 0x4D in SynMongoDB.pas line 5894 in inboxmonitor.exe  8696  15:29:46:415
      0x009A592B mORMotMongoDB::TSQLRestStorageMongoDB::EngineList + 0x303 in mORMotMongoDB.pas line 1294 in inboxmonitor.exe 8696  15:29:46:415
      0x006E0144 mORMot::TSQLRestServer::EngineList + 0x8C in mORMot.pas line 40522 in inboxmonitor.exe 8696  15:29:46:416
      0x006C6D4A mORMot::TSQLRest::ExecuteList + 0x36 in mORMot.pas line 33981 in inboxmonitor.exe  8696  15:29:46:416
      0x006C3E48 mORMot::TSQLRest::MultiFieldValues + 0x64 in mORMot.pas line 33081 in inboxmonitor.exe 8696  15:29:46:416
      0x006B9C4B mORMot::TSQLRecord::TSQLRecord + 0x43 in mORMot.pas line 30380 in inboxmonitor.exe 8696  15:29:46:416
      0x009B4ADD FBLServer_mongo::TBounceServer::GetStats + 0x921 in FBLServer_mongo.pas line 2758 in inboxmonitor.exe  8696  15:29:46:416

one more:

First chance exception 0xC0000005 ACCESS_VIOLATION occurred at 0x00405C19, write of address 0x30543931 at 0x00405C19 (in D:\Project XE10\InboxMonitor\Win32\Debug\inboxmonitor.exe) 2940  15:29:46:391
      0x00405C19 System::SysFreeMem + 0x81 in GETMEM.INC line 2996 in inboxmonitor.exe  2940  15:29:46:391
      0x00406FF6 System::FreeMem + 0xA in System.pas line 4655 in inboxmonitor.exe  2940  15:29:46:391
      0x0040B681 System::LStrClr + 0x21 in System.pas line 24722 in inboxmonitor.exe  2940  15:29:46:391
      0x0099A23D SynMongoDB::TMongoConnection::Send + 0x175 in SynMongoDB.pas line 5072 in inboxmonitor.exe 2940  15:29:46:391
      0x0099A7E8 SynMongoDB::TMongoConnection::GetReply + 0x8C in SynMongoDB.pas line 5132 in inboxmonitor.exe  2940  15:29:46:391
      0x0099A627 SynMongoDB::TMongoConnection::GetCursor + 0x3B in SynMongoDB.pas line 5113 in inboxmonitor.exe 2940  15:29:46:391
      0x00999D06 SynMongoDB::TMongoConnection::GetRepliesAndFree + 0xC6 in SynMongoDB.pas line 4993 in inboxmonitor.exe 2940  15:29:46:391
      0x00999A5A SynMongoDB::TMongoConnection::GetBSONAndFree + 0x56 in SynMongoDB.pas line 4946 in inboxmonitor.exe  2940  15:29:46:391
      0x0099F4AD SynMongoDB::TMongoCollection::FindBSON + 0x4D in SynMongoDB.pas line 5894 in inboxmonitor.exe  2940  15:29:46:391
      0x009A592B mORMotMongoDB::TSQLRestStorageMongoDB::EngineList + 0x303 in mORMotMongoDB.pas line 1294 in inboxmonitor.exe 2940  15:29:46:391
      0x006E0144 mORMot::TSQLRestServer::EngineList + 0x8C in mORMot.pas line 40522 in inboxmonitor.exe 2940  15:29:46:391
      0x006C6DBF mORMot::TSQLRest::ExecuteJson + 0x23 in mORMot.pas line 33989 in inboxmonitor.exe  2940  15:29:46:391
      0x009B1642 FBLServer_mongo::TBounceServer::DownloadList + 0x13D6 in FBLServer_mongo.pas line 2153 in inboxmonitor.exe

DigDiver · mORMot 1

With the latest mORMot, I also have strange problems with mORMotMongoDB. Sometimes, I get AV in module mORMot.pas in the destructor TSQLTable.Destroy, when I make a stress test of my service that uses MongoDB using JMeter. When I revert mORMotMongoDB.pas to the version from May 4, strange errors are gone.

DigDiver · mORMot 1

When Object contains a packed record with the property of TDateTime type, then the property is stored without "Z". The property with TDateTime type inside the object is stored with "Z".

 Type
  TSettings = packed record
   processed   : Int64;
   startDay    : TDateTime;
 end;

 Type
  TUser = class(TSynPersistent)
 private
  FSettings  : Tsettings;
  flastLogin : TDateTime;
 published
  property Settings    : TSettings   read FSettings    write FSetings;
  property lastLogin   : TDateTime read flastLogin    write flastLogin;
end;

{
	"lastLogin": "2016-05-20T13:40:58Z",
	"Settings": {
		"processed": 73,
		"startDay": "2016-04-05T06:30:23"
	}
}

DigDiver · mORMot 1

Thanks! Works fine.

DigDiver · mORMot 1

Since the property Stats is declared as "readonly"

property Stats: TSQLRestServerMonitor read fStats;

how to clear stats on a working server?

I want to collect the server stats grouped by time to build the server stats timeline.

For example, I want to save the server stats to DB each minute (or hour), after that clear the stats to begin collecting new counters and so on.

DigDiver · mORMot 1

igors233 wrote:

Just and idea, have you called CreateMissingTables for your ORM server?

Yes, of course

DigDiver · mORMot 1

There is a big problem with using MongoDB when adding a new property into the model.

For example, we have a model like this:

 Type
  TDownloads = class(TSQLRecord)
    private
     FUserID         : Int64;
     FManual         : Boolean;
     FFileName       : RawUTF8;
     FSignedURL      : RawUTF8;
     FCreated        : TDateTime;
     FExpirationDate : TDateTime;
     FStatus         : RawUTF8;
    published
     property UserID         : Int64      read FUserID         write FUserID;
     property Manual         : Boolean    read FManual         write FManual;
     property FileName       : RawUTF8    read FFileName       write FFileName;
     property SignedURL      : RawUTF8    read FSignedURL      write FSignedURL;
     property Created        : TDateTime  read FCreated        write FCreated;
     property ExpirationDate : TDateTime  read FExpirationDate write FExpirationDate;
     property Status         : RawUTF8    read FStatus         write FStatus;
  end;

We add some records to the DB. In the some situations, I need to expand the model (add a new property). In my example - property Success : Boolean read FSuccess write FSuccess;

 Type
  TDownloads = class(TSQLRecord)
    private
     FUserID         : Int64;
     FManual         : Boolean;
     FFileName       : RawUTF8;
     FSignedURL      : RawUTF8;
     FCreated        : TDateTime;
     FExpirationDate : TDateTime;
     FStatus         : RawUTF8;
     FSuccess        : Boolean;
    published
     property UserID         : Int64      read FUserID         write FUserID;
     property Manual         : Boolean    read FManual         write FManual;
     property FileName       : RawUTF8    read FFileName       write FFileName;
     property SignedURL      : RawUTF8    read FSignedURL      write FSignedURL;
     property Created        : TDateTime  read FCreated        write FCreated;
     property ExpirationDate : TDateTime  read FExpirationDate write FExpirationDate;
     property Status         : RawUTF8    read FStatus         write FStatus;
     property Success        : Boolean    read FSuccess        write FSuccess;
  end;

And after this modification, I cannot get records from DB at all.

In the function TSQLRestStorageMongoDB.GetJSONValues the exception missing column - count=% expected:% is raised:

      if col<>colCount then
        raise EORMMongoDBException.CreateUTF8(
          '%.GetJSONValues(%): missing column - count=% expected:%',
          [self,StoredClass,col,colCount]);

When I use SQLite instead of MongoDB, there is no such a problem.

DigDiver · mORMot 1

How to pass TRawUTF8DynArray as a parameter into the CreateAndFillPrepare method?

When I use this code:

json := RawUTF8ArrayToQuotedCSV(TestList);
TestItem := TTestItem.CreateAndFillPrepare(TestServer, 'UserID=? and TestID in ('+json+')',[ID]);

In the server stats too many items like this:

SELECT Fields FROM TestItem WHERE UserID=? and TestID in ('26251f02bf2b21621e5297280109405e','9e138417d45e3861d2e1ced1f1aecd98','85972f9771f0d101e7d23345de9091ca','9e72e03729466b547d2eb0df3284e65e','4c75495f5c3594366bb81db8cd92430a','a42e97f22fcf092a2d90a0dfa35f7a5c','b4ad5bd3207cdfab4292be0fd9e6078d','aa73aa046427e8e4579dcb40677a3b26','d1cf8af213338a6a9f4a20f39be61199','e006e51d831d310e5d9d4a0b82dffb67'): {...

I wold like to see

SELECT Fields FROM TestItem WHERE UserID=? and TestID in (?)

DigDiver · mORMot 1

How to perform sorting of the results of a query using the methods Find* (FindJSON, FindDocs) of the TMongoCollection class?

It's possible to use Server.ExecuteJson([TSQLRecord, SQL]) method, which transmits $orderby by the $query operator, but these operators are deprecated.

Deprecated since version 3.2:

$query operator.
The $query operator is deprecated. Use the cursor methods to apply options to a cursor.
$orderby operator.
The $orderby operator is deprecated. Use cursor.sort() instead.

DigDiver · mORMot 1

Please download the test project. https://www.dropbox.com/s/pmuwgvr3za72z … s.zip?dl=0
Compile it and run.
Then quickly click two or more times on the button "Call Method".
the program goes into an infinite loop in the function TSynBackgroundThreadMethodAbstract.RunAndWait
AcquireThread - returns flagFinished

DigDiver · mORMot 1

hnb wrote:

"one more request" is the real problem in my production :\. In "real" environment with many WiFi it occurs useless very often (note this is not related to BackgroundThread )...

If SettingsClient.OnIdle is not assigned, any request via SettingsClient will block the User Interface, and the user cannot perform any other actions.
But when SettingsClient.OnIdle is assigned, the UI will be active during the execution of the remote request, and the user can perform any other actions which can produce the infinite loop in the BackgroundThread.
Therefore it is necessary to check OnIdleBackgroundThreadActive

DigDiver · mORMot 1

The problem is that when Client executes a request to the slow server, the BackgroundThread is in active state. And when BackgroundThread executes a request (waiting for the server's response), the client can perform one more request to the server. And at this point, the problem occurs.
I found the solution:

 if dm.SettingsClient.OnIdleBackgroundThreadActive then exit;

or better:

 while dm.SettingsClient.OnIdleBackgroundThreadActive do
  begin
   sleep(30);
   Application.ProcessMessages;
  end;

DigDiver · mORMot 1

But in the real application, it works (flagFinished).
What is the best solution?

DigDiver · mORMot 1

The problem comes back with [f539bea2f8] check-in

In my code I often use Client.UpdateFromServer, that sends two requests to the server with a small time interval between the requests.

 if Assigned(js_accounts) then
   begin
    if not dm.SettingsClient.UpdateFromServer([js_accounts], FRefreshed) then
     if dm.SettingsClient.LastErrorCode <> 503 then
      ShowLastClientError(dm.SettingsClient);
    end

And software goes into an infinite loop in the function TSynBackgroundThreadMethodAbstract.RunAndWait.
AcquireThread - returns flagFinished

DigDiver · mORMot 1

Thanks, now works fine!

DigDiver · mORMot 1

I think, that after Check-in [361888727f] the problem with the executing remote request in a background thread (Client.OnIdle := OnIdleCallback) began.

1. For some unknown reason, the program often goes into an infinite loop in the function TSynBackgroundThreadMethodAbstract.RunAndWait

AcquireThread - returns flagFinished

function TSynBackgroundThreadMethodAbstract.RunAndWait(OpaqueParam: pointer): boolean;
..
  repeat
    case AcquireThread of
    flagDestroying: exit;
    flagIdle: break; // we acquired the background thread
    end;
    case OnIdleProcessNotify(start) of // Windows.GetTickCount64 res is 10-16 ms
    0..20:    SleepHiRes(0);
    21..100:  SleepHiRes(1);
    101..900: SleepHiRes(5);
    else      SleepHiRes(50);
    end;
  until false;

2. Or in an infinite loop in the procedure TSynBackgroundThreadMethodAbstract.WaitForFinished(start: Int64)

line: while FixedWaitFor(fCallerEvent,100)<>wrSignaled do
OnIdleProcessNotify(start);

procedure TSynBackgroundThreadMethodAbstract.WaitForFinished(start: Int64);
...
    if Assigned(fOnIdle) then begin
      while FixedWaitFor(fCallerEvent,100)<>wrSignaled do
        OnIdleProcessNotify(start);

DigDiver · mORMot 1

There is a problem with the synchronization start.

Steps to reproduce the problem:

1. Starting MasterServer
2. MasterServer.RecordVersionSynchronizeMasterStart();

3. Starting SlaveServer
4. SlaveServer.RecordVersionSynchronizeSlaveStart(TUser, MasterClient, NotifySync);

In this case RecordVersionSynchronizeSlaveStart returns false (retry failure)

If I start MasterServer, then make changes to TSQLRecordPeopleVersioned, and after that start SlaveServer, synchronization will start working.

1. Starting MasterServer
2. MasterServer.RecordVersionSynchronizeMasterStart();

3. Make any changes in the TSQLRecordPeopleVersioned

4. Starting SlaveServer
5. SlaveServer.RecordVersionSynchronizeSlaveStart(TUser, MasterClient, NotifySync);

Below is the log for the scenario 1:

MasterServer

20160311 13074854  +    mORMotSQLite3.TSQLRestServerDB(07316160).URI(POST Accounts/ServiceRecordVersion._contract_?session_signature=017E1BAE00017E2E93B827B4 inlen=2)
20160311 13074854 auth  	mORMot.TSQLRestRoutingREST(072D0800) SuperPuperAdmin/25041838 
20160311 13074854 call  	mORMotSQLite3.TSQLRestServerDB(07316160) IServiceRecordVersion._contract_[]
20160311 13074854 srvr  	mORMotSQLite3.TSQLRestServerDB(07316160) SuperPuperAdmin  POST Accounts/ServiceRecordVersion._contract_ SOA-Interface -> 200 with outlen=31 in 342 us
20160311 13074854 ret   	mORMotSQLite3.TSQLRestServerDB(07316160) {"result":["5DDC975F57031DB5"]}

20160311 13074854  +    mORMotSQLite3.TSQLRestServerDB(07316160).URI(GET Accounts?session_signature=017E1BAE00017E2E0BB76508 inlen=121)
20160311 13074854 auth  	mORMot.TSQLRestRoutingREST(072D0C20) SuperPuperAdmin/25041838 
20160311 13074854 DB    	mORMotSQLite3.TSQLRestServerDB(07316160) prepared 35us accounts.db3 SELECT ID,Deleted FROM TableDeleted WHERE ID>? and ID<? order by ID limit 10000
20160311 13074854 SQL   	mORMotSQLite3.TSQLRestServerDB(07316160) 366us returned 0 row as 56 bytes SELECT ID,Deleted FROM TableDeleted WHERE ID>:(576460752303423557): and ID<:(864691128455135232): order by ID limit 10000
20160311 13074854 res   	SynSQLite3.TSQLDatabase(04263628) {"fieldCount":2,"values":["ID","Deleted"],"rowCount":0} 
20160311 13074854 srvr  	mORMotSQLite3.TSQLRestServerDB(07316160) SuperPuperAdmin  GET Accounts/ ORM-Get -> 200 with outlen=56 in 1027 us
20160311 13074854  -    00.001.562
20160311 13074854  +    mORMotSQLite3.TSQLRestServerDB(07316160).URI(GET Accounts?session_signature=017E1BAE00017E2E0BB76508 inlen=307)
20160311 13074854 auth  	mORMot.TSQLRestRoutingREST(072D1040) SuperPuperAdmin/25041838 
20160311 13074854 SQL   	SynSQLite3.TSQLDatabase(04263628) from cache accounts.db3 SELECT ID,Created,Modified,OrderID,TotalCheck,TotalInDay,TotalInMonth,StartMonth,StartDay,FreePoints,PaidPoints,SmtpUserName,FirstName,LastName,Email,RefNo,Blocked,FreeUser,APIKey,Subscription,AllowedIPS,TotalZeroReported,EmptyTestReported,Version FROM User WHERE Version>:(69): order by Version limit 10000
20160311 13074854 res   	SynSQLite3.TSQLDatabase(04263628) {"fieldCount":24,"values":["ID","Created","Modified","OrderID","TotalCheck","TotalInDay","TotalInMonth","StartMonth","StartDay","FreePoints","PaidPoints","SmtpUserName","FirstName","LastName","Email","RefNo","Blocked","FreeUser","APIKey","Subscription","AllowedIPS","TotalZeroReported","EmptyTestReported","Version"],"rowCount":0} 
20160311 13074854 srvr  	mORMotSQLite3.TSQLRestServerDB(07316160) SuperPuperAdmin  GET Accounts/ ORM-Get -> 200 with outlen=331 in 1260 us
20160311 13074854  -    00.001.817
20160311 13074854  +    mORMotSQLite3.TSQLRestServerDB(07316160).URI(GET Accounts?session_signature=017E1BAE00017E2E0BB76508 inlen=121)
20160311 13074854 auth  	mORMot.TSQLRestRoutingREST(072D0C20) SuperPuperAdmin/25041838 
20160311 13074854 SQL   	SynSQLite3.TSQLDatabase(04263628) from cache accounts.db3 SELECT ID,Deleted FROM TableDeleted WHERE ID>:(576460752303423557): and ID<:(864691128455135232): order by ID limit 10000
20160311 13074854 res   	SynSQLite3.TSQLDatabase(04263628) {"fieldCount":2,"values":["ID","Deleted"],"rowCount":0} 
20160311 13074854 srvr  	mORMotSQLite3.TSQLRestServerDB(07316160) SuperPuperAdmin  GET Accounts/ ORM-Get -> 200 with outlen=56 in 2639 us
20160311 13074854  -    00.003.075
20160311 13074854  +    mORMotSQLite3.TSQLRestServerDB(07316160).URI(POST Accounts/ServiceRecordVersion.Subscribe?session_signature=017E1BAE00017E2E93B5E950 inlen=13)
20160311 13074854 auth  	mORMot.TSQLRestRoutingREST(072D0C20) SuperPuperAdmin/25041838 
20160311 13074854 call  	mORMotSQLite3.TSQLRestServerDB(07316160) ServiceRecordVersion.Subscribe["User",69,1]
20160311 13074854 trace 	mORMotSQLite3.TSQLRestServerDB(07316160) TInterfacedObjectFakeServer(04251800:1).Destroy IServiceRecordVersion
20160311 13074854 srvr  	mORMotSQLite3.TSQLRestServerDB(07316160) SuperPuperAdmin  POST Accounts/ServiceRecordVersion.Subscribe SOA-Interface -> 200 with outlen=18 in 619 us
20160311 13074854 ret   	mORMotSQLite3.TSQLRestServerDB(07316160) {"result":[false]}

SlaveServer

20160311 13074853  +    mORMot.TServiceFactoryClient(0414C590).InternalInvoke IServiceRecordVersion._contract_ [] 
20160311 13074853  +    	mORMotHttpClient.TSQLHttpClientWebsockets(0403E830).0092152C mORMotHttpClient.TSQLHttpClientGeneric.InternalURI (464) 
20160311 13074853 clnt  		mORMotHttpClient.TSQLHttpClientWebsockets(0403E830) POST Accounts/ServiceRecordVersion._contract_?session_signature=017E1BAE00017E2E93B827B4 status=200 state=13
20160311 13074853  -    	00.001.216
20160311 13074853 ret   	mORMot.TServiceFactoryClient(0414C590) {"result":["5DDC975F57031DB5"]}
20160311 13074853  -    00.001.230
20160311 13074853  +    mORMotSQLite3.TSQLRestServerDB(073225C0).RecordVersionSynchronizeSlave TUser
20160311 13074853 SQL   	mORMotSQLite3.TSQLRestServerDB(073225C0) 149us returned 1 row as 22 bytes SELECT max(Version) FROM User LIMIT 1
20160311 13074853 res   	SynSQLite3.TSQLDatabase(0413A8F0) [{"max(Version)":69}] 
20160311 13074853 DB    	mORMotSQLite3.TSQLRestServerDB(073225C0) prepared 29us users_slave.db3 SELECT max(ID) FROM TableDeleted WHERE ID>? and ID<? LIMIT 1
20160311 13074853 SQL   	mORMotSQLite3.TSQLRestServerDB(073225C0) 76us returned 1 row as 19 bytes SELECT max(ID) FROM TableDeleted WHERE ID>:(576460752303423488): and ID<:(864691128455135232): LIMIT 1
20160311 13074853 res   	SynSQLite3.TSQLDatabase(0413A8F0) [{"max(ID)":null}] 
20160311 13074853  +    	mORMotSQLite3.TSQLRestServerDB(073225C0).RecordVersionSynchronizeSlaveToBatch TUser
20160311 13074853  +    		mORMotHttpClient.TSQLHttpClientWebsockets(0403E830).0092152C mORMotHttpClient.TSQLHttpClientGeneric.InternalURI (464) 
20160311 13074854 clnt  			mORMotHttpClient.TSQLHttpClientWebsockets(0403E830) GET Accounts?session_signature=017E1BAE00017E2E0BB76508 status=200 state=13
20160311 13074854  -    		00.002.499
20160311 13074854  +    		mORMotHttpClient.TSQLHttpClientWebsockets(0403E830).0092152C mORMotHttpClient.TSQLHttpClientGeneric.InternalURI (464) 
20160311 13074854 clnt  			mORMotHttpClient.TSQLHttpClientWebsockets(0403E830) GET Accounts?session_signature=017E1BAE00017E2E0BB76508 status=200 state=13
20160311 13074854  -    		00.001.795
20160311 13074854  -    	00.004.336
20160311 13074854  -    00.004.629

Below is the log for the scenario 2:

MasterServer

20160311 13171223  +    MainServer.TAPI1Server(040EC6F0).URI(GET api/v1/CreateTest?apikey=xxxxx&Limit=22&offset=5&Groups=2049 inlen=0)
20160311 13171223 call  	MainServer.TAPI1Server(040EC6F0) CreateTest apikey=xxxxx&Limit=22&offset=5&Groups=2049
20160311 13171223 DB    	mORMotSQLite3.TSQLRestServerDB(07306160) prepared 71us accounts.db3 SELECT ID,Blocked FROM User WHERE APiKey=?
20160311 13171223 SQL   	mORMotSQLite3.TSQLRestServerDB(07306160) 571us returned 1 row as 25 bytes SELECT ID,Blocked FROM User WHERE APiKey=:('xxxxx'):
20160311 13171224 res   	SynSQLite3.TSQLDatabase(04143628) [{"ID":579,"Blocked":0}] 
20160311 13171224 SQL   	mORMotSQLite3.TSQLRestServerDB(07306160) 217us returned 1 row as 530 bytes SELECT ID,Created,Modified,OrderID,TotalCheck,TotalInDay,TotalInMonth,StartMonth,StartDay,FreePoints,PaidPoints,SmtpUserName,FirstName,LastName,Email,RefNo,Blocked,FreeUser,APIKey,Subscription,AllowedIPS,TotalZeroReported,EmptyTestReported FROM User WHERE ID in (579)
20160311 13171224 res   	SynSQLite3.TSQLDatabase(04143628) [{"ID":579,"Created":135293311538,"Modified":135301219462,"OrderID":"xxx","TotalCheck":618,"TotalInDay":46,"TotalInMonth":55,"StartMonth":"2016-02-15T16:46:05","StartDay":"2016-03-11T06:52:27","FreePoints":0,"PaidPoints":0,"SmtpUserName":"xxx","FirstName":"xxx","LastName":"xxx","Email":"xxx@xxx.com","RefNo":"47119952","Blocked":0,"FreeUser":0,"APIKey":"xxx","Subscription":3,"AllowedIPS":50,"TotalZeroReported":1,"Empty... (truncated) length=530
20160311 13171224 SQL   	mORMotSQLite3.TSQLRestServerDB(07306160) 174us returned 1 row as 22 bytes SELECT max(Version) FROM User LIMIT 1
20160311 13171224 res   	SynSQLite3.TSQLDatabase(04143628) [{"max(Version)":69}] 
20160311 13171224 DB    	mORMotSQLite3.TSQLRestServerDB(07306160) prepared 50us accounts.db3 SELECT max(ID) FROM TableDeleted WHERE ID>? and ID<? LIMIT 1
20160311 13171224 SQL   	mORMotSQLite3.TSQLRestServerDB(07306160) 517us returned 1 row as 19 bytes SELECT max(ID) FROM TableDeleted WHERE ID>:(864691128455135232): and ID<:(1152921504606846976): LIMIT 1
20160311 13171224 res   	SynSQLite3.TSQLDatabase(04143628) [{"max(ID)":null}] 
20160311 13171224 cache 	SynSQLite3.TSQLDatabase(04143628) accounts.db3 cache flushed
20160311 13171224 DB    	mORMotSQLite3.TSQLRestServerDB(07306160) prepared 59us accounts.db3 UPDATE User SET Modified=?,TotalCheck=?,TotalInDay=?,TotalInMonth=?,StartMonth=?,StartDay=?,FreePoints=?,PaidPoints=?,Version=? WHERE RowID=?;
20160311 13171227 SQL   	mORMotSQLite3.TSQLRestServerDB(07306160) 44.17ms  UPDATE User SET Modified=:(135301223500):,TotalCheck=:(619):,TotalInDay=:(47):,TotalInMonth=:(56):,StartMonth=:('2016-02-15T16:46:05'):,StartDay=:('2016-03-11T06:52:27'):,FreePoints=:(0):,PaidPoints=:(0):,Version=:(70): WHERE RowID=:(579):;
20160311 13171227 SQL   	mORMotSQLite3.TSQLRestServerDB(07306160) 376us returned 4 rows as 2386 bytes SELECT 
20160311 13171227 cache 	SynSQLite3.TSQLDatabase(04143628) accounts.db3 cache flushed
20160311 13171721 trace mORMotSQLite3.TSQLRestServerDB(07306160) BeginCurrentThread(TWebSocketServerResp) ThreadID=00002A28 ThreadCount=33
20160311 13171721  +    mORMotSQLite3.TSQLRestServerDB(07306160).URI(GET Accounts/TimeStamp inlen=0)
20160311 13171721 call  	mORMotSQLite3.TSQLRestServerDB(07306160) TimeStamp 
20160311 13171721 srvr  	mORMotSQLite3.TSQLRestServerDB(07306160)   GET Accounts/TimeStamp SOA-Method -> 200 with outlen=12 in 250 us
20160311 13171721 ret   	mORMotSQLite3.TSQLRestServerDB(07306160) 135301223505
20160311 13171721  -    00.001.024
20160311 13171721  +    mORMotSQLite3.TSQLRestServerDB(07306160).URI(GET Accounts/Auth?UserName=SuperPuperAdmin inlen=0)
20160311 13171721 call  	mORMotSQLite3.TSQLRestServerDB(07306160) Auth UserName=SuperPuperAdmin
20160311 13171721 srvr  	mORMotSQLite3.TSQLRestServerDB(07306160)   GET Accounts/Auth SOA-Method -> 200 with outlen=77 in 265 us
20160311 13171721 ret   	mORMotSQLite3.TSQLRestServerDB(07306160) {"result":"0009ea1df10edeb0e3b634afe2f34b53463bc2e3155a4c3df79654d475a38755"}
20160311 13171721  -    00.001.214
20160311 13171721  +    mORMotSQLite3.TSQLRestServerDB(07306160).URI(GET Accounts/Auth?UserName=SuperPuperAdmin&Password=e31a3c47f2bb426b5a0d2dd17d992ded960aeeef5bd0864e8cc069a29d797608&ClientNonce=0009ea1df10edeb0e3b634afe2f34b53463bc2e3155a4c3df79654d475a38755 inlen=0)
20160311 13171722 call  	mORMotSQLite3.TSQLRestServerDB(07306160) Auth UserName=SuperPuperAdmin&Password=e31a3c47f2bb426b5a0d2dd17d992ded960aeeef5bd0864e8cc069a29d797608&ClientNonce=0009ea1df10edeb0e3b634afe2f34b53463bc2e3155a4c3df79654d475a38755
20160311 13171722 DB    	mORMotSQLite3.TSQLRestServerDB(07306160) prepared 70us accounts.db3 SELECT ID,LogonName,DisplayName,PasswordHashHexa,GroupRights FROM AuthUser WHERE LogonName=?
20160311 13171722 SQL   	mORMotSQLite3.TSQLRestServerDB(07306160) 689us returned 1 row as 165 bytes SELECT ID,LogonName,DisplayName,PasswordHashHexa,GroupRights FROM AuthUser WHERE LogonName=:('SuperPuperAdmin'):
20160311 13171722 res   	SynSQLite3.TSQLDatabase(04143628) [{"ID":1,"LogonName":"SuperPuperAdmin","DisplayName":"Admin","PasswordHashHexa":"2f79797e7e41e761eed0d588e58bdc7bf8577e3576347246234a0f3b35c8e62a","GroupRights":1}] 
20160311 13171722 DB    	mORMotSQLite3.TSQLRestServerDB(07306160) prepared 55us accounts.db3 SELECT ID,Ident,SessionTimeout,AccessRights FROM AuthGroup WHERE RowID=?;
20160311 13171722 SQL   	mORMotSQLite3.TSQLRestServerDB(07306160) 744us returned 1 row as 99 bytes SELECT ID,Ident,SessionTimeout,AccessRights FROM AuthGroup WHERE RowID=:(1):;
20160311 13171722 res   	SynSQLite3.TSQLDatabase(04143628) [{"ID":1,"Ident":"Admin","SessionTimeout":10,"AccessRights":"47,1-256,0,1-256,0,1-256,0,1-256,0"}] 
20160311 13171722 DB    	mORMotSQLite3.TSQLRestServerDB(07306160) prepared 66us accounts.db3 SELECT Data FROM AuthUser WHERE RowID=?
20160311 13171722 SQL   	mORMotSQLite3.TSQLRestServerDB(07306160) 464us returned 0 bytes SELECT Data FROM AuthUser WHERE RowID=:(1):
20160311 13171722 auth  	mORMot.TAuthSession(040F5480) New "Admin" session SuperPuperAdmin/25596674 created at /1 running 
20160311 13171722 srvr  	mORMotSQLite3.TSQLRestServerDB(07306160)   GET Accounts/Auth SOA-Method -> 200 with outlen=219 in 6185 us
20160311 13171722 ret   	mORMotSQLite3.TSQLRestServerDB(07306160) {"result":"25596674+cd89e2471509ffe1828420fc6e31b634709f4a45b220b08a4fdaf9b0a0ce2941","logonid":1,"logonname":"SuperPuperAdmin","logondisplay":"Admin","logongroup":1,"server":"xxx- server","version":"1.0.0.0"}
20160311 13171722  -    00.007.607
20160311 13171722  +    mORMotSQLite3.TSQLRestServerDB(07306160).URI(POST Accounts/ServiceRecordVersion._contract_?session_signature=01869302000186DBC723D6B3 inlen=2)
20160311 13171722 auth  	mORMot.TSQLRestRoutingREST(072C0800) SuperPuperAdmin/25596674 
20160311 13171722 call  	mORMotSQLite3.TSQLRestServerDB(07306160) IServiceRecordVersion._contract_[]
20160311 13171722 srvr  	mORMotSQLite3.TSQLRestServerDB(07306160) SuperPuperAdmin  POST Accounts/ServiceRecordVersion._contract_ SOA-Interface -> 200 with outlen=31 in 537 us
20160311 13171722 ret   	mORMotSQLite3.TSQLRestServerDB(07306160) {"result":["5DDC975F57031DB5"]}
20160311 13171722  -    00.001.540
20160311 13171722  +    mORMotSQLite3.TSQLRestServerDB(07306160).URI(GET Accounts?session_signature=01869302000186DB21ED5E68 inlen=307)
20160311 13171722 auth  	mORMot.TSQLRestRoutingREST(072C0C20) SuperPuperAdmin/25596674 
20160311 13171722 DB    	mORMotSQLite3.TSQLRestServerDB(07306160) prepared 115us accounts.db3 SELECT ID,Created,Modified,OrderID,TotalCheck,TotalInDay,TotalInMonth,StartMonth,StartDay,FreePoints,PaidPoints,SmtpUserName,FirstName,LastName,Email,RefNo,Blocked,FreeUser,APIKey,Subscription,AllowedIPS,TotalZeroReported,EmptyTestReported,Version FROM User WHERE Version>? order by Version limit 10000
20160311 13171722 SQL   	mORMotSQLite3.TSQLRestServerDB(07306160) 815us returned 1 row as 543 bytes SELECT ID,Created,Modified,OrderID,TotalCheck,TotalInDay,TotalInMonth,StartMonth,StartDay,FreePoints,PaidPoints,SmtpUserName,FirstName,LastName,Email,RefNo,Blocked,FreeUser,APIKey,Subscription,AllowedIPS,TotalZeroReported,EmptyTestReported,Version FROM User WHERE Version>:(69): order by Version limit 10000
20160311 13171722 res   	SynSQLite3.TSQLDatabase(04143628) [{"ID":579,"Created":135293311538,"Modified":135301223500,"OrderID":"xxxxx","TotalCheck":619,"TotalInDay":47,"TotalInMonth":56,"StartMonth":"2016-02-15T16:46:05","StartDay":"2016-03-11T06:52:27","FreePoints":0,"PaidPoints":0,"SmtpUserName":"xxx","FirstName":"xxx","LastName":"xxx","Email":"xxx@xxx.com","RefNo":"47119952","Blocked":0,"FreeUser":0,"APIKey":"xxx","Subscription":3,"AllowedIPS":50,"TotalZeroReported":1,"Empty... (truncated) length=543
20160311 13171722 srvr  	mORMotSQLite3.TSQLRestServerDB(07306160) SuperPuperAdmin  GET Accounts/ ORM-Get -> 200 with outlen=543 in 3036 us

SlaveServer

20160311 13171722  +    mORMot.TServiceFactoryClient(0414C590).InternalInvoke IServiceRecordVersion._contract_ [] 
20160311 13171722  +    	mORMotHttpClient.TSQLHttpClientWebsockets(0403EE30).0092152C mORMotHttpClient.TSQLHttpClientGeneric.InternalURI (464) 
20160311 13171722 clnt  		mORMotHttpClient.TSQLHttpClientWebsockets(0403EE30) POST Accounts/ServiceRecordVersion._contract_?session_signature=01869302000186DBC723D6B3 status=200 state=20
20160311 13171722  -    	00.001.842
20160311 13171722 ret   	mORMot.TServiceFactoryClient(0414C590) {"result":["5DDC975F57031DB5"]}
20160311 13171722  -    00.001.858
20160311 13171722  +    mORMotSQLite3.TSQLRestServerDB(073725C0).RecordVersionSynchronizeSlave TUser
20160311 13171722 SQL   	mORMotSQLite3.TSQLRestServerDB(073725C0) 224us returned 1 row as 22 bytes SELECT max(Version) FROM User LIMIT 1
20160311 13171722 res   	SynSQLite3.TSQLDatabase(0413A8F0) [{"max(Version)":69}] 
20160311 13171722 DB    	mORMotSQLite3.TSQLRestServerDB(073725C0) prepared 48us users_slave.db3 SELECT max(ID) FROM TableDeleted WHERE ID>? and ID<? LIMIT 1
20160311 13171722 SQL   	mORMotSQLite3.TSQLRestServerDB(073725C0) 117us returned 1 row as 19 bytes SELECT max(ID) FROM TableDeleted WHERE ID>:(576460752303423488): and ID<:(864691128455135232): LIMIT 1
20160311 13171722 res   	SynSQLite3.TSQLDatabase(0413A8F0) [{"max(ID)":null}] 
20160311 13171722  +    	mORMotSQLite3.TSQLRestServerDB(073725C0).RecordVersionSynchronizeSlaveToBatch TUser
20160311 13171722  +    		mORMotHttpClient.TSQLHttpClientWebsockets(0403EE30).0092152C mORMotHttpClient.TSQLHttpClientGeneric.InternalURI (464) 
20160311 13171722 clnt  			mORMotHttpClient.TSQLHttpClientWebsockets(0403EE30) GET Accounts?session_signature=01869302000186DB21ED5E68 status=200 state=20
20160311 13171722  -    		00.004.111
20160311 13171722  +    		mORMotHttpClient.TSQLHttpClientWebsockets(0403EE30).0092152C mORMotHttpClient.TSQLHttpClientGeneric.InternalURI (464) 
20160311 13171723 clnt  			mORMotHttpClient.TSQLHttpClientWebsockets(0403EE30) GET Accounts?session_signature=01869302000186DB21ED5E68 status=200 state=20
20160311 13171723  -    		00.002.357
20160311 13171723 DB    		mORMotSQLite3.TSQLRestServerDB(073725C0) prepared 34us users_slave.db3 SELECT ID FROM User WHERE RowID=? LIMIT 1
20160311 13171723 SQL   		mORMotSQLite3.TSQLRestServerDB(073725C0) 134us returned 1 row as 13 bytes SELECT ID FROM User WHERE RowID=:(579): LIMIT 1
20160311 13171723 res   		SynSQLite3.TSQLDatabase(0413A8F0) [{"ID":579}] 
20160311 13171723  -    	00.007.160
20160311 13171723  +    	mORMotSQLite3.TSQLRestServerDB(073725C0).EngineBatchSend  inlen=599
20160311 13171723 cache 		SynSQLite3.TSQLDatabase(0413A8F0) users_slave.db3 cache flushed
20160311 13171723 DB    		mORMotSQLite3.TSQLRestServerDB(073725C0) prepared 80us users_slave.db3 UPDATE User SET Created=?,Modified=?,OrderID=?,TotalCheck=?,TotalInDay=?,TotalInMonth=?,StartMonth=?,StartDay=?,FreePoints=?,PaidPoints=?,SmtpUserName=?,FirstName=?,LastName=?,Email=?,RefNo=?,Blocked=?,FreeUser=?,APIKey=?,Subscription=?,AllowedIPS=?,TotalZeroReported=?,EmptyTestReported=?,Version=? WHERE RowID=?;
20160311 13171725 SQL   		mORMotSQLite3.TSQLRestServerDB(073725C0) 38.63ms  UPDATE User SET Created=:(135293311538):,Modified=:(135301223500):,OrderID=:('xxxxx'):,TotalCheck=:(619):,TotalInDay=:(47):,TotalInMonth=:(56):,StartMonth=:('2016-02-15T16:46:05'):,StartDay=:('2016-03-11T06:52:27'):,FreePoints=:(0):,PaidPoints=:(0):,SmtpUserName=:('xxx'):,FirstName=:('xxx'):,LastName=:('xxx'):,Email=:('xxx@xxx.com'):,RefNo=:('47119952'):,Blocked=:(0):,FreeUser=:(0):,APIKey=:('xxx'):,Subscription=:(3):,AllowedIPS=:(50):,TotalZeroReported=:(1):,EmptyTestReported=:(1):,Version=:(70): WHERE RowID=:(579):;
20160311 13171725 trace 		mORMotSQLite3.TSQLRestServerDB(073725C0) EngineBatchSend json=599 B add=0 update=1 delete=0 @User
20160311 13171725  -    	00.038.824
20160311 13171725 debug 	mORMotSQLite3.TSQLRestServerDB(073725C0) TSQLRestServerDB.RecordVersionSynchronize Added=0 Updated=1 Deleted=0 on {"TSQLHttpClientWebsockets(0403EE30)":{"Socket":{"THttpClientWebSockets(0403C300)":{"WebSockets":{"TWebSocketProcessClient(0737CC70)":{"Protocol":{"TWebSocketProtocolBinary(0414C400)":{"Compressed":true,"Encrypted":true,"FramesInBytesSocket":1308,"FramesOutBytesSocket":1180,"FramesInCompression":-9,"FramesOutCompression":-21,"Name":"synopsebinary","URI":"Accounts","LastError":"","FramesInCount":6,"FramesOutCount":6,"FramesInBytes":1197,"FramesOutBytes":972}},"InvalidPingSendCount":0}},"Sock":1220,"Server":"127.0.0.1","Port":"81","TimeOut":60000,"BytesIn":1484,"BytesOut":1547}},"KeepAliveMS":20000,"Compression":["hcSynLZ"],"Server":"127.0.0.1","Port":"81","LastErrorCode":200,"LastErrorMessage":"","MaximumAuthentificationRetry":0,"RetryOnceOnTimeout":false,"SessionID":25596674,"SessionServer":"inboxmonitor - server","SessionVersion":"1.0.0.0","ServerTimeStamp":135301223505}}
20160311 13171725  +    	mORMotSQLite3.TSQLRestServerDB(073725C0).RecordVersionSynchronizeSlaveToBatch TUser
20160311 13171725  +    		mORMotHttpClient.TSQLHttpClientWebsockets(0403EE30).0092152C mORMotHttpClient.TSQLHttpClientGeneric.InternalURI (464) 
20160311 13171725 clnt  			mORMotHttpClient.TSQLHttpClientWebsockets(0403EE30) GET Accounts?session_signature=01869302000186DB21ED5E68 status=200 state=20
20160311 13171725  -    		00.003.458
20160311 13171725  +    		mORMotHttpClient.TSQLHttpClientWebsockets(0403EE30).0092152C mORMotHttpClient.TSQLHttpClientGeneric.InternalURI (464) 
20160311 13171725 clnt  			mORMotHttpClient.TSQLHttpClientWebsockets(0403EE30) GET Accounts?session_signature=01869302000186DB21ED5E68 status=200 state=20
20160311 13171725  -    		00.002.208
20160311 13171725  -    	00.005.721
20160311 13171725  -    00.053.994

DigDiver · mORMot 1

I have a problem with the restart of real-time synchronization.

Master Server

 MasterServer :=  TSQLHttpServer.Create('81',[AccountServer],'+',useBidirSocket);
 MasterServer.WebSocketsEnable(AccountServer,'password');

 AccountServer.RecordVersionSynchronizeMasterStart();

Slave Server

MasterClient := TSQLHttpClientWebsockets.Create(MasterServerAddress,'81', AccountModel );
MasterClient.WebSocketsUpgrade('password');
MasterClient.SetUser('UserName','Password');

Result := AccountSlaveServer.RecordVersionSynchronizeSlaveStart(TUser ,MasterClient, NotifySync);

Then Master Server stopped and started again.

In the Slave Server I have the code that checks availability of the Master Server:

 if not  MasterClient.ServerTimeStampSynchronize then
  begin
   // Stop Synchronization
   AccountSlaveServer.RecordVersionSynchronizeSlaveStop(TUser);
   MasterClient.SessionClose;
   FOnErrorMessage('WebSockets', 'MasterClient', 'Session Closed');
  end
 else
  if MasterClient.SessionUser = nil then
   begin
    // Login to the Master Server
    Status       := MasterClient.WebSocketsUpgrade('bdfyjdbx66_gtnhjdbx70');
    MasterClient.SetUser('UserName','Password');

    // Start Synchronization

    try
     Result := AccountSlaveServer.RecordVersionSynchronizeSlaveStart(TUser ,MasterClient, NotifySync);
     if Not Result then
      begin
       FOnErrorMessage('Bounce', 'RecordVersionSynchronizeSlaveStart', 'Not Stated');
      end;
    except
     On e: Exception do
      begin
        FOnErrorMessage('Bounce', 'RecordVersionSynchronizeSlaveStart', E.Message);
      end;
    end;
   end;

And in RecordVersionSynchronizeSlaveStart method in the line:

    if service.Subscribe(tableName,current,callback) then begin

Debugger jumps to:

  
  // WELCOME ABOARD: you just landed in TInterfacedObjectFake.FakeCall() !
  // if your debugger reached here, you are executing a "fake" interface
  // forged to call a remote SOA server or mock/stub an interface

How to properly restart synchronization?

DigDiver · mORMot 1

In Delphi XE10 I can add & before keyword:

 Type
  TProduct = class(TSynPersistent)
private
  Fblablabla : RAWUTF8 ;
  FUnit : RawUTF8;
  FBegin : RawUTF8;
published
  property blablabla: RAWUTF8 read Fblablabla write Fblablabla;
  property &Unit : RawUTF8 read FUnit write FUnit;
  property &begin: RawUTF8 read FBegin write FBegin;
end;

I have test it and works fine:

procedure TForm14.Button8Click(Sender: TObject);
var
 FProduct : TProduct;
 JSON : RawUTF8;
 FValid : Boolean;
begin

 FProduct := TProduct.Create;
 try
  FProduct.&begin := 'Begin value';
  FProduct.&Unit  := 'Unit value';
  FProduct.blablabla := 'Bla';
  JSON :=   ObjectToJSON(FProduct);

  Memo1.Lines.add(UTF8ToString(JSON));

 finally
  FProduct.Free
 end;

 FProduct := TProduct.Create;
 try
  JSONToObject(FProduct, Pointer(JSON), FValid);
  ShowMessage(FProduct.&begin);
 finally
  FProduct.Free;
 end;
end;

DigDiver · mORMot 1

Does anybody try to use Elevate Web Builder?

DigDiver · mORMot 1

In my opinion, it is better to use the service method and API version.
Current version: http://myserver.com/V1/method_name

After schema is changed use new end point URL:
http://myserver.com/V2/method_name

So old clients will use V1, new user V2.

DigDiver · mORMot 1

Solution: {$ifndef CPUX64} cdecl; {$ENDIF}

procedure InternalFieldsEqual(Context: TSQLite3FunctionContext;
  argc: integer; var argv: TSQLite3ValueArray); {$ifndef CPUX64} cdecl; {$ENDIF}

DigDiver · mORMot 1

procedure InternalFieldsEqual(Context: TSQLite3FunctionContext;
  argc: integer; var argv: TSQLite3ValueArray); cdecl;
begin
...
end;
...
GroupServer.db.RegisterSQLFunction(InternalFieldsEqual, 3, 'FIELDSEQUAL');

When I try to compile project as 64 bit (Delphi 10 Seattle), I've got error:
[dcc64 Error] WP_server.pas(6084): E2250 There is no overloaded version of 'RegisterSQLFunction' that can be called with these arguments

32 bit compilation ok.

DigDiver · mORMot 1

3) I would recommend to use FireDac with direct Mysql connection. Using OnRecover, OnLost, ,OnRestored events of TFDConnection you can perform automatically reconnect.

DigDiver · mORMot 1

Delivery and spam testing service and IP Reputation Monitor.

After your message reached a mailbox provider, what happened? Marketers need tools to measure their Inbox placement and determine whether their messages are reaching their recipients or getting stuck in spam folders.

With G-Lock Apps you will know your Inbox placement and detect potential delivery issues before you send real email campaigns.

It’s very easy to send email, but it’s very hard to do it right. Don’t make the mistakes that will have you marked as a spammer; protect your brand’s reputation now!

Our seed list based deliverability test covering dozens of Inboxes worldwide discovers how your email is placed across major ISP, at what Gmail tab your email is delivered and what ISP blocked your message. Plus, it verifies the sender authentication such as DKIM and SPF, checks the email against spam filters and tracks email delivery duration.

Here is a sample delivery/email placement report

Valuable email placement reports help you track and fix possible deliverability problems to land in the recipient’s Inbox.

If you can find the time to run delivery tests before you email all your subscribers, you'll know if your emails trigger anti-spam filters, and you'll have a chance to tweak your content and land in everyone's Inbox. Plus, a test will show you if there is a temporary issue with email delivery at your ESP and you'll be able to delay your email campaign until the problem is fixed, or send the campaign through a different ESP if it's important and must be delivered on time. Temporary delivery issues may occur with any ESP and g-lock apps deliverability/Inbox Placement tests will help you reveal them in advance and take steps to get email campaigns sent seamlessly.

Back-end - mORMot server on Amazon EC2 instance (Windows Server 2012)
Front- end - jQuery + custom library

DigDiver · mORMot 1

Thanks for the fix. It's working now.

ab wrote:

The safest is to create you own TSQLTableJSON instance, then set the field types as you expect..

The problem is that JSON returned from the server by calling:

 Res := FProp.ExecuteInlined(SQL, True);
 FResult := Res.FetchAllAsJSON;
 Ctxt.Returns(FResult);

mORMot Open Source

#1 mORMot 1 » MongoDB replicaset PrimarySteppedDown » 2019-01-10 13:18:53

#2 mORMot 1 » TMongoCollection and collation » 2018-09-14 07:51:10

#3 Re: mORMot 1 » StatsAsJson([withTables,withMethods]) missing Text in json output » 2018-06-11 09:05:19

#4 mORMot 1 » StatsAsJson([withTables,withMethods]) missing Text in json output » 2018-06-11 08:33:52

#5 Re: mORMot 1 » [suggestion] function TSQLRecord.FillOne proposal » 2018-05-02 12:59:57

#6 Re: mORMot 1 » [suggestion] function TSQLRecord.FillOne proposal » 2018-04-30 11:06:10

#7 mORMot 1 » [suggestion] function TSQLRecord.FillOne proposal » 2018-04-12 12:15:35

#8 mORMot 1 » Statistics in the form of a time series » 2018-03-24 10:32:16

#9 Re: mORMot 1 » TSynSystemTime serious bug » 2018-03-24 10:05:29

#10 mORMot 1 » TTextWriter.Text new behavior? » 2018-03-23 12:50:09

#11 mORMot 1 » TSynSystemTime serious bug » 2018-03-23 11:20:57

#12 mORMot 1 » Where to find THttpApiWebSocketServer ? » 2017-05-29 06:26:27

#13 Re: mORMot 1 » Epoll under linux » 2017-05-20 13:04:07

#14 Re: mORMot 1 » TSQLHttpServer with custom ThreadClass » 2017-05-17 14:01:55

#15 mORMot 1 » TSQLHttpServer with custom ThreadClass » 2017-05-17 12:30:17

#16 mORMot 1 » Forward a request to other server » 2017-05-03 09:35:19

#17 Re: mORMot 1 » Json de-serialize help needed » 2017-03-09 06:16:17

#18 mORMot 1 » SMTP server for incoming emails. » 2016-12-21 08:25:57

#19 mORMot 1 » TSystemUse CPU Kernel and User value alwais 0 » 2016-10-03 13:07:20

#20 Re: mORMot 1 » SetUser in TSQLRestClientRedirect problem » 2016-06-22 07:49:05

#21 Re: mORMot 1 » TSQLRestStorageMongoDB.GetJSONValues Memory corrupt » 2016-06-20 11:27:18

#22 mORMot 1 » SetUser in TSQLRestClientRedirect problem » 2016-06-20 09:56:08

#23 Re: mORMot 1 » TSQLRestStorageMongoDB.GetJSONValues Memory corrupt » 2016-06-14 11:38:16

#24 Re: mORMot 1 » TSQLRestStorageMongoDB.GetJSONValues Memory corrupt » 2016-06-13 14:17:45

#25 Re: mORMot 1 » TSQLRestStorageMongoDB.GetJSONValues Memory corrupt » 2016-06-10 12:50:12

#26 Re: mORMot 1 » TSQLRestStorageMongoDB.GetJSONValues Memory corrupt » 2016-06-10 07:56:00

#27 mORMot 1 » ObjecttoJSON with woDateTimeWithZSuffix » 2016-05-20 13:55:18

#28 Re: mORMot 1 » MongoDB .GetJSONValues(%): missing column - count=% expected:% » 2016-05-17 11:44:21

#29 mORMot 1 » Clear Server statistics » 2016-05-17 11:42:17

#30 Re: mORMot 1 » MongoDB .GetJSONValues(%): missing column - count=% expected:% » 2016-05-12 13:11:16

#31 mORMot 1 » MongoDB .GetJSONValues(%): missing column - count=% expected:% » 2016-05-12 10:21:38

#32 mORMot 1 » CreateAndFillPrepare and TRawUTF8DynArray » 2016-05-12 08:54:10

#33 mORMot 1 » Mongodb $query,$orderby operators is deprecated » 2016-04-26 07:53:41

#34 Re: mORMot 1 » Problem with TSynBackgroundThreadMethodAbstract » 2016-04-01 08:17:13

#35 Re: mORMot 1 » Problem with TSynBackgroundThreadMethodAbstract » 2016-03-17 13:05:15

#36 Re: mORMot 1 » Problem with TSynBackgroundThreadMethodAbstract » 2016-03-17 10:24:11

#37 Re: mORMot 1 » Problem with TSynBackgroundThreadMethodAbstract » 2016-03-17 08:39:12

#38 Re: mORMot 1 » Problem with TSynBackgroundThreadMethodAbstract » 2016-03-17 08:06:09

#39 Re: mORMot 1 » Problem with TSynBackgroundThreadMethodAbstract » 2016-03-17 06:37:04

#40 mORMot 1 » Problem with TSynBackgroundThreadMethodAbstract » 2016-03-16 14:33:56

#41 Re: mORMot 1 » Restart real-time synchronization » 2016-03-11 13:56:04

#42 mORMot 1 » Restart real-time synchronization » 2016-03-11 09:08:53

#43 Re: mORMot 1 » JSON with class and Reserved Word or KeyWord » 2016-02-18 09:42:26

#44 Re: mORMot 1 » Smart Mobile Studio » 2015-12-16 06:40:42

#45 Re: mORMot 1 » Best practice when database schema will change in future » 2015-11-20 12:18:22

#46 Re: mORMot 1 » Delphi 64bit and db.RegisterSQLFunction » 2015-10-28 06:22:14

#47 mORMot 1 » Delphi 64bit and db.RegisterSQLFunction » 2015-10-27 14:18:23

#48 Re: mORMot 1 » Database connection exception handling » 2015-10-07 07:52:06

#49 Re: mORMot 1 » Softwares created using mORMot » 2015-10-07 07:43:19

#50 Re: mORMot 1 » Bug in function Iso8601ToTimeLogPUTF8Char » 2015-06-24 08:30:36

Board footer