mORMot Open Source

@mvp
Looking at http.sys websocket implementation, I feel it very close and native to me, as I remember well back on 2013, when I fight making it work. I would like to give it a try with mormot, and already had adopted it, now serving real-time communication between internal services stable, at extremely high rate and no latency. So, thank you very much for your and Vadim contribution, now it is not sitting in the corner

Speaking about it, I was always wondering about migrating on Linux. Recently was reading a lot about RabbitMQ also.
But I did not felt comfortable going this way. I am stuck with Window's SRWLock, ConditionVariables, Queues. Will have hard time moving all of them, and to be honest, I am very satisfied about their performance.

I am very keen to ask you about your impression, pros and cons migrating services to linux, and rabbitmq performance itself in terms of throughput and latency? I am sure you put it on a stress

yeah, past three years I experience random AVs as described. One on million, but still random AVs, was unable to track them.

mvp, does it resolves this old timer in your opinion:
https://synopse.info/forum/viewtopic.php?id=3283

Just compiled the new sources, and cross fingers for next 24 hours

I guess the code

      while item[itemcount].FromNext(row.Item.Data.DocList) do begin
        inc(itemcount);
        if itemcount>itemsize then begin
          inc(itemsize);
          Setlength(item,itemsize); // a field was deleted from TSQLRecord
        end;
      end;

in some cases item[itemcount] can be out of range, addressing "future" record while row.Item.Data.DocList is not at its betEOF, leading in assigning TBSONElement record properties in not assigned heap. Later increasing item array Length may lead to unpredictable memory behaviour on next iterations on same heap area.
Or something like this

Thanks ab,

Along with not obvious AVs, I got this madExcept log:

exception message  : Access violation at address 0000000000411D75 in module 'test.exe'. Write of address 0000000000FACCF0.

thread $1184:
00411d75 +015 test.exe System                        @LStrClr
0080fff6 +0a6 test.exe SynCommons         34391   +5 TJSONCustomParserRTTI.FinalizeNestedRecord
008100e4 +044 test.exe SynCommons         34421   +7 TJSONCustomParserRTTI.FinalizeNestedArray
00810126 +016 test.exe SynCommons         34429   +1 TJSONCustomParserRTTI.AllocateNestedArray
008105a3 +303 test.exe SynCommons         34493  +29 ProcessValue
00811320 +510 test.exe SynCommons         34668  +64 TJSONCustomParserRTTI.ReadOneLevel
00811e47 +037 test.exe SynCommons         34818   +2 TJSONRecordAbstract.CustomReader
0080dee3 +2b3 test.exe SynCommons         33861  +19 RecordLoadJSON

Please, note, that I do not have any string properties defined in the record!

I think RecordLoadJSON in some odd cases brokes input buffer and/or its own props buffer.
Even in other records containing RawUTF8 I got broken reference to them causing system finalizearray to AV on record cleanup upon procedure end.

Record definition is as follow:

  TTestDTO = packed record
    Id: Integer;
    Seen: Integer;
  end;

  TTestArrDTO = packed record
    arr: array of TTestDTO;
  end;

  __TTestDTOJSON =
    'Id Integer ' +
    'Seen Integer';

  __TTestArrDTOJSON =
    'arr [ ' + __TParserOddSeenDTOJSON + ' ]';

  TTextWriter.RegisterCustomJSONSerializerFromText(TypeInfo(TTestArrDTO), __TTestArrDTOJSON).Options := [soReadIgnoreUnknownFields];

Using Delphi XE7, Debug 64bit

When I thread protect the call to RecordLoadJSON() everything goes just fine.

Any input where and how I should proceed will be appreciated.

Indeed, TEXT is preffered and recommended.
There are many other "on the fly" methods in postgresql to limit the length later if needed at almost no penalty.

Adding dPostgreSQL in DB_HANDLEINDEXONBLOBS did the trick, now multi indexes are created right.
Next problem is that CreateSQLMultiIndex throws an exception on second program run.
As far as I know, there is no statement IF NOT EXISTS for index creation in Postgresql, hope I am wrong.

But ZEOS should have information about current table indexes, and by its name mormot can determine whenever to try create the index or not?
Or maybe do as described here http://dba.stackexchange.com/questions/ … -not-exist for Postgresql 9.0+

thanks, and please excuse me.
will try tomorrow limiting text size with limit attribute, making it varchar I guess, in order to be involved in multi index.
than will try  create multi field index again.

ok, to be more precise:

here is the case when length(FieldNames) <> 1 in TSQLRestStorageExternal.CreateSQLMultiIndex():

  if not (fProperties.DBMS in DB_HANDLEINDEXONBLOBS) then
    // BLOB fields cannot be indexed (only in SQLite3)
    for i := 0 to n-1 do begin
      extfield := fFieldsInternalToExternal[IntFieldIndex[i]+1];
      if (extfield>=0) and
         (fFieldsExternal[extfield].ColumnType in [ftBlob,ftUTF8]) and
         (fFieldsExternal[extfield].ColumnLength<=0) then begin
        if i=0 then
          exit; // impossible to create an index with no field!
        //SetLength(ExtFieldNames,i); // truncate index to the last indexable field   // commend by chapa
        break;
      end;
    end;

Imagine we index on "integer,text,text"
In cycle we pass i=0 as condition fFieldsExternal[extfield].ColumnType in [ftBlob,ftUTF8] I think should not met.
Than, on next iteration, when i=1 I guess above if condition is met, we get into begin ... end, but i<>0 and we do not exit, but break the cycle setting length of the ExtFieldNames to 1 (leaving only first column to be indexed).
Later on "SQL := fProperties.SQLAddIndex(fTableName,ExtFieldNames,Unique,Descending,IndexName);" here ExtFieldNames point only on index "integer", but not "integer,text,text".

Indeed, I was not clear enough.
I am aware of caching mechanism of mORMot and multithreading.

Take into consideration following use case.

We have one writer and many readers. Writer try to pass real-time data source, which changes really often (50000 times per second), and latest (most up to date) state need to be able to be read from clients with as less latency.
Only recent state should be kept in the DB.

Writer can insert/update/delete at high rate. All record states can be cached for later sync with DB.
In this case 1000 updates/sec on one record can later touch DB for update only 1-2 times, with most recent state for sync.
Meantime, upon client request, server will return either from DB/internalcache, or most recent record for sync, which is not already persisted with DB.

The purpose and final goal is, with compliments to DB's "DiskSync=Off" or common DB fault tolerant mechanisms, we will have super fast in-memory sync&flush structures and possibility to update/insert and query records at really fast rate, to reflect real-time data, and only most recent record (or even only modified properties of records (column)) to be synced (flushed to db) at later time (after period of time, after number of changes, etc).

Of course this may be set up only for one table, where we can afford data loss of last record states for sync in case of disaster. They may even flush to binary file upon shutdown and later come up into the memory if server restart occur (like mORMot's Memory Tables)

All above can of course be done with custom in-memory data structures without overhead of mORMot, but it will be custom solution. Much better to have REST and mORMot over it.
Such a functionality will be Redis killer for Delphi programmers.

I know there is TSQLRestStorageInMemory and similar, which are fast, but are still not a real DB.
As one may have millions of records which are synced into a DB, and at a time only 100 records are "live" and updated at high rate.
Currently, can not load millions records into TSQLRestStorageInMemory nor update DB backed storage at high rate.
Maybe TSQLRestStorageInMemory which syncs with corresponding RestStorage and some manager in front of them will do the trick.

Such implementation may later be used for replication.

Hope it is more clear now

Also,

CreateMissingTables() seems to explicit create unique index by id:
CREATE UNIQUE INDEX ndxsamplerecordid ON public.samplerecord USING btree (id);

but Postgresql creates index on the fly when PrimaryKey Constraint specified in Create Table:
CREATE TABLE public.samplerecord (
  id INTEGER NOT NULL,
  ...
  CONSTRAINT samplerecord_pkey PRIMARY KEY(id)
)

All tables end up with duplicate index for primary id key.
Extra index, extra disk size, extra row update time.

Thanks for sharing this info

Multithreaded x64 debugging is so hard ... ide crashes at random

Anyway, trying to ObjectToJSON() an object, which contains TObjectList as a property.
The AV is on the next line 31130 at TJSONSerializer.WriteObject:

if (not TSQLRecord(Value).fFill.JoinedFields) and (P^.PropType^^.ClassSQLFieldType=sftID) then

here Value if of class TObjectList, recognized as Kind tkClass.
but why you try to cast this object as TSQLRecord and try to access a field of it (fFill in this case)? this definitely will lead to random AV ...

Simple fix works if compiler optimizations is turned on and exchange the if's AND statements like:

if (P^.PropType^^.ClassSQLFieldType=sftID) and (not TSQLRecord(Value).fFill.JoinedFields) then

At run-time first will be checked P^.PropType^^.ClassSQLFieldType which is sftUnknown and skip erroneous cast and go the else clause and finally return json without the AV

Hope my explanations are understandable and will help.

Thanks ab,
Yes, source json looks like the one you typed, with many spaces and #10 #13.
Will test it tomorrow over my source json and let you know results.
Thanks once again.

It was fun explaining my wife what mORMot is all about watching the demo at Sunday family's dinner time.
Now, my 4 and 5 years girls have more clear idea about the work at the office during daytime ...

Forgot to mention, fiddler also tried, no Content-Encoding header in the response, body is decompressed.
Fiddler is a proxy, they use .NET classes to access Inet, AFAIK.

Once I used to write protocol handlers. Before response hit the client it can be manipulated by several handlers. So I guess it is something like that sitting between the client and actual response.
For protocol handler point of view it does not mind who the client is.

Will give a try to wireshark tomorrow and let you know.

I believe it is some protocol handler on my win box.
Tried to disable the antivirus, but no luck, same problem.

Some protocol handler seem to decompress on the fly the http compressed response content and serve it uncompressed skipping Content-Encoding: deflate header from the http response.
If someone know how I can list/trace all protocol handlers in order to catch who is doing this will help a lot.
Thanks.

ab, after updating the source with the new TSQLRestServerURIContext I got following:

20131029 16374619  " EXCOS EAccessViolation (C0000005) at 001CF809 mORMot.TSQLRestRoutingREST.URIDecodeSOAByInterface (24348)  stack trace API 001865C4 SynCommons.SynRtlUnwind (31512) 0000945C System.@HandleOnException
i := Server.Services.fListInterfaceMethod.IndexOf(URI); -> here Server.Services are nil

This happen every time when trying to call invalid (non-existing) method name of the rest server (and no interface services defined).
Maybe some handled error will be more suitable than showing AV exception with debug information to the client.

Also I was unable to get compression working.
Defined in the project COMPRESSDEFLATE, also tried to manually RegisterCompress, but with no luck.
Browser request compression, but the server never compress (tried also with big json as response).

Try for example 01 - In Memory ORM with ReportMemoryLeaksOnShutdown := True; --> Leaks

I suspect even "uses SynCommons" leaks.
Another example is SynCommons Initialization GarbageCollectorFreeAndNil(SynAnsiConvertList,TObjectList.Create)
Here instance of TObjectList leaks. (not freed on finalization)

Will investigate GarbageCollector, bu I think Ab should have a clue where the problem should be and save me some time
Thanks.

Edited: It was my mistake, above initialization and finalization works fine. Sorry

Thanks ab