mORMot Open Source

Would be handy. Otherwise everyone has to revert his local *.lpi file(s) before commiting. That's quite error-prone.

Edit:
Please also add to gitignore:
tests/data/*
tests/*.log
tests/*.res

Edit2:
I merged my cloned branch with mORMot2:master.
One of the last commits causes mormot2tests failing to compile (too many params)...
See pull request for fix: https://github.com/synopse/mORMot2/pull … 14c64369a4

I'm totally with you over here. Splitting into smaller units is a good idea, but don't split into sub-projects. That's worth nothing and only introduces new problems. One project to rule them all ...

About compiler support:
Hmm, well, thats a difficult decision...
Delphi 5 and Kylix support is no longer needed for sure. I think no one is using those compilers in combination with mORMot seriously.
But then things get complicated. I guess there are some users for every other version...
Let's wait for the result of the survey which versions are worth to be supported in mORMot2.
We're using Delphi 10.3 and D7 (only one older project left) over here. But the D7 project has to be merged to 10.3 anyway.
The current mORMot codebase is very stable and mature in general. Only bugfixes in 1.18 version from now on will be perfectly fine.

Maybe it's even worth the big cut to support Delphi 10.3+ and FPC only?
Being able to use current language features in mORMot2 would be nice, wouldn't it?

But that won't work in reality I guess

Well, let‘s start like this: If you really want to store large blob data in a database, then sqlite is one of your best friends to do so. That's because of its basic design. It‘s a big advantage to have your storage engine running in-process for that task. Put that large blob table into its own db file and use data sharding principles if possible. Don‘t put that table in your main db file. 

A short story with some oversimplified theory:

Once upon a time in database land, there was a performance issue regarding a Firebird RDBMS based 3-tire app. Must have been about 12 years ago approximately.

Everything worked like a charm, but once one specific table has reached about 250.000 records containing 180 GB of blob data, things started getting really weird... Indexes were fine, db settings were fine, page size was fine, cache options were fine etc... 
Adding new records to that large blob table was fine, selects were not that fast but ok, updates were fine.

But: working with other havy used tables in that db became ultra slow.

At the end it breaks down to simple physics...

We had other tables in that db with a lot of text blob data of various length (0-16k bytes, iso8859-1 codepage). 

At the end every single SQL server has to store its data into a file or set of files. There is no magic in how SQL servers work. A flat record file containing an array of fixed length records paired with a well implemented b-tree for indexing will always be the most performant way of accessing data. This is what all sql servers are doing at the end. They store fixed length data blocks with a bit of metadata info and your data as payload. Depending on the implementation of the sql server some or many tuning options and clever optimizations come into play. You can tune page and cache sizes, there are algorithms calculating expected table growth etc..

Data on disk is physically stored like that in sequence:

[Table A, Record 1]
[Table A, Record 2]
[Table A, Record 3]
[Table B, Record 1]
[Table B, Record 2] 
[Table C, Record 1]
[Table A, Record 4]
[Table C, Record 2]
...

I think you got it, data is appended at the end of the file in creation order.

RDBMS try to arrange records of same type/table as close together as possible.

[Table A, Record 1]
[Table A, Record 2]
[Table A, Record 3]
[
  EMPTY BLOCK for 10 new Table A records
]
[Table B, Record 1]
[Table B, Record 2] 
[
  EMPTY BLOCK for 10 new Table B records
]
[Table C, Record 1]
[
  EMPTY BLOCK for 10 new Table C records
]
[Table A, Record xxx]
...

If you have a lot of variable-length data records (with text-blobs, big varchars, etc..) and lot of various inserts, those reserved blocks become unbalanced.
This results in your data being spread around just everywhere in your db pages.

The physical HDD heads have to do a lot of movement to collect all data from db pages on disc. Modern SSD drivers don't face that problem as bad as physical drives because of their architecture. But even SSDs have to do a lot of calculation in such a case. If there are many concurrent users who are requesting that data all the time, then the HDD will go crazy. This is what has happend in my case.

Things are getting even worse the more data is inserted into db.

Well, in Firebird RDBMS there is a way to reorganize the ODS (on disc structure): Do a simple DB backup and restore. When restoring db, all the table data is written back in sequence, so all Table A records reside next to each other on disk.

That helped a lot and improved performance, but we were forced to do those backup/restore jobs quite regularly.

It was absolutely no fun to do so. It took a lot of time to backup/restore those 200 GB of data. Data was still growing and soon we've reached a point where such backup/restore tasks took longer then the out-of-office times of the users.

The DB became unmaintainable.

Solution was to store those blobs outside of db, that fixed all those problems.

So, to come to an conclusion: It depends a lot on your use-case and data types if it is a good idea to store large blobs in db.
My general advice is: don't do it.
Do it only if you exactly know what and why you are doing it.

No, you got me wrong here.

My advice is to use the ORM part as a simple data storage layer only. All the domain specific functions should reside in higher level services. Don't use TSQLRecords in your domain logic. The domain should know nothing about TSQLRecords.

If you you are in a large/complicated domain, then keep your CQRS Services who are responsible for storing TSQLRecord* data as dumb as possible. By following mORMot's DDD principles you end with a TSQLProductRecord=class(TSQLRecord)... and a correspondig TProductRecord = class(TSynPersistent). TSQLProductRecord is your ORM storage class, TProductRecord is your PODO (plain old Delphi object). Your bussines rules should aways work on TProductRecord, not on TSQLRecord* classes.

E.g.: sometimes it's not enough to store any kind of item arrays as Variant only in your entity. Searching for specific items would result in loading all entities, then searching each entity for the array-item.
If you have another higher-level service in front of your CQRS service, then the .Add method could utilize various CQRS services and split your PODO values to different TSQLRecords...

I hope you get what I mean.

There‘s simply not enough memory available for such an amount of data in X32. You could switch to X64, but this sounds like a design issue to me. Dou you really have to fetch all that data at once?

Hi,
InnoSetup has built-in support for LZMA file compression using a dll. Have a look at https://github.com/jrsoftware/issrc/blo … s/LZMA.pas
Another option would be to go for translating headers and directly linking the C .obj file.

Pavel, I think i traced the problem down and have a fix available...

I've added a breakpoint to "function THttpApiWebSocketConnection.ProcessActions(ActionQueue: WEB_SOCKET_ACTION_QUEUE): boolean;":

Following two Actions happen when closing the tab with Firefox:

1) WEB_SOCKET_INDICATE_RECEIVE_COMPLETE_ACTION
  BufferType = WEB_SOCKET_CLOSE_BUFFER_TYPE

2) WEB_SOCKET_SEND_TO_NETWORK_ACTION

Following single action happens when closing the tab with Safari:

1) WEB_SOCKET_RECEIVE_FROM_NETWORK_ACTION
THttpApiWebSocketConnection.ReadData... is called.
Inside this function -> Err= ERROR_HANDLE_EOF -> Disconnect is called.

I've changed procedure ReadData to a function:

function THttpApiWebSocketConnection.ReadData(const WebsocketBufferData): integer;
var Err: HRESULT;
    fBytesRead: cardinal;
    aBuf: WEB_SOCKET_BUFFER_DATA absolute WebsocketBufferData;
begin
  result:=0;
  if fWSHandle = nil then
    exit;
  Err := Http.ReceiveRequestEntityBody(fProtocol.fServer.FReqQueue, fOpaqueHTTPRequestId, 0,
    aBuf.pbBuffer, aBuf.ulBufferLength, fBytesRead, @self.fOverlapped);
  case Err of
    ERROR_HANDLE_EOF:
    begin
//     Disconnect;
      Result:=-1;
    end;
    ERROR_IO_PENDING: ; //
    NO_ERROR: ;//
  else
    // todo: close connection
  end;
end;

and the calling function

function THttpApiWebSocketConnection.ProcessActions(ActionQueue: WEB_SOCKET_ACTION_QUEUE): boolean;
...
    case Action of
...
      WEB_SOCKET_RECEIVE_FROM_NETWORK_ACTION: begin
        for i := 0 to ulDataBufferCount - 1 do
          if ReadData(Buffer[i])=-1 then begin
            EnterCriticalSection(fProtocol.fSafe);
            try
              fProtocol.RemoveConnection(fIndex);
            finally
              LeaveCriticalSection(fProtocol.fSafe);
            end;
            fState := wsClosedByClient
            fBuffer:='';
            fCloseStatus := 1001;//Buffer[0].Reserved1;
            EWebSocketApi.RaiseOnError(hCompleteAction, WebSocketAPI.CompleteAction(fWSHandle, ActionContext, 0));
            result := False;
            exit;
          end;
        fLastActionContext := ActionContext;
        result := False;
        exit;
      end;
...

With this fix everything works as expected. Could you have a look at that fix and incorporate it into trunk if ok?

Cheers,
oz.

Thanks Arnaud!

Thanks for the informations @mpv!

Well explained! We are using the same approach over here.
Just a little question: is it save to let one TSQLRestServer instance be served by different THttpServers at the same time? Meaning one WebSocket Server for server-to-server communication and one http.sys server for an Admin-MVC Client frontend? I remember trying that some time ago and faced problems as far as i can remeber. But as far as I understand it should be possible by design.

Sorry for replying late, the issue is fixed indeed. Thank you!

Hi, current WebSocket implementation requires exclusive port usage. You can not share the same port with your IIS instance when using any other kind then http.sys mode. Just specify another port in the constructor.

Hi,
did you have a look at TMVCRunWithViews.SetCache? Looks like a good starting point for further investigation...
Cheers, oz.

No worries, you're doing a great job maintaining this huge and fantastic project!
I'm the one who is to blame for releasing a build of our software without testing it enough
Gladly it was good enough to "eat" the exception without any further harm in this situation.
I tried to fix the root of this issue by myself, but had to give up after 10 minutes because a quick fix was required to get the system up again asap. I'm into mORMot for about a year now but there are parts who are still difficult to get into.
I've already pushed a new version for this customer with your fix included.

I just helped myself with introducing an empty try...except...end block to TDynArrayHashed.HashAdd() for "eating" the exception. Things seem to work, my logs are full of "Assertion failed" exceptions, but at least the system on my customer's server is up again. But this is not the solution obviously.

If so, then a quick test using Delphi 7 compiler & FullDebugMode should work without any problems. Does it?

Just have a look at: http://synopse.info/files/html/Synopse% … #TITLE_231 for a starting point...

You're welcome, i'm happy my demo program did help!
I just did some tests, the issue is gone, thanks.

I don't have compiler at hand right now, but try if following change helps to get sources compiled:

function TECDHEProtocol.Clone: IProtocol;
begin
  result := TECDHEProtocol.CreateFrom(self);
end;

What's the return value of "Server.Add(Customer, true);"? Not "0"?