mORMot Open Source

I think I  can understand your code somewhat better now: THMAC_SHA256 is created with the private key and it can be updated later without resupplying the key.

Incoming https can be forwarded to a mORMot server. But the connection has to be upgraded to http 1.1 to avoid  performance issues.

This is an example with the required changes, which are the same for setting up both the http and https section.

Edit  /etc/nginx/sitesenabled as root.

## Add upstream for keepalive
upstream http_backend {
    # ip:port for the backend servers
    server  127.0.0.1:8888;
    server  127.0.0.1:8889;
   # The number of inactive connections kept open. The oldest one is closed when the limit is reached.
    keepalive 100;
}

server {
    location /someurl {
             proxy_pass http://127.0.0.1:8888;
        # ....
     
## Add for keepalive START
       proxy_read_timeout     300;
       proxy_connect_timeout  300;
   
       # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
       proxy_http_version 1.1;
   
      # Remove the Connection header if the client sends it,
      # it could be "close" to close a keepalive connection
      proxy_set_header Connection "";
## Add for keepalive END
    }

I think  I can define the problem more clearly now:

Nginx  can setup the request it is forwarding to the backend server several ways. These are the ones MORMot can handle and I have tried:
HTTP/1.0 (default) ---------------------------------------> mORMot HttpWebserver :  after a ~5 sec timeout there is an error displayed on the console "Errno Recv()=11" , and after that the response gets delivered back to the proxy.
HTTP/1.1 (this requires certain settings for Nginx)  -->  mORMot HttpWebserver :  instant reply
   
There are two possible solutions : 

After setting up the reverse proxy for Nginx  for forwarding  both http & https incoming requests via HTTP/1.1. this is what happens:
a) incoming HTTP request   - Nginx sends HTTP/1.1.
b) incoming HTTPS request - Nginx sends HTTP/1.0.
 
Since b) looks to be a bug I have created a bug report for the Nginx team.

While this works out I am looking at  the mORMot  side  if the cause for "Errno  Recv()=11" can be found and solved.

When Nginx forwards a https request to the mORMot server as a http request it  ignores  "proxy_http_version  1.1", which is required for  keep-alive. Could be a bug.  So it seems there is no mORMot only solution for this. Unless it is somehow possible to upgrade to v1.1 from the mORMot side.

After several misconceptions about what could be the root cause, finally I have the right answer.

With the help of wireshark it turned out that when Nginx forwards the request the header is set to close the connection. It has to be "keepalive"  to have a fast response.
And this is how to do it.

Edit  /etc/nginx/sitesenabled as root.

# Add upstream for keepalive
upstream http_backend {
    # ip:port for the backend servers
    server  127.0.0.1:8888;
    server  127.0.0.1:8889;
   # The number of inactive connections kept open. The oldest one is closed when the limit is reached.
    keepalive 100;
}

server {
    location /someurl {
             proxy_pass http://127.0.0.1:8888;
        # ....
     
# Add for keepalive START
             proxy_read_timeout     300;
         proxy_connect_timeout  300;
   
          # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
       proxy_http_version 1.1;
   
      # Remove the Connection header if the client sends it,
      # it could be "close" to close a keepalive connection
        proxy_set_header Connection "";
# Add for keepalive END
    }

I assume that EWB server worked out of the box because it ignores this part of the header ad keeps it alive anyway.

It seems that  Nginx  holds back the answer because it is still waiting for something from the mORMot server. Which may never arrive, and in this case the answer is finally sent to the client after a timeout occurs. So the speed issue may be solvable by fine-tuning Nginx.  But the problem could also be something missing from the mORMot response, or a missing  low level signal ...

The Zeos only test app is as fast as it can be expected. It seems like a mORMot issue now within TSQLDBZEOSConnectionProperties.Execute.

I have noticed one difference  between fb 2.1 & 2.5. 

TSQLDBZEOSConnectionProperties.Create

2.1 needed all information in the Servername Parameter.
2.5 worked when all params were assigned properly

RObyDP

You are  focusing on speed, but compression ratio and CPU load are very important too.  Can you share results with these included?

The gain in compression speed may easily be lost  while transferring less compressed data on the wire. A closer to real life test would be to extend the test case with transferring the data via WAN to a mORMot client.

Currently mORMot is at its best on 32Bit, it is excellent for cheep linux VPS's. Probably most users would be interested in 32Bit solutions.

Erick,

I have EWB too. Exploring SMS these days.

SMS compiler is much more advanced than EWB. It has a very interesting memory model. Supports the use of pointers and delphi style memory allocation instead of relying directly on DOM, which makes possible to write shared code with delphi/fpc much more easily. This feature being new I am not sure how stable it is. 

On the other hand EWB is stronger on the GUI front , plus Controls can be bound to datasets in delphi style. 
Inmemory datasets with transaction handling are also more advanced in EWB.

IDE:

SMS text editor is better, more like Delphi/Lazarus
Property editors are lacking in both products for me.

EWB support is quality but releases often get heavily delayed. 

I wish we could have the best of both world in one product.

Ok, thanks.

This is VERY helpful. Thank you.

fList  and fListInterfaceMethod  are not independent. They have a one to many relationship which can be exploited. If updated in the right order the data structure they represent remains valid even between the two atomic operations. The trick is that when expanding the list fListInterfaceMethod must be updated first and fList is to be updated last. I had a quick look at he code and it seems the TServiceContainer.AddServiceInternal is where changes could be done. When removing items the update order is reversed, fList  must be updated first. When there are both updates and deletes it is a two phase process. All updates first and then all deletes in the second phase.

This is easy. Since with my suggestion updates are forced to be serialized it is enough  to check if the item to be added/removed already exists in NewList .

Requirements can be different from yours. Some servers need to be on truly 24/7 and it must be possible to change/extend its functionality without restarting or going offline for maintenance. Which can be achieved via plugins or/and scripting, but this is the point when updating the list of interfaces in a thread safe fashion comes to the picture. Currently a mORMot based server must go offline to safely update the published interfaces.

This may still cause problems if any code is using the original object at the time of the exchange. It could work if all the routines accessing  the object  do so by assigning it to a local variable which is the only point of access from there. But freeing the object when not needed any more could be tricky in this case.  Using interfaces (which are reference counted)  should be the solution. It seems to me that thread safety  cannot be accomplished without AB having to change some code.

TOmniValue might be worth  a look for ideas.

http://www.thedelphigeek.com/2010/03/sp … e-and.html

AB,

I think the record versioning approach in itself is problematic. The replication  can cause inconsistent states for periods of times. Following the unit of work pattern and assigning the same version number to every record modified by processing a "unit of work" could ensure consistency. In simple cases like updating a single record it would be almost the same as record versioning.

They both seem to be good. Maybe we should create a positive/negative list for both.

" fallback to the optimal mechanism depending on platform: epoll, kqueue, IOCP, event ports"

this goes for Websocket Plus Plus too. It can be linked with different libraries. Boost being one of them uses the best choice for every platform. I only outlined epoll because the platform most developer has to deal with beside Windows is Linux, so this is where mORMot could use the improvement the most.

I think the question here is more than just async IO,  websockets support is a huge topic as well:

WSPP passes all the Autobhan tests, it implements most  websocket features, much more than mORMot does. For example one I am interested in is  scatter&gather. I would argue that what it has to offer on  the websockets front carries quite a weight.

It has the ability to respond to HTTP requests other than WebSocket Upgrade requests.

I may be wrong but it seems to me that it has all the low level stuff based on epoll, and mORMot has all the high level implementation to handle the http requests in its own httpserver implementation. They seem like a good match.  It could be more than two birds with one stone to provide efficient websocket and http server implementation for a lot of platforms with a single library. Having the core compiled with eg a GNU compiler could boost the performance as well. You already have the sqlite obj files compiled from C in the framework.  WebsocketPP could become part of mORMot in the same manner.

It seems to be a typical head office with local offices scenario.  Search for "Replication use cases" in the documentation.

You don not have to change the existing code, ORM classes are just and other way to access the same data. They can coexist with any DAC, though one have to be mindful of caching. You can write a simple generator to  automatically create ORM classes from database metadata. The framework handles it the other way around.

It is worth discovering what mORMot has to offer. Eg   interface based services, websockets  two way communication, callbacks via interfaces, live replication ...   

No, I meant one  slave db for each client. But now I can see that you do not even need that since you already have the local office servers as slaves.

You might want to check mORMot's up and coming bigdata support.
An SQLite  database   can be smaller than a firebird database with the same data. It is worth creating a test export to see the ratio.

Even from a single database you can export  & compress the  data  you want to have on your local server. Once again interface based services can be useful here.

mORMot tends to be well optimized. You may find it's replication to be gentle enough with the bandwith and sqlite to be fast enough applying the updates. Search for "Data access benchmark" in the documentation.
Though SQLite does not support stored procedures, so it may not be suitable to easily replace the local office servers. But in the cloud either SQLite or PostgreSQL can be a better candidate because of the way they store records they tend to use less disk space and memory.

It is worth testing  your usage scenarios  with mORMot for speed and resource consumption before making design decisions.

emk,

Applying meta changes to all these databases can be painful. If you have only a single executable it may be down for a while until all updates are finished.

A single database as AB has suggested can save you a lot of headaches.  I think the best  would be probably to set up a selective master-slave replication where the master database has all the data and each slave database contains only client specific data.  mORMot supports  replication where slave servers are online only for the time of the replication . You only need a second server which can act as the slave server on behalf of all client databases.  It could iterate through the client databases periodically and run the replications one by one. This way you can have the best of both architectures without the disadvantages. All clients can access the server any time.  Also significantly less resources are used.   You may even include both  servers (Master & Slave) into a single  executable.

Cheers,
Leslie

Thanks!

I have searched the crossplatform units for 'TSQLRestBatch  and came up empty. I should have searched just for 'batch'.

Ab,

"since each client is a server"

I think you probably have not understood the concept. I  have not  mentioned clients at all - clients are serviced by the frontend server, it is business as usual.
It is the server side  which has two layers. One accessible from the outside word, but the other one -the backend - with the the sensitive data is not. 

"Just use a mORMot server for what you call a "client", and a mORMot client on the "server" side."

This is the normal logic  for a frontend server to connect to a backend server. The one thing I would like to change if possible is  how the connection is initiated. Normally the frontend server would initiate the connection as a client to the backend server. I would like the backend server to initiate the connection which is used by the frontend server to call the published services of the backend server.  From the connection viewpoint the backend is the client and the frontend is the server. But from the viewpoint of the services the frontend  is the the client and the backend  is the server.

This setup would shield the backend server from being hacked from a compromised frontend  server.
I know this may seem an overkill, but there is data sensitive and valuable enough worth the extra mile.