FireBird 2.1 - extremely slow

Leslie7 · 2016-10-06 00:03:36

Hi,

I have an unusual setup: mostly windows apps with wine but it works really fast.

Ubuntu 14.04 32Bit

Nginx (Linux) is the front. It handles the static part and forwards the data requests to
the EWB Webserver(Wine) - which comes with Elevate Web Builder. It uses odbc for data connections to FireBird 2.1 (Wine).

Now that I have a little more time the goal is to create a fully native, mORMot based solution for Linux. The webserver is already up and running, but at this stage it still connects to the Firebird server( wine - localhost -> not embedded!). The problem is that a simple query takes 2-5 seconds to execute via TSQLDBZEOSConnectionProperties. Before this I haven't had any performance or stability issues, wine works surprisingly well, the same queries took only a few ms with the EWB server. So even though this is a kinda strange setup, I find it hard to believe that wine is at fault here. But I cannot really think of anything else either. Maybe someone with more experience ...

Last edited by Leslie7 (2016-10-06 05:29:15)

ab · 2016-10-06 06:29:46

Did you report this on the Zeos forum?
They are very reactive and helpful.

Leslie7 · 2016-10-06 07:46:43

No, not yet. I was hoping that maybe it is just a configuration issue with mORMot. Maybe something like connection pooling is required ... My other idea was that maybe the client not matching the server version could be the problem.

Anyway, I know a little more know. After migrating the database to linux native firebird 2.5 server - so wine is gone now - the speed is at least as terrible. So it is definitely not wine related.

I will check with a mORMotless Zeos only test app now ...

Leslie7 · 2016-10-06 08:05:01

The Zeos only test app is as fast as it can be expected. It seems like a mORMot issue now within TSQLDBZEOSConnectionProperties.Execute.

I have noticed one difference between fb 2.1 & 2.5.

TSQLDBZEOSConnectionProperties.Create

2.1 needed all information in the Servername Parameter.
2.5 worked when all params were assigned properly

Leslie7 · 2016-10-06 15:12:23

It is not TSQLDBZEOSConnectionProperties either. Sending a data url to the mORMot server directly the json answer is returned as fast as expected. Which leaves Nginx itself or the Nginx-mORMot server communication at suspect.

Last edited by Leslie7 (2016-10-06 19:14:01)

Leslie7 · 2016-10-06 19:12:28

It seems that Nginx holds back the answer because it is still waiting for something from the mORMot server. Which may never arrive, and in this case the answer is finally sent to the client after a timeout occurs. So the speed issue may be solvable by fine-tuning Nginx. But the problem could also be something missing from the mORMot response, or a missing low level signal ...

Last edited by Leslie7 (2016-10-06 19:18:42)

Leslie7 · 2016-10-06 20:44:31

Considering that EWB server works fine with NGinx as a reverse proxy it suggests that the mORMot server's response must be different in some small detail.

Leslie7 · 2016-10-07 09:40:37

After several misconceptions about what could be the root cause, finally I have the right answer.

With the help of wireshark it turned out that when Nginx forwards the request the header is set to close the connection. It has to be "keepalive" to have a fast response.
And this is how to do it.

Edit /etc/nginx/sitesenabled as root.

# Add upstream for keepalive
upstream http_backend {
# ip:port for the backend servers
server 127.0.0.1:8888;
server 127.0.0.1:8889;
# The number of inactive connections kept open. The oldest one is closed when the limit is reached.
keepalive 100;
}

server {
location /someurl {
proxy_pass http://127.0.0.1:8888;
# ....

# Add for keepalive START
proxy_read_timeout 300;
   proxy_connect_timeout 300;

      # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
   proxy_http_version 1.1;

   # Remove the Connection header if the client sends it,
   # it could be "close" to close a keepalive connection
   proxy_set_header Connection "";
# Add for keepalive END
}

I assume that EWB server worked out of the box because it ignores this part of the header ad keeps it alive anyway.

Last edited by Leslie7 (2016-10-07 09:44:51)

Leslie7 · 2016-10-07 10:42:15

Still not quite through yet.

For https connections this solution does not seem to work. Maybe it is easier to handle it on the mORMot side: is it possible to tell TTHttpServer to ignore this setting in the header and force it to "keepalive"?

Leslie7 · 2016-10-07 11:45:06

When Nginx forwards a https request to the mORMot server as a http request it ignores "proxy_http_version 1.1", which is required for keep-alive. Could be a bug. So it seems there is no mORMot only solution for this. Unless it is somehow possible to upgrade to v1.1 from the mORMot side.

ab · 2016-10-07 15:39:53

???
mORMot HTTP servers handles HTTP/1.1 requests.

Leslie7 · 2016-10-07 18:53:14

Yes, of course, but Nginx fills the header in the request for v1.0. Is it possible to handle it if it was v1.1 and 2keep it alive" regardless what the header sais. Since the EWB server did not require any extra configuration it worked with nginx out of the box, I assume this should be possible.

ab · 2016-10-07 19:14:49

I still do not understand the problem.
If the proxy does not support HTTP/1.1, mORMot should be able to handle HTTP/1.0 as expected...

Leslie7 · 2016-10-07 22:56:04

I think I can define the problem more clearly now:

Nginx can setup the request it is forwarding to the backend server several ways. These are the ones MORMot can handle and I have tried:
HTTP/1.0 (default) ---------------------------------------> mORMot HttpWebserver : after a ~5 sec timeout there is an error displayed on the console "Errno Recv()=11" , and after that the response gets delivered back to the proxy.
HTTP/1.1 (this requires certain settings for Nginx) --> mORMot HttpWebserver : instant reply

There are two possible solutions :

After setting up the reverse proxy for Nginx for forwarding both http & https incoming requests via HTTP/1.1. this is what happens:
a) incoming HTTP request - Nginx sends HTTP/1.1.
b) incoming HTTPS request - Nginx sends HTTP/1.0.

Since b) looks to be a bug I have created a bug report for the Nginx team.

While this works out I am looking at the mORMot side if the cause for "Errno Recv()=11" can be found and solved.

Last edited by Leslie7 (2016-10-07 23:15:21)

Leslie7 · 2016-10-08 08:12:17

For HTTP/1.0 requests Wireshark shows 5 frames before the Recv timeout occurs. I can send you the capture file if you want to investigate.

But for HTTPS the problem was something totally unexpected: I had kept the Nginx config file open in the editor for a while. Saved the file a couple of times and there was never any error. But after closing the editor and opening the file again all changes were gone. (The editor had root rights, there was no reason I am aware of for this.) So while I thought I was testing several settings for HTTPS, it turned out that it had the same partially setup configuration all the time. Brrrr.

Finally everything is working now perfectly with HTTP/1.1. Plus the responses from my new mORMot server feels snappier and uses less cpu time than the previous solution.

Last edited by Leslie7 (2016-10-09 22:31:21)

mORMot Open Source

#1 2016-10-06 00:03:36

FireBird 2.1 - extremely slow

#2 2016-10-06 06:29:46

Re: FireBird 2.1 - extremely slow

#3 2016-10-06 07:46:43

Re: FireBird 2.1 - extremely slow

#4 2016-10-06 08:05:01

Re: FireBird 2.1 - extremely slow

#5 2016-10-06 15:12:23

Re: FireBird 2.1 - extremely slow

#6 2016-10-06 19:12:28

Re: FireBird 2.1 - extremely slow

#7 2016-10-06 20:44:31

Re: FireBird 2.1 - extremely slow

#8 2016-10-07 09:40:37

Re: FireBird 2.1 - extremely slow

#9 2016-10-07 10:42:15

Re: FireBird 2.1 - extremely slow

#10 2016-10-07 11:45:06

Re: FireBird 2.1 - extremely slow

#11 2016-10-07 15:39:53

Re: FireBird 2.1 - extremely slow

#12 2016-10-07 18:53:14

Re: FireBird 2.1 - extremely slow

#13 2016-10-07 19:14:49

Re: FireBird 2.1 - extremely slow

#14 2016-10-07 22:56:04

Re: FireBird 2.1 - extremely slow

#15 2016-10-08 08:12:17

Re: FireBird 2.1 - extremely slow

Board footer