Read Time Out

mdbs99 · 2020-12-08 15:15:09

I'm using the default configuration for my HTTP Server (i.e., 32 threads, etc...) and I got some issues from some consumers in Java: Read Time Out error.

I got nothing on log so, I imagine that all threads are working while a new request came and the server throw away after a while... (?)
My question is: how can I get aware about these requests that weren't process?

Thanks.

mpv · 2020-12-08 15:44:48

On Windows you can monitor (using perfmon) a HTTP Service Request Queues
On Linux - THTTPServer.ThreadPool.PendingContextCount & THTTPServer.ThreadPool.ContentionAbortCount

mdbs99 · 2020-12-09 11:15:06

I would like to log all in the app server. Is that any chance to mORMot do that automatically, as it does for all requests?

My issue is that a consumer says that he/she got a time out, but I have nothing in my logs because the request never touch the business rules (my code, not mORMot) at all. I would like to see how many requests have not been process to make changes based on this information.

ab · 2020-12-09 14:40:12

Are you using a reverse proxy in-between?
Are you running on Windows?

On Windows, the http.sys Web server is very stable, so I doubt there is any problem on that level.

If your requests take 100ms to the DB, perhaps there is some contention on server side.

pvn0 · 2020-12-09 14:40:56

mdbs99 wrote:

I would like to log all in the app server. Is that any chance to mORMot do that automatically, as it does for all requests?

Not atm, but you can add it, follow my instructions here.

This is only going to work if your application is actually handling the connection. If the thread contention is high then you should see it in the logs after applying that fix.

That much said, you should write your service code to return as soon as possible. In other words you should not have any code in your service method that could potentially block for a longer period of time.
If this is indeed what is happening then for a cheap and dirty fix you could increase the number of threads in the threadpool and see if that helps but that's just avoiding the issue.

If you need to perform some blocking work then you should queue that work and simply tell the client to pool the server in an interval until that work is done.

ab · 2020-12-09 16:49:50

@pvn0
I guess mdbs99 is on Windows, and here the http.sys server won't be taken into account by your instructions.

@mdbs99
Did you check https://docs.microsoft.com/en-us/window … or-httpsys as mpv proposed?

pvn0 · 2020-12-09 21:05:17

@ab that's true and I considered that but afaik he never said he is using http.sys also in another thread he said he is using plain mORMot so who knows

mdbs99 · 2020-12-10 02:48:29

ab wrote:

Are you using a reverse proxy in-between?
Are you running on Windows?
On Windows, the http.sys Web server is very stable, so I doubt there is any problem on that level.
If your requests take 100ms to the DB, perhaps there is some contention on server side.

No proxy. Yes, on Windows.
Yes the server is very stable, but it seems that my app is receiving more requests that it can handle.
Unfortunately, responses are taking ~1.5s in average. That is because my app is calling another microservice (Java) to do some stuff and it takes almost 1s to do it — I'm working to remove such dependency...

---

pvn0 wrote:

Not atm, but you can add it, follow my instructions here.

Thanks. But as @ab and I has said, I'm using Windows.

---

ab wrote:

@mdbs99
Did you check https://docs.microsoft.com/en-us/window … or-httpsys as mpv proposed?

I looked quickly... I didn't know that tool. Sounds interesting... but as I understood, it just allowing me monitoring in real-time but no logging.
My problem is that they are complain about time out. But as I don't have any logs, because the request didn't touch my code, it looks like perfect from my point of view... so, I need (or I guess I need it) to have logs for those requests as a prove that they have happened.

pvn0 · 2020-12-10 08:22:57

You still haven't confirmed if you are using http.sys?

ab · 2020-12-10 10:06:36

From the existing logs, you should be able to find out how much time each request take, and how the thread pool is used.
If indeed the Enter/Leave of TSQLRestServer.URI takes a lot of time, and all threads are used, then you have a thread retention.
Since the problem is that there is a blocking process, and you need to wait, it is difficult to change anything.

You could try to increase the number of threads from 32 to 256 (which is the highest value allowed). It would consume a little more memory, but may help preventing timeouts.

mdbs99 · 2020-12-10 11:58:38

pvn0 wrote:

You still haven't confirmed if you are using http.sys?

Yes, I'm.

ab wrote:

From the existing logs, you should be able to find out how much time each request take, and how the thread pool is used.
If indeed the Enter/Leave of TSQLRestServer.URI takes a lot of time, and all threads are used, then you have a thread retention.
Since the problem is that there is a blocking process, and you need to wait, it is difficult to change anything.
You could try to increase the number of threads from 32 to 256 (which is the highest value allowed). It would consume a little more memory, but may help preventing timeouts.

I've already increased the number of threads from 32 to 64 and I didn't have complains so far.

This code is the same I've talked in another thread: a rewrite of code from Desktop perspective to Web. Some process take 500ms and that is great in Deskop, but not for Web. I still have a lot work to do...

mORMot Open Source

#1 2020-12-08 15:15:09

Read Time Out

#2 2020-12-08 15:44:48

Re: Read Time Out

#3 2020-12-09 11:15:06

Re: Read Time Out

#4 2020-12-09 14:40:12

Re: Read Time Out

#5 2020-12-09 14:40:56

Re: Read Time Out

#6 2020-12-09 16:49:50

Re: Read Time Out

#7 2020-12-09 21:05:17

Re: Read Time Out

#8 2020-12-10 02:48:29

Re: Read Time Out

#9 2020-12-10 08:22:57

Re: Read Time Out

#10 2020-12-10 10:06:36

Re: Read Time Out

#11 2020-12-10 11:58:38

Re: Read Time Out

Board footer