#1 2025-02-03 12:00:03

itSDS
Member
From: Germany
Registered: 2014-04-24
Posts: 525

unexpected request timeouts

Hi Arnaud, this topic is not very easy, but i can not find the reason.

Using a m2 TRestHttpServer with WEBSOCKETS_DEFAULT_MODE and TRestHttpClientSocket only http connection
we get unexpected request timeouts on client site.

In Log we can see timestamp request to server, auth, instance and contract request.
But then using one of the interface function sometimes gives "request timeout"

On Server site we can see all as normal. but if request timeout nothing is in log on server site.
We checked firewall / content filter there should be nothing.

Do you have any idea what else we can check / Enable any extra log or so ?


Rad Studio 12.1 Santorini

Offline

#2 2025-02-03 15:08:22

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,845
Website

Re: unexpected request timeouts

What do you call "request timeout"?
Is it a HTTP_TIMEOUT = 408 error code?

On server side:
Is it visible on the server side, via TRestServerUriContext.ExecuteCommand() as TimeOut method - i.e. InternalLog('TimeOut %.Execute(%) after % ms' ?
Do you use the HeaderRetrieveAbortDelay option?

On client side:
It may come from a long SockReceivePending()=cspNoData line 2373 of THttpClientSocket from mormot.net.client.
Is it the case?
As written in the comment:

            // timeout may happen not because the server took its time, but
            // because the network is down: sadly, the socket is still reported
            // as OK by the OS (on both Windows and POSIX)

Edit: Please try with https://github.com/synopse/mORMot2/commit/0a2c8c467

Offline

#3 2025-02-03 15:18:59

danielkuettner
Member
From: Germany
Registered: 2014-08-06
Posts: 374

Re: unexpected request timeouts

@ab
Today we had "same" behavior like @itSDS (I'm looking for errors on our side first and since weeks), but it's not a client issue on our side.
1. service isn't responsive any more (/timestamp over curl gives timeout too).
2. absolut no cpu-usage in any thread
3. no exceptions or error in (verbose) log file before.
4. absolut no timeout in postgres.

Our service runs since yesterday evening and we had about 40' request per hour till the issue at 2:20 pm.
We are using mORMot commit #fe67be7b8 / heap.inc / linux (fpc 3.2.0).

Last edited by danielkuettner (2025-02-03 16:05:54)

Offline

#4 2025-02-03 16:48:59

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,845
Website

Re: unexpected request timeouts

@daniel
Could you send me a link to some server logs in verbose mode by email (to keep them private), so that I could look into them?

Offline

#5 2025-02-03 17:10:43

danielkuettner
Member
From: Germany
Registered: 2014-08-06
Posts: 374

Re: unexpected request timeouts

Ok

Offline

#6 2025-02-03 19:20:01

itSDS
Member
From: Germany
Registered: 2014-04-24
Posts: 525

Re: unexpected request timeouts

@AB yes its a 408 error. I made a documented client / Server / Log and sent it to you also.


Rad Studio 12.1 Santorini

Offline

#7 2025-02-04 08:17:12

itSDS
Member
From: Germany
Registered: 2014-04-24
Posts: 525

Re: unexpected request timeouts

Hi Arnaud, i checked all of this:

ab wrote:

What do you call "request timeout"?
Is it a HTTP_TIMEOUT = 408 error code?

Yes

ab wrote:

On server side:
Is it visible on the server side, via TRestServerUriContext.ExecuteCommand() as TimeOut method - i.e. InternalLog('TimeOut %.Execute(%) after % ms' ?

No on Server site i can see nothing

ab wrote:

Do you use the HeaderRetrieveAbortDelay option?

No

ab wrote:

On client side:
It may come from a long SockReceivePending()=cspNoData line 2373 of THttpClientSocket from mormot.net.client.
Is it the case?

I don't know, i will add DoLog To OnLog

And the Connection is still alive - Server is responding to other clients
Client is sending next Request if i start any

The Only Thing i observed is, that the Packed is "bigger" lets say 600KB to 2MB


Rad Studio 12.1 Santorini

Offline

#8 2025-02-04 08:42:29

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,845
Website

Re: unexpected request timeouts

itSDS wrote:

And the Connection is still alive - Server is responding to other clients
Client is sending next Request if i start any

@StefanDS
So your case seems different to Daniel's, who has a stuck server with no response at all

The Only Thing i observed is, that the Packed is "bigger" lets say 600KB to 2MB

Interresting.
Could you send me the full log by email (link?), please?

Offline

#9 Yesterday 15:50:15

danielkuettner
Member
From: Germany
Registered: 2014-08-06
Posts: 374

Re: unexpected request timeouts

Hello Arnaud,

I could find the location of our culprit today.
But I'm not able to understand it by now and have to test more.

Thanks
Daniel

Offline

#10 Today 12:42:23

itSDS
Member
From: Germany
Registered: 2014-04-24
Posts: 525

Re: unexpected request timeouts

Hi Arnaud yes i think daniel has an other problem.
I sent you new Logs today with low level logs
And yes my Problem is primary with big data packets (exact size unkown) but its reproducable in different client server scenarios.


Rad Studio 12.1 Santorini

Offline

Board footer

Powered by FluxBB