how to break/reject unplanned heavy request

jaclas · 2017-10-26 07:44:40

I have a service with a method for uploading files.

TFileDownloadUploadRest = class(TRestService)
[...]
procedure Upload(Ctxt: TSQLRestServerURIContext);

The service uses another method to register a file, which is then uploaded using the Upload() method.
However, I would like to protect server from receiving unregistered (and often big!) file, to not waste the transfer, because only after receiving the whole file the server can reject it.
Is it possible to intercept the start of a transfer (call Upload()) and break it when transfer is unplanned?

Last edited by jaclas (2017-10-26 12:06:08)

ab · 2017-10-26 13:49:40

When requests do arrive to TSQLRestServer.URI, the body has been fully downloaded in the HTTP server class.

There is currently no callback at HTTP server level, to check for headers, and reject the request.
I will add this feature.

jaclas · 2017-10-26 13:52:39

Thx, I am looking forward to this extension!

mpv · 2017-10-26 15:06:31

@AB - please, look at branch HTTPServerEvents - I'm implement where all necessery events (for HTTP.sys based server). branch description.
I'm use it on hi-load productions without problems and will be very happy if it be merged to trunk

ab · 2017-10-26 15:21:19

I've just introduced THttpServerGeneric.OnBeforeBody event, allowing request abortion just after the headers are retrieved.
See https://synopse.info/fossil/info/e9ae45e587

It will work for both Socket and HTTP.sys servers.

@mpv
Yes I will merge the branches, but I was not pleased by the fact that new units were introduced in the branch.

mpv · 2017-10-26 15:31:15

Ups. I try to merge HTTPServerEvents with trunk, but you implement a OnBeforeBody event. @AB, please, take a look in my implementation - it add 4 event, one of it is near the same as OnBeforeBody
Code from branch:

/// event handler called after HTTP request headers retrieved
    // but before HTTP body is readed from a socket.
    // The main purpose is to analyse a request headers (for example Content-Length)
    // and make some decisions BEFORE recive a request body.
    // In case handler return a non 2XX code server will send a responce to client
    // without reading a request body (for example 413 Payload Too Large)
    // - warning: this handler must be thread-safe (can be called by several
    // threads simultaneously)
    property OnAfterHeaderRetrieved: TOnHttpServerRequest

Last edited by mpv (2017-10-26 15:33:02)

mpv · 2017-10-26 15:45:04

I'm merge a trunk (without OnBeforeBody comit) into HTTPServerEvents. AB - this branch don't introduce a new units (another one WinWebSocket - does).

ab · 2017-10-26 15:54:17

The OnAfterHeaderRetrieved() signature can't work with socket-based service, since at that time there is no THttpServerRequest instance.

But we may merge the other callbacks.

jaclas · 2017-10-27 07:11:03

I try use this new event, but... something is wrong.
My event body:

function TssHTTPServer.OnBeforeBodyEvent(const aURL,aMethod,aInHeaders, aInContentType,aRemoteIP: SockString; aContentLength: integer): Cardinal;
begin
  TSQLLog.Family.SynLog.Log(sllDebug, 'TssHTTPServer.OnBeforeBodyEvent(%, %)', [aURL,aMethod]);
  Result := STATUS_NOTFOUND;
end;

And I send from IE post requests:

first is catched:

2017-10-27 08:52:35.50 debug TssHTTPServer.OnBeforeBodyEvent(/file/upload/service/7865865875fgfiug786tg76, POST)

...but after this IE still wait for response for looong time (turns the wheel)...

next request from IE is ...handled normally!

2017-10-27 09:08:30.39  +    ss.Rest.FileDownloads.TFileDownloadUploadRest(08063CC0).URI(POST file/upload/service/7865865875fgfiug786tg76 inlen=37516350)

what is wrong?

Last edited by jaclas (2017-10-27 07:12:50)

ab · 2017-10-27 07:15:53

Using Sockets or HTTPAPI server class?
Which version of IE?

jaclas · 2017-10-27 07:24:17

HTTPAPI and IE11 (from win7)

jaclas · 2017-10-27 09:35:35

Imho problem is in server constructor and created threads without event handler:

if ServerThreadPoolCount>1 then
THttpApiServer(fHttpServer).Clone(ServerThreadPoolCount-1);

ab · 2017-10-27 18:32:58

Please check https://synopse.info/fossil/info/f81cd8395f

mpv · 2017-10-28 09:53:15

ab wrote:

The OnAfterHeaderRetrieved() signature can't work with socket-based service, since at that time there is no THttpServerRequest instance.
But we may merge the other callbacks.

May be modify the THttpServer.Execute method and include where code from ClientCrtSock.GetRequest? In this case we can create a Context: THttpServerRequest inside the THttpServer.Execute like it done in the THttpApiServer.Execute. IMHO this make both server implementation more similar..

jaclas · 2017-10-28 14:00:45

Requests eject now works properly.

But IE strangely handle the reject. Maybe this is a IE11 bug.

mpv · 2017-10-28 14:27:38

Returning of 404 Not Found is wrong from HTTP POV. Valid responce code is 413 in your case..

Last edited by mpv (2017-10-28 14:28:09)

jaclas · 2017-10-29 08:48:55

413 does not change anything, IE still behaves unpredictably.

ab · 2017-10-29 09:13:35

I've just introduced THttpServerGeneric.MaximumAllowedContentLength property, which should work on any kind of Http server.
It returns STATUS_PAYLOADTOOLARGE = 413 error if "Content-Length" header overflows this value.
See https://synopse.info/fossil/info/9b189d64e0

jaclas · 2017-10-29 13:04:58

Hmmm... still exist some problems.
I add this line:

 Self.fHttpServer.MaximumAllowedContentLength := 5000;

and I post 4 times the same request, in log I get:

2017-10-29 13:55:30.54 trace TFileDownloadUploadRest(07FA3CC0) EndCurrentThread(THttpApiServer) ThreadID=00000430 ThreadCount=-1
2017-10-29 13:55:49.45 trace TFileDownloadUploadRest(07FA3CC0) EndCurrentThread(THttpApiServer) ThreadID=000011BC ThreadCount=-2
2017-10-29 13:55:59.47 trace TFileDownloadUploadRest(07FA3CC0) EndCurrentThread(THttpApiServer) ThreadID=00000E5C ThreadCount=-3
2017-10-29 13:56:08.54 trace TFileDownloadUploadRest(07FA3CC0) EndCurrentThread(THttpApiServer) ThreadID=00001220 ThreadCount=-4

Each request release the thread, probably due to an internal exception:

Last edited by jaclas (2017-10-29 15:40:54)

ab · 2017-10-29 17:44:58

Please check https://synopse.info/fossil/info/7de5aa5961
There was indeed a problem in my implementation.

jaclas · 2017-10-30 07:47:05

Looks like it's ok now, big thanks Arnaud!

mORMot Open Source

#1 2017-10-26 07:44:40

how to break/reject unplanned heavy request

#2 2017-10-26 13:49:40

Re: how to break/reject unplanned heavy request

#3 2017-10-26 13:52:39

Re: how to break/reject unplanned heavy request

#4 2017-10-26 15:06:31

Re: how to break/reject unplanned heavy request

#5 2017-10-26 15:21:19

Re: how to break/reject unplanned heavy request

#6 2017-10-26 15:31:15

Re: how to break/reject unplanned heavy request

#7 2017-10-26 15:45:04

Re: how to break/reject unplanned heavy request

#8 2017-10-26 15:54:17

Re: how to break/reject unplanned heavy request

#9 2017-10-27 07:11:03

Re: how to break/reject unplanned heavy request

#10 2017-10-27 07:15:53

Re: how to break/reject unplanned heavy request

#11 2017-10-27 07:24:17

Re: how to break/reject unplanned heavy request

#12 2017-10-27 09:35:35

Re: how to break/reject unplanned heavy request

#13 2017-10-27 18:32:58

Re: how to break/reject unplanned heavy request

#14 2017-10-28 09:53:15

Re: how to break/reject unplanned heavy request

#15 2017-10-28 14:00:45

Re: how to break/reject unplanned heavy request

#16 2017-10-28 14:27:38

Re: how to break/reject unplanned heavy request

#17 2017-10-29 08:48:55

Re: how to break/reject unplanned heavy request

#18 2017-10-29 09:13:35

Re: how to break/reject unplanned heavy request

#19 2017-10-29 13:04:58

Re: how to break/reject unplanned heavy request

#20 2017-10-29 17:44:58

Re: how to break/reject unplanned heavy request

#21 2017-10-30 07:47:05

Re: how to break/reject unplanned heavy request

Board footer