Blocking worker threads

squirrel · 2023-09-28 10:45:43

I have interface based services running on TRestHttpServer, which provide reports to third parties. One of the reports can be very slow and can take 15 or 20 seconds to complete. Is it possible to offload such a rest request to its own task without blocking the other worker threads or the http server?

Not sure I'm explaining it very clearly. To test for options, I've changed this sample \mORMot2\ex\ThirdPartyDemos\martin-doyle\04-InterfacedBasedServices and added the following function to the TExampleService rest interface:

function TExampleService.Time(var WaitSeconds: Integer): RawJson;
var
  StartTime: TDateTime;
begin
  StartTime := now;
  Sleep(WaitSeconds*1000);
  Result := '{"startTime": "' + DateToISO8601(StartTime) + '", "endTime": "' + DateToISO8601(now) + '"}';
end;

Then reduced the threadpoolcount from 4 to 1. The effect is, as expected, that several requests to this function are serialized, so running:

curl http://localhost:11111/root/example/Time?WaitSeconds=10

at the same time in different windows produce the following results:

{"result":[10,{"startTime": "2023-09-28T12:33:55.415Z", "endTime": "2023-09-28T12:34:05.420Z"}]}
{"result":[10,{"startTime": "2023-09-28T12:34:05.421Z", "endTime": "2023-09-28T12:34:15.430Z"}]}

This has the effect that the requests are executed one after the other, blocking new requests. My initial thought was that mORMot's new async http server would be ideal for this. But since these are rest requests already used by 3rd parties, I can not change that to websockets at this stage.

mpv · 2023-09-28 19:31:58

The simplest is to increase a threadPoolSize to 32 or 64

squirrel · 2023-09-28 20:59:45

Increasing the thread pool size is the quick solution, yes. But that misses the point. The point I was trying to make, is that there are different uses for rest calls on the same system. Some are quick and high priority, such as logging in and editing records. It is reasonable to expect them to complete in a few ms, and not consume 32 threads. However, there are other types of functionality that takes longer, but are lower priority, such as running complicated reports.

The idea is to be able to put "slow" type tasks in their own queue and not allow them to bring the whole rest server to a halt. If the thread pool size is simply increased to 32, then running 32 slow reports (or 1 slow report being used by 40 users) will effectively prevent other users from completing higher priority tasks such as logging in and editing records until those slow tasks have been completed.

If there is a functionality to create "tasks" for those requests, they can be placed in a queue or something to manage them, releasing them from the rest server's thread pool until they complete and perform a callback to instruct the rest server to complete the request.

mpv · 2023-09-29 06:10:01

We have a similar "reporting" service.
We have a reverse proxy what proxies all "fast" requests such as logging in and editing records to one instance, all "report" requests - to another.
First we scale our "report" instance by increasing threadPoolSize.
When this was not enough - by adding hosts + load balancer on reverse proxy level.
After the database became a bottleneck, we change the user API - some "reports" requests now processed to clients synchronously, but if we detect what calculating of the response MAY take a long time (by using some kind of heuristic) we answer to client with requestID, put request into durable queue (database table with row locking for a while, but will migrate to redis).
We have a pool of workers what pull a queue, process requests using database replicas, and put responses back into queue.
Client use pooling to detect what "report" for their requestID is ready.

Currently our load is ~100 "report" per second. Some reports are generated in 0.1 seconds, some - up to 20 minutes (hundreds of pages)

ab · 2023-09-29 08:53:06

You should switch from a blocking scheme to a non-blocking one.

Either use client polling (as mpv proposes) or use an interface callback for real-time notification.

Look at
https://github.com/synopse/mORMot2/blob … server.dpr
and
https://github.com/synopse/mORMot2/blob … client.dpr
IMHO this is exactly your use case. The sample uses a "sleep".

Callback interfaces make it pretty simple to code and work with such long term process.
For the execution, you could use threads (as with this sample), but you could use a queue if you don't want too many concurrent process.

squirrel · 2023-09-29 10:06:39

Thanks ab.

I did look at the long work sample before posting and it did look promissing. But it looks like that would require clients redevelop their existing solutions to use websockets.

Always difficult justifying breaking backwards compatibility to 3rd parties. Also not too sure how to modify almost 200 rest interfaces and their clients to fit into that workflow.

ab · 2023-09-29 16:36:09

If you can't use Websockets then polling is the only reasonable solution over HTTP.

HTTP/HTTPS was never meant to have staying connections, and ServerSideEvents is not to be advised.

Polling every second could be good enough.

mORMot Open Source

#1 2023-09-28 10:45:43

Blocking worker threads

#2 2023-09-28 19:31:58

Re: Blocking worker threads

#3 2023-09-28 20:59:45

Re: Blocking worker threads

#4 2023-09-29 06:10:01

Re: Blocking worker threads

#5 2023-09-29 08:53:06

Re: Blocking worker threads

#6 2023-09-29 10:06:39

Re: Blocking worker threads

#7 2023-09-29 16:36:09

Re: Blocking worker threads

Board footer