mORMot Open Source

Eugene Ilyin

Member

From: milky_way/orion_arm/sun/earth

Registered: 2016-03-27

Posts: 132

Website

TWinHttpAPI.OnDownload freeze during the data feeds fetching

Hi,

I have an issue with data feeds fetching from different APIs
with the current TWinHttpAPI.OnDownload implementation.

The issue is that data chunks are not provided to the client
immediately when delivered over the network but freeze in
almost infinite ReadData waiting.

To demonstrate the issue let's create some simple server
with 5 separate chunks response on each second and
simple TWinHttpAPI client to show received chunks on the console.

Minimal code to reproduce (sorry for the formatting, it's not trivial to show server/client code and avoid ab's long code warning )

program WinHttpAPIFeed; {$APPTYPE CONSOLE} {$I 'Synopse.inc'} uses SynCommons, SynCrtSock;
type
  THttpServerFeed = class(THttpServer) procedure Process(ClientSock: THttpServerSocket; ConnectionID: THttpServerConnectionID; ConnectionThread: TSynThread); override; end;
  TClientFeed = class function Download(Sender: TWinHttpAPI; CurrentSize, ContentLength, ChunkSize: Cardinal; const ChunkData): boolean; end;
procedure THttpServerFeed.Process(ClientSock: THttpServerSocket; ConnectionID: THttpServerConnectionID; ConnectionThread: TSynThread);
var Index: Integer; begin
  with ClientSock do begin
    SockSend('HTTP/1.1 200 OK'#$D#$A'Transfer-Encoding: chunked'#$D#$A); TrySockSendFlush;
    for Index := 1 to 5 do begin
      Sleep(1000); SockSend('10'#$D#$A'{"data":"feed"}'#$A); TrySockSendFlush;
    end;
    SockSend('0'#$D#$A#$D#$A); TrySockSendFlush; KeepAliveClient := False;
  end;
end;
function TClientFeed.Download(Sender: TWinHttpAPI; CurrentSize, ContentLength, ChunkSize: Cardinal; const ChunkData): boolean;
var Content: RawByteString; begin
  FastSetString(RawUTF8(Content), Pointer(@ChunkData), ChunkSize);
  Write(Content); Result := True;
end;
var Server: THttpApiServer; Client: TWinHttpAPI; ClientFeed: TClientFeed; OutHeader, OutData: SockString;
begin
  TAutoFree.Several([@Server, THttpServerFeed.Create('9000', nil, nil, '', 1),
    @Client, TWinHTTP.Create('http://localhost:9000'), @ClientFeed, TClientFeed.Create]);
  Server.AddUrl('/', '9000');
  Client.OnDownload := ClientFeed.Download;
  Client.Request('/', 'GET', 0, '', '', '', OutHeader, OutData);
  Writeln('Done'); Readln;
end.

In real life - feeds can be opened much longer than 5 seconds
with the data chunks occurred from time to time.

If you run the sample the OnDownload occurred only once after 5 seconds
period with all data fetched and once (because connection is closed), but not during
the each chunk delivery.

Solution

The problem is that QueryDataAvailable is not called before ReadData
as it shown is all Microsoft API code samples for ReadData.

Because we do not request current available data before the ReadData call we
have infinite freeze till connection close. This makes chunk by chunk fetching used
in feeds API impossible.

You can check that Windows HTTP API usage shows that QueryDataAvailable
is always called before ReadData to provide information of the current size
of bytes available to read and prevent reads from freezings.

You can check WinHttpQueryDataAvailable / WinHttpReadData samples used by TWinHTTP
or check InternetQueryDataAvailable used by TWinINet.

@ab, I've add fully backward compatible minor amendment to pull quest #307 to fix this and
makes OnDowload fetch feed chunks as expected.

Eugene Ilyin

Member

From: milky_way/orion_arm/sun/earth

Registered: 2016-03-27

Posts: 132

Website

Re: TWinHttpAPI.OnDownload freeze during the data feeds fetching

Hi ab,

Do you need some additional changes from my side to accept #307?
It changes just couple of lines:

Before:

      Bytes := InternalReadData(tmp,0);
      if Bytes=0 then
        break;

After:

      Bytes := InternalQueryDataAvailable;
      if Bytes=0 then
        break;
      Bytes := InternalReadData(tmp,0,Bytes);
      if Bytes=0 then
        break;

All SynSelfTests tests are passed.
No impact on perfomance and TWinHttpAPI.InternalReadData will not freeze anymore during the data feeds fetching.
Checked for all three cases ("OnDownload", "fetch  with Content-Length", and "fetch without Content-Length") for both implementations: TWinHTTP and TWinINet.