Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

Eugene Ilyin · 2016-04-18 01:53:03

Hi, I found a strange behavior of Ctxt.ReturnFile method in mORMot 1.18.1530 when worked under useHttpSocket (THttpServer) server.

Let's have a simple method-based service with 304-support:

TTestServer = class(TSQLRestServerFullMemory)
published
  procedure Q123(Ctxt: TSQLRestServerURIContext);
end;

procedure TTestServer.Q123(Ctxt: TSQLRestServerURIContext);
begin
  Ctxt.ReturnFile('D:\123.html', True);
end;

When you request /q123 the static content is received by browser but not GZipped (SynZip.CompressGZip is registered but not called).

The issue is in SynCrtSock.pas:

procedure THttpServer.Process
...
3681:    // handle case of direct sending of static file (as with http.sys)
3682:    if (Context.OutContent<>'') and (Context.OutContentType=HTTP_RESP_STATICFILE) then
...
3690:          Context.OutContentType := ''; // 'Content-type: ...' in OutCustomHeader

So we identify that we want to send static file and use Context.OutContentType for this need, after that we clear this field.

Then we call CompressDataAndWriteHeaders:

3732:    ClientSock.CompressDataAndWriteHeaders(Context.OutContentType,Context.fOutContent);

Then we call CompressDataAndGetHeaders:

procedure THttpSocket.CompressDataAndWriteHeaders
...
4116:    OutContentEncoding := CompressDataAndGetHeaders(fCompressHeader,fCompress,
4117:      OutContentType,OutContent);

And inside CompressDataAndGetHeaders:

function CompressDataAndGetHeaders
...
2376:  if (integer(Accepted)<>0) and (OutContentType<>'') and (Handled<>nil) then begin

As you see because .OutContentType was cleared and actual content type stored in .OutCustomHeader we didn't apply compression to response data.
That decision to use .OutContentType for 2 different ways as static file identification and as content type provider violates SRP (Single Responsibility Principle) and prevent compression of static files.

I see three possible solutions:

Restore .OutContentType in 3690 from .OutCustomHeader;
Provide separate approach for static files identification to keep SRP;
Maybe I do smth wrong and mORMot has an alternative method to return compressed static files.

Thanks

P. S.
Btw, the 304 response works perfectly in "ETag / If-None-Match" combination, but Chrome didn't add If-None-Match: for second request, anyway this is an another issue (maybe because of cache control HTTP header absent in default HTTP response).

esmondb · 2016-04-18 06:35:02

Probably the 'static content compression' feature needs to be turned on in IIS.

ab · 2016-04-18 07:41:03

@esmondb
AFAIK the IIS specific settings have nothing to do with the http.sys server.
And here Eugene Ilyin is talking about the Sockets server, not the http.sys server.

@Eugene
I think I've fixed the problem with useHttpSocket.
See http://synopse.info/fossil/info/669ae30f9a
But I suspect the problem would exist for THttpApiServer, which returns the file directly with no gzip - it is not feasible since it is not handled by the http.sys kernel mode, unless you create and return a .gz file.

Eugene Ilyin · 2016-04-18 07:45:55

Hi esmondb,

esmondb wrote:

Probably the 'static content compression' feature needs to be turned on in IIS.

If you mean useHttpApiRegisteringURI HTTP server (THttpApiServer) based on http.sys windows kernel library - it doesn't gzip files too.
This is because of performance optimization introduced by mORmot overlord. He preferred delegate file transfer to windows kernel as the best possible performance solution.
But http.sys doesn't compress files too.

As for your IIS suggestion to use StaticCompressionModule (based on Inetsrv\Compstat.dll) this is IIS specific feature available in IIS only, but not in mORMot framework.
mORMot doesn't rely or require IIS presence on windows to run.

It's interesting that 30 - MVC Server example based on mustache templates gzip all content:

Request:
  Host: localhost:8092
  User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  Accept-Language: en-US,en;q=0.5
  Accept-Encoding: gzip, deflate
  Etag: "23C0189A"

Responce:
  Accept-Encoding: gzip
  Content-Encoding: gzip
  Content-Length: 4258
  Content-Type: text/html; charset=UTF-8
...

So the mustache HTTP processor have to compute all page html/js/css content from templates and give it back as a cached data stream but not as a static file.
This decision excludes us from kernel windows http.sys sending but gives us gziped computed content based on mustache.

Interesting that blog.css request to /blog/.static/blog.css is gzipped too from 2.35 Kb file content to 0.98 Kb of actual network traffic.
The answer is that blog.css bypass windows kernel file sending and Context.OutContentType in THttpApiServer.Execute not equal HTTP_RESP_STATICFILE but "text/css".

Interesting, how is .static mustache data cached (like blog.css file):

Computed each time in every request to check against crc32c ETag value
Has some timeout to save CPU for unnecessary computation and do it periodically
Checks .static/* file date/time modification to update memory cache and recompute ETag
Subscribed to file system and hooked when file modified to update internal cache and recompute ETag
Has some lazy and permanent crc32c computation which is done only ones per first request and then always give the same static data from memory

Eugene Ilyin · 2016-04-18 08:14:33

Hi ab,

Thanks for suggested patch, what is the secret of so fast code production?
I think that I spend an hour for accurate cache tests creation (only static file preparing in tests and 304-regression checks take a lot of time for me).

As you mentioned the patch is relevant only to custom THttpServer.
For http.sys solution the .gz file preparation and handling on user temp directory required (to use the best from two worlds: the windows kernel mode file transfer and GZiped compression with 304-ETags cache).

I think about some high level delegate in function Request(Context: THttpServerRequest): Cardinal; override; in TSQLHttpServer.

Something like TStaticResourcceSQLHttpServer or TCachedSQLHttpServer with ability:

Run separate thread in background and decompress RCDATA archive included into .exe as RCDATA resource to folder:
CSIDL_APPDATA/EXE Camel Case Name With Spaces/Cache or
USER_TEMP_FOLDER/EXE Camel Case Name With Spaces/Cache
Creates for each html/css/js files filename.ext.gz variant or get it from RCDATA directly
Keep in memory only hashed list of URI/File/FileGZ/ETag/Modified records but not the actual files
In Request(Context: THttpServerRequest) method before inherited behavior check URI against hashed list and return the file with http.sys kernel mode or send 304 when If-None-Match presence found.

ab · 2016-04-18 12:32:33

See also http://synopse.info/fossil/info/5062fe031e

Perhaps storing the content directly gzipped in the resource would be the best idea.
It costs much less to decompress than to compress.
And the resulting .exe would be smaller.

An alternative may be to use a .zip file, but deflate is less supported (more bugged? deprecated?) than gzip in browsers AFAIR.
See "GZIP Vs. DEFLATE" paragraph at https://zoompf.com/blog/2012/02/lose-th … ompression

In fact, a dedicated set of classes may be needed for this process.
Not sub-class the THttpServer itself.
So having pre-gzip content may work not only for http.sys but also for sockets server.

igors233 · 2016-04-18 23:53:37

This seems to affect http://synopse.info/fossil/tktview?name=ecabf279e9 ticket as well...

IMHO for non static files it would be good to have kind of callback mechanism so that app could decide wheter file should be compressed or not.
Default logic based on file type is good but not good enough in some cases when server could return a txt file but with custom extension or something like that.

Storing static files as embedded resource in exe may not solve all cases, for example I serve static files from few folders whose content is changed outside of server app control, so it would be good to have a solution for such cases as well.

Last edited by igors233 (2016-04-18 23:58:01)

Eugene Ilyin · 2016-04-19 02:43:43

Hi ab,

We can store all static resources not only html/css, but all related html/css/js/txt/svg files and mustache (Views) templates in the packed RCDATA-format in exe.
For this internal archive we can use any appropriate compression: .gz, .zip, .bz2, .7z, or even the most extreme in the world .paq8px.

What we need - is to decompress all this static data into .gz and plain files somewhere on production disk (may be My Documents folder / User temp folder / User AppData Roaming folder) - because we want http.sys to do all dirty work for file handling/sending. That's why we need both variants of each file (compressed .gz and not compressed plain) depends on client browser HTTP request 'Accept-Encoding: ' parameter.

We need to have two ways for file update checks (by modification DateTime or by content hash (SHA2-256 or SHA2-512 or even SHA3))
We need to have four options: check file updates on every request / periodically / subscribe to FileSystem about archive folder change modifications / don't check
We need to decide now to decompress all file in the first run: in main thread blocking manner or in background thread manner
We need to decide where store cache.db TSQLite3 file with FileName,Hash,DateTime table database and updated it according to folder changes.
Recomputed cache.db is like single fossil database without content, but only file descriptors - this database can be part of this archive too.

All this gives us zero maintenance (just run single .exe) and instant deployment (just replace .exe and restart) abilities.
Also we don't need Views mustache coping during production deployment - all this will be unpacked from RCDATA internal archive.

Eugene Ilyin · 2016-06-08 23:43:20

Hi ab,

Now all resources can be synlz-packed, embedded and returned in memcached manner with TBoilerplateHTTPServer class instance.

Even more: the assets can be pre-compressed and cached as files on production server, this allows delegate file transfers to low-level HTTP api (like http.sys) even gzipp'ed variants!

And now you can embed and unpack on production your .map / .mab project files (single file run/distribution strategy).

Unfortunately there is no way to ask TSynLog* get info not from disk but from or in-memory *.mab file during stack-trace logout.
Maybe ab will add this feature in the next releases.

ab · 2016-06-09 07:09:53

I do not understand why the .map/.mab should not be part of the .exe, using the map2exe tool?

Eugene Ilyin · 2016-06-09 11:10:33

Oh, you right.
I missed that Map2Mab.dpr can embed symbol mapping it into exe.

mORMot Open Source

#1 2016-04-18 01:53:03

Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

#2 2016-04-18 06:35:02

Re: Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

#3 2016-04-18 07:41:03

Re: Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

#4 2016-04-18 07:45:55

Re: Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

#5 2016-04-18 08:14:33

Re: Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

#6 2016-04-18 12:32:33

Re: Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

#7 2016-04-18 23:53:37

Re: Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

#8 2016-04-19 02:43:43

Re: Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

#9 2016-06-08 23:43:20

Re: Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

#10 2016-06-09 07:09:53

Re: Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

#11 2016-06-09 11:10:33

Re: Why HTTP_RESP_STATICFILE (Ctxt.ReturnFile) not gzipped?

Board footer