Extra Content-Range header in response for 206 Partial Content

ecm99 · 2021-01-19 14:24:12

I am using mORMot as a webserver (THttpApiServer) to serve up a Webm file for an HTML5 video tag.
The browser requests different ranges as the video is seeked.
The problem I am having is that the 206 response from mormot includes 2 different Content-Range headers in the response, with different byte ranges specified.
My theory is that one of them is coming from the http.sys and the other is added by this line "Resp^.AddCustomHeader(@ContentRange[1],Heads,false);" in THttpApiServer.Execute().

With both content-range headers in the response, my video does not play/seek correctly in the browser.
When I comment out that line, I only get one header in the response and the video plays correctly.

The correct filesize is actually 9812907 bytes (the value shown in the first Content-Range header)

I'll admit my understanding of the interaction with http.sys is limited, so I'm not sure if I'm on the right track; or totally off-base.

Platform: Windows 10

Request
-----
GET http://localhost:9550/trailer.webm HTTP/1.1
Host: localhost:9550
Connection: keep-alive
sec-ch-ua: "Google Chrome";v="87", " Not;A Brand";v="99", "Chromium";v="87"
Accept-Encoding: identity;q=1, *;q=0
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36
Accept: */*
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: video
Referer: http://localhost:9550/index.html
Accept-Language: en-US,en;q=0.9
Range: bytes=65536-

Response
-----
HTTP/1.1 206 Partial Content
Content-Type: video/webm
Content-Range: bytes 65536-9812906/9812907
Accept-Ranges: bytes
Server: Microsoft-HTTPAPI/2.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Origin, Content-Type, X-Auth-Token, X-Requested-With, Accept
Access-Control-Expose-Headers: origin, content-length,location,server-internalstate
Accept-Encoding: synlz,gzip
Date: Tue, 19 Jan 2021 14:07:52 GMT
Content-Length: 9681835
Content-Range: bytes 65536-9747370/9747371

ab · 2021-01-19 16:19:06

HTTP_RESPONSE.AddCustomHeader() should recognize the header, and set the proper field in KnownHeaders[reqContentRange] so http.sys should not add anything...

Could you verify that the values transmitted AddCustomHeader() are indeed the expected one?
What if you remove the AddCustomHeader() call?

ecm99 · 2021-01-19 17:09:26

Ok, I gathered a bit more info..

So the HTTP request specified a range of in the request header.
"Range: bytes=65536-"

With the AddCustomHeader() call present in the Execute() function, the response includes two ContentRange headers in the response.

Here is the response:
HTTP/1.1 206 Partial Content
Content-Type: video/webm
Content-Range: bytes 65536-9812906/9812907
Accept-Ranges: bytes
Server: Microsoft-HTTPAPI/2.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Origin, Content-Type, X-Auth-Token, X-Requested-With, Accept
Access-Control-Expose-Headers: origin, content-length,location,server-internalstate
Accept-Encoding: synlz,gzip
Date: Tue, 19 Jan 2021 16:46:53 GMT
Content-Length: 9681835
Content-Range: bytes 65536-9747370/9747371

Both Content-Range headers specify the content starts at 65536 (correct) and they both have different lengths.
The top Content-Range one is the one generated by the AddCustomHeader call. It goes up until 9812906 which is the end of the file.
The second Content-Range header is one I'm presuming is added by http.sys. It only goes up to 9747370 (not quite the end of the file).
The Content-Length header is valid only for the second Content-Range header.
So it looks like http.sys is adding a content-range with a different range than addcustomheader(), and the content-length field matches the one http.sys added.

I'm trying to sniff the connection to see how many bytes of actual payload get sent (I'm assuming content-length is correct).

When I remove the AddCustomHeader() call, the response looks like this. Only one content-range line, and the content-length matches it correctly.

HTTP/1.1 206 Partial Content
Content-Type: video/webm
Accept-Ranges: bytes
Server: Microsoft-HTTPAPI/2.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Origin, Content-Type, X-Auth-Token, X-Requested-With, Accept
Access-Control-Expose-Headers: origin, content-length,location,server-internalstate
Accept-Encoding: synlz,gzip
Date: Tue, 19 Jan 2021 16:49:16 GMT
Content-Length: 9681835
Content-Range: bytes 65536-9747370/9747371

Edit: The length of the payload returned was correct for the Content-Length header that was specified.

Last edited by ecm99 (2021-01-19 17:14:06)

ab · 2021-01-19 18:40:42

So the AddCustomHeader() values are correct, but the values added by http.sys are not?

ecm99 · 2021-01-19 19:10:29

No, I would say the opposite.

The amount of payload data returned matches the one added by http.sys, and also the content-length header that is added to the response also matches the content-range header that was added by http.sys.
The content-range from AddCustomerHeader() (65535-981206) claims there should be a payload of 9747370 bytes; but only 9681835 bytes of payload are sent.

Last edited by ecm99 (2021-01-19 19:11:08)

ecm99 · 2021-01-19 19:18:02

In my brief research, I believe how your code works is you pass the flag HTTP_SEND_RESPONSE_FLAG_PROCESS_RANGES to winhttp, as well as the file handle to the complete file; and http.sys will then extract the appropriate range of bytes out of the file and return it? If my understanding is correct, I believe HTTP.sys is deciding it wants to send less data than the CustomHeader you add, so it adds a content-range with a smaller range and sets the content-length to the amount of data it decides to send.. or something like that..

I could try to make a small sample to allow you to replicate the issue; would you like that?

ecm99 · 2021-01-20 17:08:12

FYI, I sent a sample application to you by email; hopefully it came through. Let me know if you need any more info. For now I am just going to comment AddCustomHeader() as it seemingly resolves my issue.

ecm99 · 2021-02-08 18:51:05

Ok, I got back to this and did some more testing/research.. It turns out the HTTP_SEND_RESPONSE_FLAG_PROCESS_RANGES flag is actually the source of the issue.
Turning on that flag causes http.sys to add the Content-Range header itself (even if its already in the known headers). If you dont set that flag, it still uses the StartingOffset and Length supplied in the HTTP_DATA_CHUNK_FILEHANDLE structure; it just doesnt add the Content-Range header to the reply. The MSDN docs don't really make any mention of this fact.

I found an example if the chromium source of where they uses HttpSendHttpResponse to response with a 206 reply; and you can see they set the starting offset and length, and then call HttpSendHttpResponse without using the HTTP_SEND_RESPONSE_FLAG_PROCESS_RANGES flag; and they also manually add the Content-Range header to the response.

So, after all this, I believe all you need to do is remove that flag.

The other thing that I've added is a check to see if the StartingOffset is >= OutContentLength. If it is I send back 416 (range not satisfiable) as the response; instead of sending back 206.

mpv · 2021-02-08 20:15:55

I can recommend to use an nginx as a reverse proxy for you service even if you are on Windows and send files using X-Accel-Redirect header as described in THttpServer.NginxSendFileFrom
We use such technic for video on production. There are no any issue with Content-Range in nginx - you'll get it automatically.

As a side effect you'll get more secure (TLS1.2), faster (http2 / http3) and stable solution. And with rtmp module you'll get HLS for free

Also take into account what any vulnerability in HTTP.SYS is a way to code execution in the kernel mode (just google for "vulnerability in HTTP.SYS")

ab · 2021-02-08 20:50:30

@ecm99

I am a bit lost with what should be done...
Could your propose a pull request for the changes?

mORMot Open Source

#1 2021-01-19 14:24:12

Extra Content-Range header in response for 206 Partial Content

#2 2021-01-19 16:19:06

Re: Extra Content-Range header in response for 206 Partial Content

#3 2021-01-19 17:09:26

Re: Extra Content-Range header in response for 206 Partial Content

#4 2021-01-19 18:40:42

Re: Extra Content-Range header in response for 206 Partial Content

#5 2021-01-19 19:10:29

Re: Extra Content-Range header in response for 206 Partial Content

#6 2021-01-19 19:18:02

Re: Extra Content-Range header in response for 206 Partial Content

#7 2021-01-20 17:08:12

Re: Extra Content-Range header in response for 206 Partial Content

#8 2021-02-08 18:51:05

Re: Extra Content-Range header in response for 206 Partial Content

#9 2021-02-08 20:15:55

Re: Extra Content-Range header in response for 206 Partial Content

#10 2021-02-08 20:50:30

Re: Extra Content-Range header in response for 206 Partial Content

Board footer