Server-side large file upload (issue #292) — proposal

sh17 · 2026-07-17 07:41:42

Hi ab,

I'd like to pick up GitHub issue #292 and ask whether a PR in this direction would be welcome before I start coding.

The goal is to support server-side uploads larger than 2 GB without buffering the complete request body in RAM.

Current situation

As far as I can see:

THttpRequestContext.ProcessRead already supports writing the request body to ContentStream, for both Content-Length and chunked transfer encoding.
However, neither THttpServer nor THttpAsyncServer assigns ContentStream for incoming requests. Request bodies therefore end up in the Content buffer and are limited by MaxHttpInMemSize.
MultiPartFormDataDecode also expects the complete multipart body in memory.

We currently work around this with application-level chunking: create an upload, send multiple smaller PUT requests, then finalize it. This works well, but it is a custom protocol and cannot be used by normal multipart clients such as HTML forms, curl -F, or third-party applications.

Possible implementation

Because THttpAsyncServer calls the request handler only after the body has been received, a Go-style pull parser cannot be used directly. My suggestion would be to split the work into two steps.

1. Allow incoming bodies to be written to a stream

Add a hook or option to THttpServerSocketGeneric that assigns Ctxt.ContentStream after the headers have been parsed and before the body is read.

Possible variants:

An event similar to OnBeforeBody, where the application may assign a stream such as TFileStreamEx, together with rfContentStreamNeedFree.
Automatic spooling based on a configurable memory threshold and temporary directory.

I would personally prefer the event first, since it is small and leaves storage policy to the application. A threshold-based default could be added later.

Since both server implementations use ProcessRead, the same mechanism should work for THttpServer and THttpAsyncServer. The request handler would receive either Content or ContentStream, depending on how the body was stored.

2. Add a streaming multipart reader

As a follow-up, a TMultiPartReader could parse multipart data incrementally from any TStream. The handler could then process one part at a time and copy file parts directly to their final destination.

Open questions

How should this interact with Content-Encoding, since decompression currently operates on the in-memory buffer?
Is freeing ContentStream through ProcessDone sufficient for aborted connections?
Would you prefer an application-provided stream, automatic threshold-based spooling, or both?

Would a PR starting with step 1 be welcome?

ab · 2026-07-17 08:38:53

The problem with OnBeforeBody is that there is no output parameter to the TOnHttpServerBeforeBody callback.
We could return a specific non standard response code like HTTP_STREAMINPUT = 888 to trigger the TFileStreamEx creation.
But there is no easy way of customizing the file name, which could be nice touch.
And OnBeforeBody was meant to intercept the request ASAP, not customize its process. And it is not in fact called in the proper location in the servers code, e.g. THttpAsyncServerConnection.DecodeHeaders is too soon to assign a TStream.
So I guess a new dedicated event callback could be needed.

Also how to supply the information to the THttpServerRequest instance?
Currently THttpServerRequest.InContent contains the message body.
We could use a similar trick as the output, i.e. if THttpServerRequest.InContentType = STATICFILE_CONTENT_TYPE then THttpServerRequest.InContent does not contain the message body, but the UTF-8 file name of the temporary file.
This is the only reasonable choice I see so that we remain compatible with the existing TRestUriParams logic at REST level... but with the problem of have twice the 'content-type' field in the headers: for the output, it is not a big issue because it is hidden in mormot code; but for the input, it seems a bit too complex for end-user code.
So I guess we would need to automatically detect this pattern in TRestUriContext and decode into a new TRestUriContext.InStream property.
At SOA interface level, we could simply detect a method with a single TStream input parameter, and put the TRestUriContext.InStream in it, or create a transient TRawByteStringString from Call^.InBody (in a way similar to a single RawByteString input parameter is detected to bypass JSON).

About decompression, it should be disabled for the input, as it is currently for the output.
On this URI, the server should return HTTP_UNSUPPORTEDMEDIATYPE = 415 and "Accept-Encoding: identity" if a compression content-encoding is used from the client side.
A minimal support of gzip compression could be done with an explicit TSynZipDecompressor.Create(szcfGZ).
And we definitively need a new THttpMultiPartDecoder in addition to mormot.net.client THttpMultiPartStream for proper multi-part decoding on the fly.

I am also thinking about the ability to not fill a local file, but use a blocking mormot TPipeStream as an option.
Most reverse proxies use a local file for caching the input.
But for a REST server in code, it could make sense to just stream the input as soon as we get it. And for this a mormot TPipeStream is just perfect.
The only drawback is that it is blocking on the HTTP input thread so it could make sense for THttpServer which has its own thread per connection, but not for THttpAsyncServer, which could lead to DoS attacks.
One benefit of the TPipeStream usage could be that it could be used in non-blocking mode, and TPipeStream.Pending could be checked outside of blocking REST process, but e.g. for implementing a reverse proxy with minimal overhead (which is also a feature linked to this issue #292).
In practice, the new dedicated event callback I wrote about before could be the way to create either a temporary TFileStreamEx or a TPipeStream.

Still some discussion ahead...
As you can see, implementing this issue is not trivial, because it should be able to be the foundation of the other features of mORMot: it is not just about adding TFileStream support.

sh17 · 2026-07-17 11:10:52

Thanks for the detailed analysis.

A few thoughts:

Dedicated event: If it receives the parsed headers and Content-Length and returns the destination TStream (or nil for the current in-memory behavior), the application can choose between a temporary TFileStreamEx and a TPipeStream per request. The main DoS consideration could then be documented: pipes should be fine with THttpServer, where each connection has its own thread, or with non-blocking/Pending-style consumption; a temporary file would be the safer default for THttpAsyncServer.
Using STATICFILE_CONTENT_TYPE for the input side makes sense and mirrors the existing output logic. Exposing it through TRestUriContext.InStream, together with detecting a single TStream parameter at the SOA level, should also avoid exposing two competing content types to users.
415 with "Accept-Encoding: identity" for compressed input sounds good. Explicit gzip support through TSynZipDecompressor could always be added later if there is actual demand.

The server-side foundation still depends on the event and stream design, but THttpMultiPartDecoder seems fairly self-contained. I could prepare a PR for an incremental boundary scanner operating on any TStream, processing one part at a time, roughly like Go's mime/multipart.Reader. This would let handlers consume a spooled or piped request body without loading the whole multipart payload into memory.

I would include round-trip tests against THttpMultiPartStream, covering preamble and epilogue handling, CRLF edge cases, nested content-type parameters, and boundaries split across read buffers.

Would you prefer it in mormot.net.http, or in mormot.core.buffers next to MultiPartFormDataDecode?

For the server part, I can then adapt to whichever event/stream design you choose.

ab · 2026-07-17 11:35:05

Good idea to start with THttpMultiPartDecoder = class(TStreamWithNoSeek)
Perhaps in mormot.net.client with the existing THttpMultiPartStream.

But I am not sure it should be class(TStreamWithNoSeek).
Perhaps it could be used at the end of the chain, converting a TFileStreamEx / TPipeStream into a set of blocking calls with name/filename/contenttype/encoding and a TStream content.
So a single local file could be written on disk and decoded when needed.
Or better to split directly into multiple files? But then the TStream pattern is not very good for this, since we miss the multi-part info.
What do you think?

ab · 2026-07-17 11:46:01

Note that on production, for huge content I usually used nginx as reverse proxy, then let it decode everything into a single file, and transmit the file name as header to a mormot server method-based-service.
It works nice since years.
(that's one of the reasons I did not push too much this feature request until I actually need proper reverse proxy support in pure mormot)

About potential THttpMultiPartDecoder interface and usage, please look at the ending proposal of
https://grok.com/share/bGVnYWN5_6eb3b4d … b80142a2da

mORMot Open Source

#1 2026-07-17 07:41:42