Server-side large file upload (issue #292) — proposal

sh17 · 2026-07-17 07:41:42

Hi ab,

I'd like to pick up GitHub issue #292 and ask whether a PR in this direction would be welcome before I start coding.

The goal is to support server-side uploads larger than 2 GB without buffering the complete request body in RAM.

Current situation

As far as I can see:

THttpRequestContext.ProcessRead already supports writing the request body to ContentStream, for both Content-Length and chunked transfer encoding.
However, neither THttpServer nor THttpAsyncServer assigns ContentStream for incoming requests. Request bodies therefore end up in the Content buffer and are limited by MaxHttpInMemSize.
MultiPartFormDataDecode also expects the complete multipart body in memory.

We currently work around this with application-level chunking: create an upload, send multiple smaller PUT requests, then finalize it. This works well, but it is a custom protocol and cannot be used by normal multipart clients such as HTML forms, curl -F, or third-party applications.

Possible implementation

Because THttpAsyncServer calls the request handler only after the body has been received, a Go-style pull parser cannot be used directly. My suggestion would be to split the work into two steps.

1. Allow incoming bodies to be written to a stream

Add a hook or option to THttpServerSocketGeneric that assigns Ctxt.ContentStream after the headers have been parsed and before the body is read.

Possible variants:

An event similar to OnBeforeBody, where the application may assign a stream such as TFileStreamEx, together with rfContentStreamNeedFree.
Automatic spooling based on a configurable memory threshold and temporary directory.

I would personally prefer the event first, since it is small and leaves storage policy to the application. A threshold-based default could be added later.

Since both server implementations use ProcessRead, the same mechanism should work for THttpServer and THttpAsyncServer. The request handler would receive either Content or ContentStream, depending on how the body was stored.

2. Add a streaming multipart reader

As a follow-up, a TMultiPartReader could parse multipart data incrementally from any TStream. The handler could then process one part at a time and copy file parts directly to their final destination.

Open questions

How should this interact with Content-Encoding, since decompression currently operates on the in-memory buffer?
Is freeing ContentStream through ProcessDone sufficient for aborted connections?
Would you prefer an application-provided stream, automatic threshold-based spooling, or both?

Would a PR starting with step 1 be welcome?

ab · 2026-07-17 08:38:53

The problem with OnBeforeBody is that there is no output parameter to the TOnHttpServerBeforeBody callback.
We could return a specific non standard response code like HTTP_STREAMINPUT = 888 to trigger the TFileStreamEx creation.
But there is no easy way of customizing the file name, which could be nice touch.
And OnBeforeBody was meant to intercept the request ASAP, not customize its process. And it is not in fact called in the proper location in the servers code, e.g. THttpAsyncServerConnection.DecodeHeaders is too soon to assign a TStream.
So I guess a new dedicated event callback could be needed.

Also how to supply the information to the THttpServerRequest instance?
Currently THttpServerRequest.InContent contains the message body.
We could use a similar trick as the output, i.e. if THttpServerRequest.InContentType = STATICFILE_CONTENT_TYPE then THttpServerRequest.InContent does not contain the message body, but the UTF-8 file name of the temporary file.
This is the only reasonable choice I see so that we remain compatible with the existing TRestUriParams logic at REST level... but with the problem of have twice the 'content-type' field in the headers: for the output, it is not a big issue because it is hidden in mormot code; but for the input, it seems a bit too complex for end-user code.
So I guess we would need to automatically detect this pattern in TRestUriContext and decode into a new TRestUriContext.InStream property.
At SOA interface level, we could simply detect a method with a single TStream input parameter, and put the TRestUriContext.InStream in it, or create a transient TRawByteStringString from Call^.InBody (in a way similar to a single RawByteString input parameter is detected to bypass JSON).

About decompression, it should be disabled for the input, as it is currently for the output.
On this URI, the server should return HTTP_UNSUPPORTEDMEDIATYPE = 415 and "Accept-Encoding: identity" if a compression content-encoding is used from the client side.
A minimal support of gzip compression could be done with an explicit TSynZipDecompressor.Create(szcfGZ).
And we definitively need a new THttpMultiPartDecoder in addition to mormot.net.client THttpMultiPartStream for proper multi-part decoding on the fly.

I am also thinking about the ability to not fill a local file, but use a blocking mormot TPipeStream as an option.
Most reverse proxies use a local file for caching the input.
But for a REST server in code, it could make sense to just stream the input as soon as we get it. And for this a mormot TPipeStream is just perfect.
The only drawback is that it is blocking on the HTTP input thread so it could make sense for THttpServer which has its own thread per connection, but not for THttpAsyncServer, which could lead to DoS attacks.
One benefit of the TPipeStream usage could be that it could be used in non-blocking mode, and TPipeStream.Pending could be checked outside of blocking REST process, but e.g. for implementing a reverse proxy with minimal overhead (which is also a feature linked to this issue #292).
In practice, the new dedicated event callback I wrote about before could be the way to create either a temporary TFileStreamEx or a TPipeStream.

Still some discussion ahead...
As you can see, implementing this issue is not trivial, because it should be able to be the foundation of the other features of mORMot: it is not just about adding TFileStream support.

sh17 · 2026-07-17 11:10:52

Thanks for the detailed analysis.

A few thoughts:

Dedicated event: If it receives the parsed headers and Content-Length and returns the destination TStream (or nil for the current in-memory behavior), the application can choose between a temporary TFileStreamEx and a TPipeStream per request. The main DoS consideration could then be documented: pipes should be fine with THttpServer, where each connection has its own thread, or with non-blocking/Pending-style consumption; a temporary file would be the safer default for THttpAsyncServer.
Using STATICFILE_CONTENT_TYPE for the input side makes sense and mirrors the existing output logic. Exposing it through TRestUriContext.InStream, together with detecting a single TStream parameter at the SOA level, should also avoid exposing two competing content types to users.
415 with "Accept-Encoding: identity" for compressed input sounds good. Explicit gzip support through TSynZipDecompressor could always be added later if there is actual demand.

The server-side foundation still depends on the event and stream design, but THttpMultiPartDecoder seems fairly self-contained. I could prepare a PR for an incremental boundary scanner operating on any TStream, processing one part at a time, roughly like Go's mime/multipart.Reader. This would let handlers consume a spooled or piped request body without loading the whole multipart payload into memory.

I would include round-trip tests against THttpMultiPartStream, covering preamble and epilogue handling, CRLF edge cases, nested content-type parameters, and boundaries split across read buffers.

Would you prefer it in mormot.net.http, or in mormot.core.buffers next to MultiPartFormDataDecode?

For the server part, I can then adapt to whichever event/stream design you choose.

ab · 2026-07-17 11:35:05

Good idea to start with THttpMultiPartDecoder = class(TStreamWithNoSeek)
Perhaps in mormot.net.client with the existing THttpMultiPartStream.

But I am not sure it should be class(TStreamWithNoSeek).
Perhaps it could be used at the end of the chain, converting a TFileStreamEx / TPipeStream into a set of blocking calls with name/filename/contenttype/encoding and a TStream content.
So a single local file could be written on disk and decoded when needed.
Or better to split directly into multiple files? But then the TStream pattern is not very good for this, since we miss the multi-part info.
What do you think?

ab · 2026-07-17 11:46:01

Note that on production, for huge content I usually used nginx as reverse proxy, then let it decode everything into a single file, and transmit the file name as header to a mormot server method-based-service.
It works nice since years.
(that's one of the reasons I did not push too much this feature request until I actually need proper reverse proxy support in pure mormot)

About potential THttpMultiPartDecoder interface and usage, please look at the ending proposal of
https://grok.com/share/bGVnYWN5_6eb3b4d … b80142a2da

sh17 · 2026-07-27 13:02:21

Quick update: the first `THttpMultiPartDecoder` version is working and passes the round-trip tests.

Before opening the PR: where should the decoder tests go?

Extend the existing multipart tests in test.core.base.pas or add them to test.net.proto.pas, next to the implementation?

ab · 2026-07-27 17:16:03

Currently, multipart forms are tested in TTestCoreBase.MimeTypes.

I guess we could add another method like TTestCoreBase.MultiPartDecoder.
- and eventually move the existing multipart tests to it too (make sense?).

sh17 · 2026-07-27 19:50:07

Done as suggested: I added a new TTestCoreBase.MultiPartDecoder method and moved the multipart tests there from MimeTypes. MimeTypes now only covers MIME recognition, while the MIM test data array is shared by both.

The existing encoder round-trips are now also decoded incrementally by THttpMultiPartDecoder. This includes Rfc2388NestedFiles=true, where a second decoder reads the nested multipart/mixed directly from the outer Current.Content stream.

I also added decoder-specific tests for a 300 KB section containing many CRLF--boundary near-matches, several work-buffer sizes, a source returning only 1–7 bytes per Read(), delimiters split at every buffer position, token and quoted-pair parameters, padding, preamble and epilogue, the header-line limit, truncated input raising EHttpMultiPart, and edge cases for MultiPartFormDataBoundary().

1,310 assertions, green with FPC aarch64-linux and Delphi Win32/Win64.
the whole TTestCoreBase suite still passes on both.

One issue I ran into: Current.Content.Size reports the number of bytes read so far because the section length is not known in advance. Therefore Delphi's CopyFrom(src, 0) silently copies nothing. The documentation recommends StreamCopyUntilEnd() or a normal Read() loop instead, as already done in mormot.crypt.core and mormot.core.zip.

Two questions before I open the PR:

* Current is a record property, so each access copies its four RawUtf8 fields. The documentation recommends caching Current.Content in a local variable. Would a pointer-returning getter, similar to PHttpMultiPartStreamSection, be preferable?

* MultiPartFormDataBoundary() currently sits in mormot.net.client next to the decoder. Would it fit better in mormot.core.buffers next to the other MultiPartFormData* functions, so it could also be reused by MultiPartFormDataDecode?

https://github.com/landrix/mORMot2/tree … rt-decoder

Last edited by sh17 (2026-07-28 09:37:25)

ab · 2026-07-28 19:34:14

The code in branch seems just great!

I guess you could make a pull request.

About the Current property, perhaps you could keep the main property, but also define some smaller properties pointing to the local record using read fCurrent.SomeField to access it per field if needed.

I guess mormot.net.client is the right place.
mormot.core.buffer is already too big.

sh17 · 2026-07-30 06:10:47

Thanks for the quick review and merge.

I can take the server-side implementation next, based on the approach outlined earlier in this thread:

* Add a `THttpServerGeneric` callback that is invoked after the request headers have been parsed, but before the body is read. The callback would return the destination `TStream` for the request body; returning `nil` would preserve the existing in-memory behavior. Most of the required plumbing is already in place, since `THttpRequestContext.ProcessRead` writes to `ContentStream` when one is assigned.
* Use a spooled temporary file as the default stream implementation. This should also work safely with `THttpAsyncServer` and its relatively small worker pool. `TPipeStream` could remain available for blocking server implementations.
* Expose the uploaded body to the handler using `InContentType = STATICFILE_CONTENT_TYPE` and `InContent` set to the temporary file path, mirroring the existing response-side mechanism.
* Wrap this at the REST layer through a `TRestUriContext.InStream` property.
* Extend SOA method handling so that an interface method with a single `TStream` parameter can consume the request body directly.
* In streaming mode, reject compressed request bodies with HTTP `415 Unsupported Media Type` and return `Accept-Encoding: identity`.
* Feed the resulting stream into `THttpMultiPartDecoder`, allowing multipart/form-data uploads of arbitrary size with constant memory usage. This should also address issue #292.

Does this match the architecture you had in mind?

I would follow the same workflow as before: agree on the public interface here first, then implement it on a branch together with the relevant tests.

zen010101 · Yesterday 06:42:39

Just adding a real-world use case in support of this plan.

We are migrating a Java + Tomcat web admin application to mORMot 2, running on embedded aarch64 Linux boxes with 2 GB of RAM. The single biggest HTTP payload there is the firmware upgrade package upload: 100+ MB in one POST. Buffering such a body in memory is not an option on these devices, so a callback that lets the server hand the incoming body to a TStream (with the spooled temp file as safe default) once headers are parsed is exactly what we need — +1 to all seven points from our side.

One emphasis: coverage of the async server family (the "safe for the async family" design in point 2) is the part that matters most to us, since THttpAsyncServer is where we want to end up.

And an offer: once there is a branch to test, we can run it on real aarch64 hardware with FPC — real upgrade packages, slow links, interrupted uploads, concurrent uploads, disk-full — and report the results back here.

ab · Yesterday 20:37:18

@sh17
I validate this proposal, it sounds just like what we need!

@zen010101
Thanks for the feedback.
It is very interesting to see about the mORMot usage. Especially the aarch64 linux boxes HW.

zen010101 · Today 02:13:00

Thanks ab — happy to share a bit more.

The boxes are Rockchip RK3588 / RK3576 / RV1126B (aarch64 Linux, 2-4 GB RAM) doing real-time video transcoding with the SoC hardware codecs, dozens of channels per box. The transcoding API server is built on mORMot 2: THttpAsyncServer for the REST API, TSynLog for logging, TDocVariant/JSON everywhere. Static-linked dependency-free binaries are a perfect match for embedded rootfs images.

The next step — and the reason behind our recent i18n and XML activity — is migrating the companion web admin (currently Java + Tomcat) to mORMot MVC on the same boxes, which will remove the whole JRE from the firmware. That is also where the 100+ MB upgrade-package upload use case comes from.

sh17 · Today 04:10:49

The server side is ready for testing:

https://github.com/landrix/mORMot2/tree … put-stream

I added a new THttpServerGeneric.OnBodyDownload event for both THttpServer and THttpAsyncServer. The event returns a TStream, for example a TFileStreamEx pointing to a temporary file, and the request body is written there without keeping it in memory.

The request then reaches the callbacks with InContentType = STATICFILE_CONTENT_TYPE and InContent containing the file name, similar to how file-based responses are handled. The temporary file is deleted after the request unless the handler renames or moves it.

Compressed request bodies currently return 415 with Accept-Encoding: identity. Chunked bodies are limited by MaximumAllowedContentLength, since their final size is not known in advance. The previous 1 GB in-memory limit does not apply to streamed bodies.

While testing this, I found two existing bugs in the asynchronous request parser and fixed them in the same branch:

* THttpRequestContext.ProcessRead did not consume the actual chunk data from the input buffer. As a result, chunked uploads to THttpAsyncServer could stall or be rejected.
* DoProcessParseLine could read one byte beyond the buffer when a network packet ended exactly on the #13 of a CRLF sequence.

There is also a new TNetworkProtocols.BodyDownload test with 98 assertions. It covers an 8 MB spooled upload, chunked uploads through a raw socket, multipart decoding directly from the spooled file using THttpMultiPartDecoder, rejected compressed bodies, cleanup after aborted uploads, and both server implementations.

The tests are green with FPC on aarch64-linux and Delphi Win32/Win64.

@zen010101: this is the branch to try on your machines. Disk-full situations, slow connections and interrupted uploads are the cases I would especially like to see tested.

I will open the PR after getting some real-world feedback here.

ab · Today 07:20:29

@sh17
Some minor remarks (I can't review or comment on your repo):
- THttpAsyncServerConnection.DoHeaders: move "if strm <> nil then" logic inside a sub function, make it virtual, and put the fn: TFileName local variable in it (a managed variable on stack creates an hidden try..finally which slows down the process on most FPC targets).
- Same remark about THttpServerSocket.GetRequest: use a sub function to remove the local fn: TFileName.
- I am not sure about the TOnHttpServerBodyDownload logic. It returns a TStream but its lifetime should also be customizable. Even keeping and transmitting the TFileStreamEx instance could be beneficial to the end user which won't need to open it. Ask the AI about potential practices, trying to at least propose what nginx does as reverse proxy.
- I don't find the ContentInputName identifier very explicit. Even ContentInputFileName could be more explicit, but we don't know what do do with it. In some usage, it could be needed to keep it. And it would make sense to use a new flag, like rfContentFileNameNeedDelete in ResponseFlags. See previous point about the logic to be reviewed. Or just a new class TFileStreamEventuallyDelete or something like that could be used.
- Are you sure about "Http.ProcessDone; // e.g. free any pending body download stream / spool file" in THttpSocket.Destroy? I need to review it in its various usecases.
- Tests are great!

Just a first pass.

sh17 · Today 08:12:43

* Done in 92470c90 https://github.com/landrix/mORMot2/comm … 54d9abce28 :
both hooks now live in virtual sub-methods
(THttpAsyncServerConnection.DoBodyDownload and
THttpServerSocket.DoBodyDownload), and the managed locals are out of the
OnRead / GetRequest hot paths.

* Regarding THttpSocket.Destroy calling Http.ProcessDone: I checked every
place that sets rfContentStreamNeedFree.

In the THttpSocket hierarchy, the flag is only set for streams created by
the server:

* ContentFromFile responses
* the progressive static-file path
* the new OnBodyDownload spool

The first two are released through ProcessDone in the
THttpServer.Process finally block. The body-download spool is released in
DownloadBody.

THttpClientSocket never assigns Http.ContentStream; client downloads use
the GetBody(DestStream) parameter, and ownership stays with the caller.

So the new call should be a no-op everywhere except for the case it is meant
to cover: a connection being closed after the event in GetRequest, but
before deferred body retrieval by the thread pool or THttpServerResp.

I would still appreciate a second check in case I missed an ownership path.

* On stream lifetime, nginx as a reverse proxy spools request bodies to a
temporary file under client_body_temp_path. proxy_request_buffering is
enabled by default.

client_body_in_file_only has two relevant modes:

* clean: nginx deletes the file when the request is complete
* on: nginx leaves the file in place, and the application becomes
responsible for it

In the latter case, the application receives the path through
$request_body_file.

The current branch effectively implements clean while exposing the file
name through InContent, similar to $request_body_file.

For customizable lifetime handling, I prefer the
TFileStreamEventuallyDelete approach because it would remove the need for
ContentInputName entirely.

TFileStreamEventuallyDelete = class(TFileStreamEx)

It would delete the file on destruction unless DeleteFileOnDestroy is set
to false.

The event would return this stream, or any other TStream as it does today.
The server would free it through the existing rfContentStreamNeedFree paths,
while the deletion policy would stay with the stream class.

The default behavior would match nginx's clean mode. A handler that wants
to keep the file could set DeleteFileOnDestroy to false or rename the file
first, which would correspond to nginx's on mode.

It would also be useful to let the handler reuse the already-open stream
instead of reopening the file by name. Once the body is complete, the
server could rewind the stream and expose it on the request, for example
through a new property on THttpServerRequestAbstract:

InContentStream: TStream

The connection would remain responsible for freeing it after the request.
InContent could still contain the file name with the STATICFILE marker,
serving as the $request_body_file equivalent.

This would also remove the current free-before-Prepare sequence in
DoRequest, because the input stream would move from Http.ContentStream to
the request instance before response processing begins.

If this design looks right, I will implement it this way. You can choose the
class name and location; mormot.core.os, next to TFileStreamEx, seems like the
natural place.

zen010101 · Today 13:18:13

Ran the body-input-stream branch on real hardware, as promised: Rockchip RK3588 (aarch64-linux, FPC 3.2.3), THttpAsyncServer with OnBodyDownload spooling to a temp file (TFileStreamEx + TemporaryFileName), the handler renaming the spool file to take ownership. Client: curl from a LAN Windows box, payload = our real 111 MB firmware upgrade package.

scenario                                       result
---------------------------------------------  ---------------------------------------------------
single 111 MB upload                           OK - MD5 matches end to end, 3.2 s on gigabit LAN
4 concurrent (1x 111 MB + 3x 20 MB)            OK - all MD5s match, no temp-file collisions,
                                               no leftovers
slow link (--limit-rate 1M, 20 MB)             OK - MD5 matches
client killed mid-transfer (~30 of 111 MB)     OK - the half-written spool file was removed by
                                               the server within ~7-22 s after the disconnect,
                                               nothing ever reached the target folder, and the
                                               server kept serving
disk full (spool dir on a 40 MB tmpfs via      spool space fully reclaimed within seconds, the
TMPDIR, uploading 111 MB)                      server stayed alive and functional - but the
                                               client got a raw connection reset (curl exit 56,
                                               no HTTP status), see below

Two observations:

- on ENOSPC the connection is simply reset, so the client gets no HTTP response at all. Would it be worth sending e.g. 507 Insufficient Storage (or plain 500) before closing, when writing to the destination stream fails? For an upgrade-package UI the distinction between "server out of disk" and "network broke" is quite valuable.

- the spool cleanup after a hard client disconnect is not immediate (measured between 7 and 22 s later) - perfectly fine for us, just documenting the observed behavior.

Nice work - this fits our use case exactly. TMPDIR being honored by TemporaryFileName() also made the disk-full isolation easy.

sh17 · Today 16:04:37

Great report, thanks.

Your ENOSPC observation is implemented in fcd6afed: when writing to the
spool stream fails, both server families now answer 507 Insufficient
Storage before closing, instead of the raw reset (new
HTTP_INSUFFICIENTSTORAGE constant + StatusCodeToText entry). One caveat:
on THttpServer the 507 is deterministic (the body is fully read before
the failing write), on THttpAsyncServer it is best effort - the failure
happens mid-transfer, so if the client is still blasting the body, the
reset may still win the race. A full lingering-close would be the next
step if that turns out to matter in practice - your curl test would tell.
Covered in the regression tests with a stream that fails after 64KB.

The 7-22s cleanup delay after a hard disconnect is the async connection
instance GC - expected, and now documented in the event comment.

https://github.com/landrix/mORMot2/tree … put-stream

mORMot Open Source

#1 2026-07-17 07:41:42