mORMot Open Source

Georg

Member

Registered: 2016-11-22

Posts: 18

Upload and download of big files

Hello,

I'm new to mORMot (btw, thanks for the nice talks about mORMot at EKON20 :-)) and I'm now trying to convert a datasnap based client server application to mORMot.
For client server services, I use an interface (as in sample 14 with a TSQLHttpServer).

So far this worked well. But now, I need to upload and download big files (up to 1 GB) between server and client.
I checked out the other threads concerning this topic, but I'm still not sure how to do it.

In the datasnap code, I had a download and upload method like this (FileName is the name on the server, ServerType is for getting the right path on the server):

procedure UploadFile(str: TStream; ServerType: string; FileName: string);
function DownloadFile(ServerType: string; FileName: string): TStream;

The data transfer was then done via the TStream object (with progress indicator in the client).

This approach is not possible out of the box in mORMot, I get an error because TStream cannot automatically be serialized.

I tried to do the transfer via a RawByteString object (as described in another thread here in the forum), but due to the size of the files, the memory consumption is to high.
I also read about doing this via a method based service, but I'm not sure if this can be mixed with my interface based service (I would rather like to stay with the interface based approach if possible).

The client is a delphi client, but later on it may also be a web (AJAX) client, so the solution should work in both cases.

What would be the best way to implement this upload and download with mORMmot?

Thanks in advance,
Georg

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: Upload and download of big files

Upload of huge files is not supported by interface-based-services.
You can make custom download - using TServiceCustomAnswer custom record type.
If you supply STATICFILE_CONTENT_TYPE constant as output header, you can let http.sys download the file in chunks.

For upload, even method-based services was not meant for big content.
The body is retrieved from the client as one RawByteString buffer.
Isn't it possible to do the upload spliting from the client side?

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

Re: Upload and download of big files

This topic is a duplicate for this one

Last edited by mpv (2016-11-22 16:38:37)

Georg

Member

Registered: 2016-11-22

Posts: 18

Re: Upload and download of big files

Thanks for your answer. Unfortunately, I don't really know how to proceed.

You can make custom download - using TServiceCustomAnswer custom record type.
If you supply STATICFILE_CONTENT_TYPE constant as output header, you can let http.sys download the file in chunks.

Can you elaborate on that? Oder maybe point me to an example or the part of the documentation where this is described?
Does this work with the TSQLHttpServer that I use in my interface-based approach?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: Upload and download of big files

Take a look at unit Unit2Static in sample 04.
This is for a method-based service, but see TServiceCalculator.Test in PeopleServer.pas of sample 27 for how to use TServiceCustomAnswer.

erick

Member

Registered: 2015-09-09

Posts: 155

Re: Upload and download of big files

It depends how big.

In my JavaScript client I load the file in JavaScript in 64 bit encoding, then upload a few kB (currently 4 kB) at a time.  This way I can pass the chunk to a mORMot SOA function and it appends it to a file without filling mORMot process' memory.  This allows me to show progress of upload too if I choose.   Even on phones, this allows you to upload many MB files.  But if you are talking GB, you will need a different solution because it relies on the JavaScript system holding the whole file in memory.

I posted a demo to the Video's topic at the top of the mORMot sticky list.

Erick

Peter Evans

Member

Registered: 2016-07-03

Posts: 32

Re: Upload and download of big files

I realise that I am a bit late to this conversation.
I haven't been in the situation where I have had to download such a large file.

My approach for smaller files is the following:-

1. Client calls the Server. This is an Interface call in mORMot.
(A method call in DataSnap)
The Server passes back the file size in bytes.
The Server passes back the CRC check sum.

2. Client calls the Server. This is a different Interface call.
Client can specify the "chunk" to be returned.
Alternative, the Server returns a set "chunk" size. If the Server does this
then the "chunk" size can be changed at the Server, on the fly, rather
than having to update all the Clients.
Server sets a flag when the end of the file is reached.

Now the Client can update the user interface, between calls to Server.
No need to perform the download in a separate thread, to make the user
interface appear responsive. Hopefully
increases the responsiveness of the Server to other Clients.

3. After file has been downloaded the Client checks the file size and CRC.