Sending files using mormot?

QUICKSORT · 2015-05-08 09:22:13

Hi,

I'm working on a program with a cloud server, that uses mormot (delphi) for all kinds of information transfer.
However I want add a file transfer system where I can keep the files in a file system on the server side (so not blob field), (and ideally have it also accessible through URL's);
Currently I'm sending the Files through SOA parameters, either as return value or a method Parameter. E.g.: a SOA call on the server called by the client

SendFile(FILE);

and

GetFile(ID): FILE;

Once the data is received, it is not put in the mormot database, but instead direcly written in a filesystem within that SOA method server side.

However I'm having some issues with this implementation. Ideally I want to send and receive a list of Bytes. I tried using TBytes, array of Bytes, ... any dynamic array I use as a parameter/return value causes in the generated mORMotClient.pas a new Dynamic array type to be created. E.g. in case of a TBytes it creates in the mORMotClient.pas the following type:

Type
  TArray<System.Byte>= array of bytes

And it also creates methods named "TArray<System.Byte>2Variant". And all of these lines give compile errors (mainly E2029, Expected ...or...or... instead of ...) which renders it useless.

So as a second option I tried to send the complete data as a string. This works perfectly with file types that contain actual text such as txt, xml, html, ... . However once I send a binary file, I don't receive the correct data on the other side of the connection. So reading and converting the binary file to string works. the string contains the data correctly, but once the file is received on the other side of the connection the data gets corrupted. E.g. If I have a string containing such a binary converted value:

'%9#0A#0IQ'DHA'.'P'

it becomes more or less something like this:

'%9#00000000A#0000000IQ'DHA'.'P'

So the file gets significantly larger containing unwanted data. I can recognize the actual file in it when looking at it, but it gets modified. So this isn't a good solution either.
Also in this implementation, when downloading a file as a return value of a SOA method client side, I HAVE to know the file size of the file to be downloaded prior to downloading, so I can allocate the correct amount on AnsiString.

So finally I am Sending the file as a string of HEX. Using the methods BinToHex and HexToBin. This works but sends double the amount of data. Also in this case I have to manually detect the text type ecodings (e.g. through file extension) and write down the file in the correct encoding so that e.g. a unicode file isn't written in binary by mistake.

So now to my question. Is there any way to avoid the issues I'm having with the above implementations? Or is there a better way to send and receive a file, most preferably as a list of Bytes? And also as a non must extra it would be very nice if I could see the download/upload progress with perhaps also the download/upload speed, together with having the files accessible through URL's.

Thanks in Advance.

Last edited by QUICKSORT (2015-05-08 09:40:12)

ab · 2015-05-08 09:54:17

Interface-based services do indeed rely on JSON for transmission, so Base64 (not hexadecimal) is used for binary content.

The easiest is to use a method-based service to handle such file transmission.
It will allow to use a perfectly REST compatibly remote access to the file resource.

See the following sample:
https://github.com/synopse/mORMot/blob/ … rClass.pas

QUICKSORT · 2015-05-08 10:58:19

I checked that earlier, initially thought it wouldn't be much different with my string issue with the TSQLRestServerURIContext using RawUTF8, but perhaps should give it a try.

Thank you for your fast reply

ab · 2015-05-08 11:24:12

About interface-based parameters, you should better try to use RawUTF8 with Base64 encoding.
I've just added RawByteString explicit support for SOA interface-based services parameters - content will be transmitted as base64-encoded JSON text.
See http://synopse.info/files/html/Synopse% … l#TITL_154

But a method-based service gives you full access to the incoming and outgoing content.
So the blob could be transmitted as binary.

And it has the benefit of being REST compatible, so very easy to work with: for instance, a simple GET on a resource would allow to download the content, from a regular browser.

QUICKSORT · 2015-05-08 13:32:59

I'm sorry I'm rather new to Delphi and Mormot, What defines an Inteface-Based service, and what a Method-Based one?

You mean the fact that the SOA classes are TInterfacedObjects, hence intefrace-based?
And don't know what you mean with method based.
As far as I can understand, both in SOA and the example a method is invoked (in may case the Get and SetFile methods, and in the example the blob method).

ab · 2015-05-08 14:03:10

See http://synopse.info/files/html/Synopse% … ml#TITL_63
and http://synopse.info/files/html/Synopse% … ml#TITL_49

QUICKSORT · 2015-05-08 14:13:41

Alright, I think I can work my way around now.
Thanks a lot!

QUICKSORT · 2015-05-12 09:53:51

I have the impression that the method based implementation is not cross platform. Can you confirm this?

ab · 2015-05-12 10:08:46

The method based implementations are cross-platform, but you have to marshal the transmitted JSON by hand.
See e.g. how authentication is made on cross-platform: it uses method-based services.

But for high level business services, interface-based services sound like a better option, since you do not have the JSON marshaling to do.

QUICKSORT · 2015-07-02 13:48:11

Hi,
So I just updated to the latest version of mORMot to try out the RawByteString you just added, but the data is still not transferred correctly;
So to be specific, this is pretty much Service method on the server side:

function GetFile(ID): RawByteString

Where the requested file is returned as a RawByteString.

I receive something that is larger and has some garbage in it mixed with the file.

QUICKSORT · 2015-07-09 14:28:08

Do I do something wrong? I mean right now on the latest version RawByteString should work right?
I Simply return a raw byte string.
Or am I supposed to create a dummy derived TSQLRecord that has a RAWByteString member variable (blob field)?
So then put the file in that dummy record, sending the record to the service, and unpacking it on the server without putting it in the dataabse?
(though I should be able to do the same by using TSQLRawBlob rught?)

QUICKSORT · 2015-08-07 09:53:10

Hey, I'd be happy if you could help me out.

So I tried using a RAWByteString in many ways.

- if I actually make the return value of the service method to a RAWByteString as:

GetFile(const parFileID: integer): RawByteString;

The value I receive from the other side is, slightly larger than the original file, and the content is completely something different, so a conversion has happened, and I don't know how to make it back like it's original. But the content of this result does not have weird unicode stuff in it. it looks like it consist out of mainly alphabet characters.

- if I change the return value of the service method to a variant as:

GetFile(const parFileID: integer): variant;

in which I put a RawBytestring using RawByteStringToVariant or a simple asignment.

The value I receive from the other side is something that looks almost exactly like what I sent, which some unicode stuff, but the file size is very slightly smaller, and when I compare the files, some characters changed like µµµµ to ???? . Some characters are missing.

But these issues only occur of binary files; Sending text based files such as txt, rtf, ... is no problem.

Again for now I still use BinTohex, which I then send as a string, which has twice the size of the source file, so not very optimal. and HexToBin also doesn't work on nextgen mobile. So I cannot use it on android/ios either.

Any help? I'm really stuck here for some time now.

ab · 2015-08-07 13:04:13

Did you try TServiceCustomAnswer?

See http://synopse.info/files/html/Synopse% … #TITLE_430

QUICKSORT · 2015-08-10 12:14:37

Problem was I apparently didn't know much about string encodings.
TServiceCustomAnswer made me realize this.

Sending RawByteString with it's UTF16 encoding was the main issue here.
Once I send it as BASE64 problem is solved.

We are still talking about a slight file size increase in this case, but that is negligible.

Thank you for your help.

mORMot Open Source

#1 2015-05-08 09:22:13

Sending files using mormot?

#2 2015-05-08 09:54:17

Re: Sending files using mormot?

#3 2015-05-08 10:58:19

Re: Sending files using mormot?

#4 2015-05-08 11:24:12

Re: Sending files using mormot?

#5 2015-05-08 13:32:59

Re: Sending files using mormot?

#6 2015-05-08 14:03:10

Re: Sending files using mormot?

#7 2015-05-08 14:13:41

Re: Sending files using mormot?

#8 2015-05-12 09:53:51

Re: Sending files using mormot?

#9 2015-05-12 10:08:46

Re: Sending files using mormot?

#10 2015-07-02 13:48:11

Re: Sending files using mormot?

#11 2015-07-09 14:28:08

Re: Sending files using mormot?

#12 2015-08-07 09:53:10

Re: Sending files using mormot?

#13 2015-08-07 13:04:13

Re: Sending files using mormot?

#14 2015-08-10 12:14:37

Re: Sending files using mormot?

Board footer