Send binary data in interface-based service - without encoding!

jaclas · 2017-11-02 11:44:46

Please add the ability to send binary data in raw mode (without base64 encoding) in interface-based services. Maybe some dedicated type like TBinaryBlob or TRAWData?

This would be very useful in some cases!

Junior/RO · 2017-11-02 16:46:31

I agree, very useful.

edwinsn · 2017-11-03 03:27:04

I agree.

Stregor · 2017-11-03 11:34:19

Will be very useful, i agree.

eraldo · 2017-11-03 16:18:47

Me too!

JD · 2017-11-04 20:03:41

I fully agree and I would love to see this functionality

ab · 2017-11-04 21:14:33

Okay, okay, I will add it!

jaclas · 2017-11-04 21:42:06

Hurrah! :-)

Thx!

JD · 2017-11-05 12:23:25

ab wrote:

Okay, okay, I will add it!

Thanks a lot!

ab · 2017-11-06 16:34:28

If your purpose is to upload some binary data, RawByteString and TSQLRawBlob input parameters will by default be transmitted as Base64 encoded JSON text.

You may define Client-Server services via methods to transmit raw binary, without the Base64 encoding overhead. It would allow low-access to the input content type and encoding, even with multi-part file upload from HTTP.

As an alternative, if you use default TSQLRestRoutingREST routing, and defined a single RawByteString or TSQLRawBlob input parameter, it will be processed as a raw POST with binary body defined with mime-type 'application/octet-stream'. This may be more optimized for remote access over the Internet.
See https://synopse.info/fossil/info/4cfc722c69

jaclas · 2017-11-14 11:14:02

Does this mean that the method can have only one parameter? Eg.

procedure SendBlob (const aBlob: RawByteString);

And if I write the method:

Procedure SendBlob(aBlobKind: Integer; const aBlob: RawByteString);

then aBlob will be encoded with Base64?

mpv · 2017-11-14 11:49:43

Yes. Current implementation use JSON as a transport for multiple parameters. There is no possibility to serialize a binary data to JSON without base64 encoding.

jaclas · 2017-11-14 13:49:29

With one parameter this is not very useful :-( There is no way to tell what is in the blob, probably need to use http header for additional info.

It would be useful to tag/register a method so that its parameters are not serialized.
Maybe worth consider to expand TServiceFactoryServer interface by adding method:

 function DoNotEncodeParams(const aMethod: array of RawUTF8): TServiceFactoryServer;

Is it possible to realize?

ab · 2017-11-14 18:32:48

Use manual serialization to a single RawByteString in such case.

There are such serialization available in the framework.

jaclas · 2018-02-12 12:30:55

I found problem on this topic.
I get exception:

TSQLRestRoutingREST.Error: 
{"errorCode":406,  "errorText":"sicShared execution failed (probably due to bad input parameters) for TMyService.SendFilePart"}

See for reason:

ab · 2018-02-12 12:58:21

How is your service defined?

One input parameter without JSON encoding is NOT supported yet, as documented and stated above in this forum thread.

Use a method-based service instead to upload some binary content from the client.

jaclas · 2018-02-12 13:13:05

Not supported?

Then why you wrote above:

"As an alternative, if you use default TSQLRestRoutingREST routing, and defined a single RawByteString or TSQLRawBlob input parameter, it will be processed as a raw POST with binary body defined with mime-type 'application/octet-stream'. This may be more optimized for remote access over the Internet.
See https://synopse.info/fossil/info/4cfc722c69"
?

And what changes has this commit? https://synopse.info/fossil/info/4cfc722c69

btw on other (text) files this way acts properly (send without json and base64 encoding)

Last edited by jaclas (2018-02-12 13:15:38)

ab · 2018-02-12 14:18:00

I forgot about this commit.
This is event documented in https://synopse.info/files/html/Synopse … l#TITL_197

Are you sure you use the proper mime type, with raw binary content (no base64 encoding)?
Try to find out using the debugger why TSQLRestRoutingREST.ExecuteSOAByInterface doesn't understand your client query.

jaclas · 2018-02-12 14:28:15

For send PDF file header is:

POST /server/SomeService.SendFilePart HTTP/1.1
Accept: */*
FILEID: 84317faa3d2516d189da1e356e12e4fe5f55eeea8970a09d26739ec690e0c4fb
Content-Type: application/pdf
User-Agent: Mozilla/5.0 (Windows; mORMot 1.18 TWinINet)
Host: localhost:8090
Connection: Keep-Alive
Accept-Encoding: gzip, deflate
Content-Length: 1048576

%PDF-1.4
%ŕáâă
2 0 obj
<</Filter /FlateDecode
/Length 11656951
>>
stream
xśĚÝÁŽćHš¬ç}_ĹÜŔNŇé$·
[...]

for XML:

POST /server/SomeService.SendFilePart HTTP/1.1
Accept: */*
FILEID: 8472f3d762a2f42ebde6a9bf9fba0806d9c8e59221fc0dafd3fe05dd1a94050c
Content-Type: application/octet-stream
User-Agent: Mozilla/5.0 (Windows; mORMot 1.18 TWinINet)
Host: localhost:8090
Connection: Keep-Alive
Accept-Encoding: gzip, deflate
Content-Length: 1048576

<?xml version="1.0" encoding="UTF-8"?>
<Migration>
<Session Type="Online">
	<Platform Type="Source">
[...]

Content-Type is different for PDF and for XML, how to force properly value of Content-Type?

ab · 2018-02-12 20:20:19

It should be exactly application/octet-stream - as documented.

jaclas · 2018-02-12 21:06:33

But how to set it? For now I don't set the Content-Type header.

jaclas · 2018-02-13 09:31:01

is not exactly like you wrote:

"if you use default TSQLRestRoutingREST routing, and defined a single RawByteString or TSQLRawBlob input parameter, it will be processed as a raw POST with binary body defined with mime-type 'application/octet-stream'"

Can you correct it?

ab · 2018-02-13 12:24:54

Yes, you are right, this is an issue.
Even if setting 'application/pdf' is the right way on the client side.

I guess that the server side is to be fixed.
Please try https://synopse.info/fossil/info/26d3008868

jaclas · 2018-02-13 13:02:50

Now is ok (probably ;-)
Thx Arnaud

Junior/RO · 2018-02-13 16:37:33

@jaclas what font do you use in the IDE?

jaclas · 2018-02-13 16:53:16

@Junior/RO

Source Code Pro Semibold

mORMot Open Source

#1 2017-11-02 11:44:46

Send binary data in interface-based service - without encoding!

#2 2017-11-02 16:46:31

Re: Send binary data in interface-based service - without encoding!

#3 2017-11-03 03:27:04

Re: Send binary data in interface-based service - without encoding!

#4 2017-11-03 11:34:19

Re: Send binary data in interface-based service - without encoding!

#5 2017-11-03 16:18:47

Re: Send binary data in interface-based service - without encoding!

#6 2017-11-04 20:03:41

Re: Send binary data in interface-based service - without encoding!

#7 2017-11-04 21:14:33

Re: Send binary data in interface-based service - without encoding!

#8 2017-11-04 21:42:06

Re: Send binary data in interface-based service - without encoding!

#9 2017-11-05 12:23:25

Re: Send binary data in interface-based service - without encoding!

#10 2017-11-06 16:34:28

Re: Send binary data in interface-based service - without encoding!

#11 2017-11-14 11:14:02

Re: Send binary data in interface-based service - without encoding!

#12 2017-11-14 11:49:43

Re: Send binary data in interface-based service - without encoding!

#13 2017-11-14 13:49:29

Re: Send binary data in interface-based service - without encoding!

#14 2017-11-14 18:32:48

Re: Send binary data in interface-based service - without encoding!

#15 2018-02-12 12:30:55

Re: Send binary data in interface-based service - without encoding!

#16 2018-02-12 12:58:21

Re: Send binary data in interface-based service - without encoding!

#17 2018-02-12 13:13:05

Re: Send binary data in interface-based service - without encoding!

#18 2018-02-12 14:18:00

Re: Send binary data in interface-based service - without encoding!

#19 2018-02-12 14:28:15

Re: Send binary data in interface-based service - without encoding!

#20 2018-02-12 20:20:19

Re: Send binary data in interface-based service - without encoding!

#21 2018-02-12 21:06:33

Re: Send binary data in interface-based service - without encoding!

#22 2018-02-13 09:31:01

Re: Send binary data in interface-based service - without encoding!

#23 2018-02-13 12:24:54

Re: Send binary data in interface-based service - without encoding!

#24 2018-02-13 13:02:50

Re: Send binary data in interface-based service - without encoding!

#25 2018-02-13 16:37:33

Re: Send binary data in interface-based service - without encoding!

#26 2018-02-13 16:53:16

Re: Send binary data in interface-based service - without encoding!

Board footer