Best way to upload a binary file (AJAX client)

CycleSoft · 2014-08-20 09:18:58

Hi,
First af all, I have to say thanks for this great artifact called mORMot!
I'm following it for a while, waiting the right project to use it, and finally the day is come!

I started to develop the backend, and I have some questions/suggestion requests on various topics, so I will ask them one by one in the hoping to help someone like me approaching for the first time to mORMot.

I followed your advice in this article
http://blog.synopse.info/post/2012/03/2 … ed-service
to return a given file (a PDF in my case) setting

Result.Header := HEADER_CONTENT_TYPE + 'application/pdf';

and it works wonderfully!

Now I have to implement a 'symmetric' method, to upload a binary file to the mORMot server.

Wich is the preferred/optimized way to do it? I mean, there is a way to avoid the Base64 encoding and save bandwidth also uploading?

ab · 2014-08-20 09:30:55

The best is to use a simple POST HTTP method with the content as body.
In fact, this is how our RESTful ORM model handle BLOB fields.

But sadly it won't work with interface-based services, only with method-based services.
It could make sense to add support for incoming raw content for interface-based services, e.g. if the only incoming parameter of a method is a "const InputBody: TServiceCustomAnswer" record, then it may contain the incoming body, without expecting the input body to be a JSON array of parameters.
You may create a feature request ticket for this, in http://synopse.info/fossil/tktnew

Note that there is no support to MIME-PART encoding yet in our framework.

DigDiver · 2014-08-20 09:55:22

In my project I use method to upload (MIME-PART encoding) content from Javascript:

procedure TSettingsServer.UploadContent(Ctxt: TSQLRestServerURIContext);
var
 FBody        : RawUTF8;
 boundary     : string;
 ContentType  : String;
 Header       : TStrings;
 Decoder      : TMimeDecoder;
 i           : Integer;
...
begin
 ...

 ContentType := UTF8ToString(FindIniNameValue(pointer(Ctxt.Call.InHead),'CONTENT-TYPE: '));

 Header := TStringList.Create;
 try
  ExtractHeaderFields([';'], [' '], PChar(ContentType), Header, False, False);
  Boundary := Header.Values['boundary'];
 finally
  Header.Free;
 end;

 FBody :=  Ctxt.Call.InBody;

 Decoder := TMimeDecoder.Create;
 try
  Decoder.AllContent := (FBody);
  Decoder.Boundary   := StringToAnsi7(boundary);
  Decoder.UploadPath := DefaultWorkPlace + Format('%s\%d\%s\%s\', [sType, FWPID, SGuid, SFolder]);
  if not ForceDirectories(Decoder.UploadPath) then
   begin
    Ctxt.Error(FormatUtf8('Unable to create folder: %', [Decoder.UploadPath]));
    exit;
   end
  else
   Decoder.ProcessMimeData(Length(FBody));
 finally
  Decoder.Free;
 end;
end;

Source of TMimeDecoder class:

unit ProcessMime;

interface

uses System.Classes, System.SysUtils, Web.HTTPApp, SynCommons;

Type
 TMimeDecoder = class
  private
  Data : RawByteString;
   function ReadMultipartRequest(const Boundary: RawByteString;
    ARequest: RawByteString; var AHeader: TStrings; var Data: RawByteString): RawByteString;
  public
    AllContent, Boundary: RawByteString;
    UploadPath : String;
    procedure ProcessMimeData(ContentLength: Cardinal);
 end;

implementation

function TMimeDecoder.ReadMultipartRequest(const Boundary: RawByteString;
  ARequest: RawByteString; var AHeader:  TStrings; var Data: RawByteString): RawByteString;
var
  Req, RHead: RawByteString;
  i: Integer;
begin
  Result := '';
  AHeader.Clear;
  Data := '';
  if (Pos(Boundary, ARequest) < Pos (Boundary + '--', ARequest))
    and (Pos(Boundary, ARequest) = 1) then
  begin
    Delete(ARequest, 1, Length(Boundary) + 2);
    Req := Copy(ARequest, 1, Pos(Boundary, ARequest) - 3);
    Delete(ARequest, 1, Length(Req) + 2);
    RHead := Copy(Req, 1, Pos(#13#10#13#10,Req)-1);
   Delete(Req, 1, Length(RHead) + 4);
    AHeader.Text := RHead;
    for i := 0 to AHeader.Count - 1 do
      if Pos(':', AHeader.Strings[i]) > 0 then
        AHeader.Strings[i] := Trim(Copy(AHeader.Strings[i], 1,
          Pos(':', AHeader.Strings[i])-1)) + '=' + Trim(Copy(AHeader.Strings[i],
          Pos(':', AHeader.Strings[i])+1, Length(AHeader.Strings[i]) -
          Pos(':', AHeader.Strings[i])));
    Data := Req;
    Result := ARequest;
  end
end;

procedure TMimeDecoder.ProcessMimeData(ContentLength: Cardinal);
var
  Header, HList: TStrings;
  FileName : String;
begin
  if ContentLength = Length(AllContent) then
    while Length(AllContent) > Length('--' + Boundary + '--' + #13#10) do
   begin
      Header := TStringList.Create;
      HList := TStringList.Create;
      try
        AllContent := ReadMultipartRequest('--' + Boundary, AllContent,
          Header, Data);
        ExtractHeaderFields([';'], [' '],
         PChar(Header.Values['Content-Disposition']), HList, False, True);
        if (Header.Values['Content-Type'] <> '') and (Data <> '') then
         begin
          FileName  := Utf8ToAnsi(ExtractFileName(HList.Values['filename']));
          FileFromString(Data, UploadPath + FileName, true);
         end
      finally
        Header.Free;
        HList.Free;
      end;
    end;
end;

end.

ab · 2014-08-20 10:04:11

Thanks for sharing!

DigDiver · 2014-08-20 10:05:12

It would be great to create a topic with links to the software created using mORMot.

ab · 2014-08-20 10:08:47

Indeed!
Sounds like if more and more projects are using our little rodent...
The cross-platform clients did increase its popularity, as far as I found out.
And when you compare with alternatives (DataSnap, or even the TMS Business Subscription) about features, implementation patterns or performance, we may honestly say that we can be proud of our Open Source solution!

Feel free to create a dedicated forum thread (as we did for third-party samples).
I will stick the topic to the "mORMot Framework" category of this forum.

We may also include the list in the documentation, for reference.

CycleSoft · 2014-08-20 10:23:10

Thanks for such a quick reply!

@ab: if I undestend well your proposal to add support for incoming raw content for interface-based services, then there will be one and only one TServiceCustomAnswer parameter, right?
If so, how the interface service will be able to properly handle such data, without any other input parameters such, say, an ID Customer to link the content to?

@DigDiver: thanks for sharing your code!
I'm not yet able to undestend well this 'low-level' use of mORMt... guess I have to dig a little under the surface!
Anyway thanks for pointing me in this direction!

CycleSoft · 2014-08-20 10:31:22

..and about the links to the softwares created with mORMot, I will add a link to mine with pleasure!
In fact, another question I have is how I can mention mORMot in my application(s)? Something like 'powered by mORMot' and a link?

ab · 2014-08-20 10:31:42

@CycleSoft
Good point.
We may allow other parameters in addition to TServiceCustomAnswer record (perhaps renamed TServiceCustomRequest ) but expecting them to be encoded at the URI level.
In all cases, this would work only for TSQLRestRoutingREST kind of routing, in its ExecuteSOAByInterface method.

Note that DigDiver code uses a method-based service, not an interface-based service here.

CycleSoft · 2014-08-20 11:00:48

ab wrote:

@CycleSoft
Good point.
We may allow other parameters in addition to TServiceCustomAnswer record (perhaps renamed TServiceCustomRequest ) but expecting them to be encoded at the URI level.
In all cases, this would work only for TSQLRestRoutingREST kind of routing, in its ExecuteSOAByInterface method.

My knowledge of mORMot (and HTTP-world in general) is too superficial to be meaningful, but my impression is that if to implement this feature you have a hard time and limitations, maybe this is not the right direction and there is a cleaner solution?
A newbie (like me) will easily be tricked, I don't even know what 'TSQLRestRoutingREST kind of routing' is

If you already are using the 'method' approach for the BLOBs, there will be a good reason, so I prefer learn something new and adopt your approach.

That said, can you only point me in the right direction? Can I find a start point in the documentation or is better to search your source code?

ab wrote:

Note that DigDiver code uses a method-based service, not an interface-based service here.

OK!

ab · 2014-08-20 11:05:02

Just take a look at the "26 - RESTful ORM" sample, which implement a BLOB uploading/download mechanism, using a method-based service.

See https://github.com/synopse/mORMot/tree/ … Tful%20ORM

CycleSoft · 2014-08-22 10:12:08

Just to 'close' the thread...
I successfully built a method service using POST requests as hinted by Arnaud, and with the very valuable help of DigDiver that shared his TMimeDecoder class that I have slightly modified ( with Delphi XE4, ExtractHeaderFields returns 'values' in the Strings parameter with still the double quotes round them, so the filename field is not usable 'as-is', not sure how it works for you DigDiver)

Again thanks to you guys!

Now I have to be sure that my implementation is thread-safe, but this is worth another..... well... thread

profh · 2014-09-09 04:59:11

many thanks to all of you.

but i have another issue:
the memory of server kept increasing after each Binary Data was posted to the server, even if the server did not handle it at all.

ab · 2014-09-09 06:12:54

In moRMot, memory blocks coming from HTTP is handled with RawByteString (or RawUTF8) variables, so there should not be any memory leak.
Unless your code still refers to the incoming data.

The framework is tested for any memory leak, and is known to run for weeks in production, without any memory increase.

I suspect there is a memory leak in your code.
For instance, you create a temporary stream on your side, without releasing it.
Enable memory leak reporting in Delphi, and find out what is wrong.

profh · 2014-09-19 09:32:37

i compiled "13 - StandAlone JSON SQL server" without changing any thing, and run the JSONSQLServer.exe, then used the code below to post data to the server, not post of REST, but post of URL, the memory of server kept increasing after each data was posted to the server.

procedure TForm8.Button11Click(Sender: TObject);
var
  tmpstream:TMemoryStream;
  ab:boolean;
  HTTP: THTTPSend;
  url:string;
  slist:TStringlist;
begin
  tmpstream := TMemoryStream.Create;
  HTTP := THTTPSend.Create;
  slist:=TStringlist.Create;
  try
    url := 'http://localhost:888/root';
    tmpstream.LoadFromFile('c:\rtl70.bpl');
    tmpstream.Position := 0;
    ab := HttpPostFile(url,'abc','def',tmpstream,slist);
    ... ...
  finally
    tmpstream.Free;
    HTTP.Free;
    slist.free;
  end;
end;

Last edited by profh (2014-09-19 09:37:41)

profh · 2014-09-19 09:52:42

i did it because i need to post data from browser, but REST methord can not be used in browser as usual .

ab · 2014-09-19 13:26:45

Weird.
There is no memory leak when running our extensive regression tests.
I doubt there is a memory leak - what does exist is that the memory increases a little for the first requests, until the thread pool is completely used and memory is stabilized.

Which version are you using? Current is 1.18.264.

What does FastMM4 state, when the server is run with EnableMemoryLeakReporting conditional?

profh · 2014-09-19 13:53:18

i used the latest version, and i enabled EnableMemoryLeakReporting, but no memory leak reported.

profh · 2014-09-19 14:02:15

please note that i just post data to server from client, and the server did not handle the data at all.
it seems that the server will be attacked by continuing data whatever the client postes.

Last edited by profh (2014-09-19 14:09:17)

ab · 2014-09-19 15:17:37

There is no memory leak IMHO, just stabilization of the memory and resource/threads allocation.

profh · 2014-09-19 16:52:25

it is strange, and it really bothers me for a little while.

mpv · 2014-09-19 17:59:47

We send gigabytes of files in production without problems. Our clients is browser what use this technics .
Is memory usage continue increasing after 100,200,300 requests?

profh · 2014-09-20 00:37:08

you are absolutely right, after certain of requests the memory usage is not continue increasing.
and thank you so much.

ab · 2014-09-20 07:18:00

This is how fastmm4 works.

profh · 2014-09-20 21:56:26

got it.
after a little while, without request, the memory usage of server will decrease.

thanks again.

mORMot Open Source

#1 2014-08-20 09:18:58

Best way to upload a binary file (AJAX client)

#2 2014-08-20 09:30:55

Re: Best way to upload a binary file (AJAX client)

#3 2014-08-20 09:55:22

Re: Best way to upload a binary file (AJAX client)

#4 2014-08-20 10:04:11

Re: Best way to upload a binary file (AJAX client)

#5 2014-08-20 10:05:12

Re: Best way to upload a binary file (AJAX client)

#6 2014-08-20 10:08:47

Re: Best way to upload a binary file (AJAX client)

#7 2014-08-20 10:23:10

Re: Best way to upload a binary file (AJAX client)

#8 2014-08-20 10:31:22

Re: Best way to upload a binary file (AJAX client)

#9 2014-08-20 10:31:42

Re: Best way to upload a binary file (AJAX client)

#10 2014-08-20 11:00:48

Re: Best way to upload a binary file (AJAX client)

#11 2014-08-20 11:05:02

Re: Best way to upload a binary file (AJAX client)

#12 2014-08-22 10:12:08

Re: Best way to upload a binary file (AJAX client)

#13 2014-09-09 04:59:11

Re: Best way to upload a binary file (AJAX client)

#14 2014-09-09 06:12:54

Re: Best way to upload a binary file (AJAX client)

#15 2014-09-19 09:32:37

Re: Best way to upload a binary file (AJAX client)

#16 2014-09-19 09:52:42

Re: Best way to upload a binary file (AJAX client)

#17 2014-09-19 13:26:45

Re: Best way to upload a binary file (AJAX client)

#18 2014-09-19 13:53:18

Re: Best way to upload a binary file (AJAX client)

#19 2014-09-19 14:02:15

Re: Best way to upload a binary file (AJAX client)

#20 2014-09-19 15:17:37

Re: Best way to upload a binary file (AJAX client)

#21 2014-09-19 16:52:25

Re: Best way to upload a binary file (AJAX client)

#22 2014-09-19 17:59:47

Re: Best way to upload a binary file (AJAX client)

#23 2014-09-20 00:37:08

Re: Best way to upload a binary file (AJAX client)

#24 2014-09-20 07:18:00

Re: Best way to upload a binary file (AJAX client)

#25 2014-09-20 21:56:26

Re: Best way to upload a binary file (AJAX client)

Board footer