Compress a File

colbert · 2013-03-11 17:48:21

Nation, hello!

I need to Compress a file (it has about 15 MB) and send it from a server to a client using Interfaced Based Services.

What would be the better approach?

Last edited by colbert (2013-03-11 17:52:57)

ab · 2013-03-11 19:04:32

By design, interface-based services should return a valid JSON content.
So it is not possible to return directly a BLOB content.

You may:
- Either use a method-based service and set the return content to be a BLOB (the fastest, and will use less bandwidth);
- Or use an interface-based service and Base64 encode the returned RawUTF8 string.

About compression, you may use a ZIP/DEFLATE algorithm, with a compression level set according to your speed and CPU server expectations (1=fast but less compressed, 9=very slow but very compressed).
Or use our SynLZ unit, which compresses very fast, using very little server side CPU, but has an inferior ratio.

colbert · 2013-03-11 19:08:34

What would be method-based service? Where can i find more information about it?
I have client-server application with ORM and TSQLRecord inherited classes, can I use the same TSQLHttpServer to provide this services?

Last edited by colbert (2013-03-11 19:20:17)

Leander007 · 2013-03-11 19:23:31

Look at example "06 - Remote JSON REST Service".

ab · 2013-03-11 19:29:57

... and do not forget the RTFM SAD.pdf
http://synopse.info/files/pdf/Synopse%2 … 201.18.pdf
(especially the 1.18 version, which is much more complete and easy to deal with)

colbert · 2013-03-12 13:50:18

Now I'm using TSynMemoryStreamMapped to get the file and TRawByteStringStream to use the BinToHex method.

procedure TDBServer.SendFile(var Ctxt: TSQLRestServerCallBackParams);
var
  aFile: TSynMemoryStreamMapped;
  Strm: TRawByteStringStream;
begin
  Strm := TRawByteStringStream.Create;

  aFile:= TSynMemoryStreamMapped.Create(ExtractFilePath(Application.ExeName) + '\Files\asd.pdf');
  aFile.SaveToStream(Strm);
  Ctxt.Results([BinToHex(Strm.DataString)]);
end;

In the client, trying to turn to a File again:

var
  aResp: RawUTF8;
  Data: array of single;
  aFile: TSynMemoryStreamMapped;
  Strm: TRawByteStringStream;
begin

  aResp := Client.CallBackGetResult('SendFile', []);
  if aResp <> '' then
  begin
    SetLength(Data, length(aResp) div 8);
    if not SynCommons.HexToBin(pointer(aResp), pointer(Data), length(Data) * 4)
    then
      exit;
  end;
  aFile:= TSynMemoryStreamMapped.Create(aResp);
  aFile.SaveToFile(ExtractFilePath(Application.ExeName) + '\Files\asd.pdf');
end;

This code does'nt work.

How can I turn it into a File again?

Last edited by colbert (2013-03-12 14:24:45)

ab · 2013-03-12 14:08:40

Some points:
- Your code is leaking memory - the aFile instance is not freed;
- You can use the StringFromFile() function which will be faster;
- Base64 encoding would be better than BinToHex/HexToBin;
- Then FileFromString() will work on the other side.

colbert · 2013-03-13 15:51:23

I'm using this method:

procedure TDBServer.SendFile(var Ctxt: TSQLRestServerCallBackParams);
var
  FileName: RawUTF8;
  Content : RawByteString;
begin
  if UrlDecodeNeedParameters(Ctxt.Parameters, 'FileName') then
  begin
    FileName := Ctxt.InputUTF8['FileName'];
    Content :=  BinToBase64(StringFromFile(ExtractFilePath(Application.ExeName) +
      '\Update\' + FileName));
    Ctxt.Results
      ([Content]);
  end
  else
    Ctxt.Error('Missing Parameter');
end;

It works, but after use it the memory of the server raises from 4MB to 20 MB.

What could be the problem? is it the Framework or the Code?

ab · 2013-03-13 16:18:21

This Client-Server protocol uses JSON in all code above, as encoded server-side via Ctxt.Results() method, but you can serve any kind of data, binary, HTML, whatever... just by overriding the content type on the server with Ctxt.Returns().

With a method-based service, you are not obliged to convert the file to base64 or hexadecimal.

Or you can return some binary file, retrieving the corresponding MIME type from its binary content:

procedure TSQLRestServer.GetFile(var Ctxt: TSQLRestServerCallBackParams);
var fileName: TFileName;
content: RawByteString;
contentType: RawUTF8;
begin
fileName := 'c:\data\'+ExtractFileName(Ctxt.Input['filename']);
content := StringFromFile(fileName);
if content='' then
Ctxt.Error('',HTML_NOTFOUND) else
Ctxt.Returns(content,HTML_SUCCESS,HEADER_CONTENT_TYPE+
GetMimeContentType(pointer(content),Length(content),fileName));
end;

You can even use a special feature of http.sys to serve a file content as fast as possible. In fact, if you specify HTTP_RESP_STATICFILE as Ctxt.OutContentType, then Ctxt.OutContent is the UTF-8 file name of a file which must be sent to the client. Note that it will work only with THttpApiServer kind of server (i.e. using high performance http.sys API). But whole file access and sending will occur in background, at the kernel level, so with best performance. See sample "09 - HttpApi web server" and HttpApiServer.dpr file.

colbert · 2013-03-13 17:56:38

Will This solve the memory issue?

How can I get the file in the client? Because now I'm using:

function DownloadFile(FileName: RawUTF8): Boolean;
var
  aResp: RawUTF8;
  aPathFile: string;
begin
  try
    aResp := Http.CallBackGetResult('SendFile', ['FileName', FileName]);
    aPathFile := ExtractFilePath(Application.ExeName) + UTF8ToString(FileName);
    if FileFromString(Base64ToBin(aResp), aPathFile) then
    begin
      FileSetDate(aResp, DateTimeToFileDate(GetDateFile(FileName))); //GetDateFile is one of my functions
      Result := True
    end
    else
      MsgErro('Error on download');

  except
    Result := False;
  end;
end;

I tried to use this code without the Base64ToBin method, but debugging I saw

aResp = ''

in other words I can't get the result with this line of code

aResp := Http.CallBackGetResult('SendFile', ['FileName', FileName]);

Last edited by colbert (2013-03-13 18:13:08)

ab · 2013-03-13 19:42:58

Do not call CallBackGetResults, since it is not a JSON response.

colbert · 2013-03-13 20:40:33

Ok, what method is used to handle JSON in this situation?

I didn't find anything like it in the SAD documentation about Method-Based service.

Last edited by colbert (2013-03-13 20:45:53)

ab · 2013-03-14 09:11:42

The CallBackGet() method is the one to be used, as stated by the SAD document, in its second part.

The corresponding client method may be defined as such:

function TMyClient.GetFile(const aFileName: RawUTF8): RawByteString;
begin
if CallBackGet('GetFile',['filename',aFileName],RawUTF8(result))<>HTML_SUCCESS then
raise Exception.CreateFmt('Impossible to get file: %s',[result]);
end;

I've updated the SAD document to add some sample code in its first part, when presenting method-based services.
Hope it helps.

Thanks for the feedback!

mORMot Open Source

#1 2013-03-11 17:48:21

Compress a File

#2 2013-03-11 19:04:32

Re: Compress a File

#3 2013-03-11 19:08:34

Re: Compress a File

#4 2013-03-11 19:23:31

Re: Compress a File

#5 2013-03-11 19:29:57

Re: Compress a File

#6 2013-03-12 13:50:18

Re: Compress a File

#7 2013-03-12 14:08:40

Re: Compress a File

#8 2013-03-13 15:51:23

Re: Compress a File

#9 2013-03-13 16:18:21

Re: Compress a File

#10 2013-03-13 17:56:38

Re: Compress a File

#11 2013-03-13 19:42:58

Re: Compress a File

#12 2013-03-13 20:40:33

Re: Compress a File

#13 2013-03-14 09:11:42

Re: Compress a File

Board footer