CrossPlatform* and BLOBs - How to LazyLoad?

MMihelic · 2016-09-06 16:32:17

Hello Everyone,

God, I hope I haven't missed this explained in the forum, documentation or in samples. I really don't won't to show with my first post that I like to skip my homework.

My working environment is: D10.1/FMX targeting Android, iOS and Win.

I was trying to do something that I assumed was natural to the way mORMot is designed, albeit using the CrossPlatform* units:

First fetch record set with simple fields only and then, when needed,
fetch BLOBs using efficient binary download (GET) and, at the end,
whenever BLOBs are small enough to hold in memory, to inject them into the existing records without triggering the changes tracking on the record.

I was expecting to find at least some helping methods published if the entire operation were not managed by the build-in logic.
What I found, at least at first glance, was just the opposite: no helpers and no examples. It looks like that even the most basic methods needed (for instance: to calculate the URI to the resource/BLOB) are private and that there are no helpers available.

Please, don't take this as criticism, I was genuinely surprised that it seems like I am the only one trying to do it in this way.

I'd like to ask you if you can give me any pointers and suggestions on how to approach this task. I would really appreciate a short answer to each of the items on my list above. Even if it is just a pointer to a keyword that I can search for by myself in the documentation.

I do have one additional question:

I would like to use the cross-OS abstraction (and automatic authorization, etc.) given by the CrossPlatform* units to download binary resources as well. Is there a suggested way on how to use the REST methods to, for instance, fetch a JPEG image as binary to a ByteArray?

At the and I'd just like to say that I have a lot of appreciations for your work. The amount of documentation is daunting but the sheer completeness of the framework you have created is amazing.

-- Kind regards, Matej.

Last edited by MMihelic (2016-09-06 16:37:01)

ab · 2016-09-06 17:12:54

Thanks for the feedback!

In fact, the BLOBs are only retrieved as JSON from the server, in SynCrossPlatformRest.pas.

By default, they are not retrieved (as with mORMot.pas) - as you expected.
But if you set '*' or a CSV of fields, including blobs, they are retrieved as base-64 encoded JSON fields - as you don't expect.
This is not consistent with what mORMot.pas does... but was done so for code simplicity.

So we need to enhance SynCrossPlatformRest.pas to have a dedicated RetrieveBlob() method, as for mORMot.pas.
It is currently missing on cross-platform client side.
On the server side, it is already ready: just use GET root/tablename/tableidnum/blobfieldname and you will retrieve the blob as one efficient binary.

Please try to implement it in SynCrossPlatformRest.pas, post some code in PasteBin, or fork the project, and we will fix and include it to the trunk.

MMihelic · 2016-09-06 17:31:59

Well, this would be my first attempt to get the BLOB separately:

procedure TTabbedForm.DataGeneratorAdapter1AfterScroll(Adapter: TBindSourceAdapter);
var
  LStream: TMemoryStream;
  LCall: TSQLRestURIParams;
begin
  LStream := TMemoryStream.Create;
  try
    if not Assigned(TSQLBiolife(Adapter.Current).Graphic) then
    begin
      if Proxy.Connect then
      begin
        Proxy.CallBackGet('Graphic',[], LCall, TSQLBiolife, TSQLBiolife(Adapter.Current).ID);
        LStream.Write(Pointer(LCall.OutBody)^, Length(LCall.OutBody));
      end
    else
      LStream.Write(Pointer(TSQLBiolife(Adapter.Current).Graphic)^, Length(TSQLBiolife(Adapter.Current).Graphic));
    end;
    ImageControl1.Bitmap.LoadFromStream(LStream);
  finally
    LStream.Free;
  end;
end;

ab · 2016-09-07 16:50:50

Please check http://synopse.info/fossil/info/d9d8322dfe
(not fully tested: we rely on you!)

MMihelic · 2016-09-08 05:26:05

Good morning.

What a pleasant and welcome surprise! I better get down to testing, then.

I'll get back with the results.

Thank you.

Last edited by MMihelic (2016-09-08 10:58:41)

MMihelic · 2016-09-10 21:13:13

For a start, a word of caution: I am a beginner is so many ways that I actually find it amusing. I am not a Delphi programmer, and I have never before worked with the Firemonkey, Livebindings or even with Delphi TDatasets. Everything is new to me. If anything I write sounds weird, wrong or over-complicated. I would welcome a constructive comment. I usually work at a design level.

This approach used in the SynCrossPlatformREST and SynCrossPlatformJSON to address the TObjectList definition is not a good one. It leads to problems due to the fact that projects that use the TObjectList-based LiveBindingsAdapter have to use the "System.Generics.Collections" and not "Contnrs". The result is a nasty type definition conflict.
```
{$ifdef NEXTGEN}
  System.Generics.Collections,
{$else}
  Contnrs,
{$endif NEXTGEN}
```
I actually had to define NEXTGEN globally in the project to get it working Windows platform project because I found no way around the type definition conflict.

As I was most interest in binary uploads and downloads of the BLOBs I've added and successfully used the following method:

function TSQLRestClientURI.UpdateBlob(Table: TSQLRecordClass; aID: TID; const BlobFieldName: string;
  const BlobData: TSQLRawBlob; const ContentType: string = 'application/octet-stream'): boolean;
var tableIndex: Integer;
    Call: TSQLRestURIParams;
begin
  tableIndex := Model.GetTableIndexExisting(Table);
  Call.Init(getURIID(tableIndex,aID)+'/'+BlobFieldName,'PUT','');
  Call.InBody := THttpBody(BlobData);
  if not ContentType.IsEmpty then
    Call.InHead := trim('Content-Type: '+ContentType);
  URI(Call);
  result := Call.OutStatus=HTTP_SUCCESS;
  Log(LOGLEVELDB[result],'%s.UpdateBlob(ID=%d,"%s") len=%d',
    [Model.Info[tableIndex].Name,aID,BlobFieldName,length(BlobData)]);
end;

The SynCrossPlatformSpecific.TWinHttpConnectionClass.URI adds Content-Type twice. When Content-type is already specified in the InHead then the TSQLRestClientHTTP.InternalURI will extract this value correctly and store it in the InDataType. However, it will not remove it from the original header. The duplicate Content-Type header is then appended in the low-level URI method call.
I have changed the following to get around it:

procedure TSQLRestClientHTTP.InternalURI(var Call: TSQLRestURIParams);
var inType: string;
    retry: integer;
begin
  inType := FindHeader(Call.InHead,'content-type: ');
  if inType='' then begin
    if OnlyJSONRequests then
      inType := JSON_CONTENT_TYPE else
      inType := 'text/plain'; // avoid slow CORS preflighted requests
//:MM:Avoid adding the Content-Type header twice
    //Call.InHead := trim(Call.InHead+#13#10'content-type: '+inType);
  end else
    inType := '';
//EOF:MM:Avoid adding the Content-Type header twice
  if fCustomHttpHeader<>'' then
...

D10.1/Android build: I had to use "Indy" based HTTP transport for the Android client to work! That is contrary to the comments within the code.
When using the "Indy" HTTP transport ProxyPort does not get assigned. WinHTTP parses port number from ProxyServer parameter, but the Indy does not.

I have already told you that I had to add an additional method override to the RetrieveList method. For me, this was the easiest way to work with the TListBindSourceAdapter which wraps around the TSQLRecords' object list. I found this approach worked well for me as I would empty the list and reload it instead of recreating it.

procedure TSQLRest.RetrieveList(Table: TSQLRecordClass; const FieldNames,
  SQLWhere: string; const BoundsSQLWhere: array of const; var destObjectList: TObjectList);
var rows: TSQLTableJSON;
    rec: TSQLRecord;
begin
  destObjectList.Clear;
  rows := MultiFieldValues(Table,FieldNames,SQLWhere,BoundsSQLWhere);
  if rows<>nil then
    try
      repeat
        rec := Table.Create;
        if not rows.FillOne(rec) then begin
          rec.Free;
          break;
        end;
        destObjectList.Add(rec);
      until false;
    finally
      rows.Free;
    end;
end;

I won't comment upon the Update method when working with BLOBs. I had some JSON encoding problems that have most probably resulted from conflicts in strings processing that I have caused by using the NEXTGEN define on the Windows platform. It's quite amazing that so much of the functionality worked correctly.

Have a nice Sunday!

Regards, Matej.

mORMot Open Source

#1 2016-09-06 16:32:17