mORMot Open Source

Is it possible to specify on which pages AddPagesToFooterAt should apply or a starting page number from which it should be printed? When the pdf starts with a coverpage(s), it would be good to not print the footer on the first page.

The Accept-Encoding in the response header from Mormot seem to always be Accept-Encoding: synlz,gzip.  But it doesn't look like that affects the outcome.  It looks like compression works fine for small strings, but that bigger strings are chunked and therefor not compressed.  I was (mistakenly) thinking that the chunking happens after compression.  Does this mean that compression specifically only work for small packages?  Wouldn't that negate the advantage of compression?  Is there a property I can set to always enforce compression if deflate is in the request headers?

I've created a test program using the latest nightly to ensure that none of my code affects the outcome.  Does it do anything wrong?

  TMyRestServer = class(TSQLRestServerFullMemory)
  published
    procedure Hello(Ctxt: TSQLRestServerURIContext);
  end;

procedure TForm1.FormCreate(Sender: TObject);
begin
  aModel := TSQLModel.Create([],'my_test');
  aRestServer := TMyRestServer.Create(aModel);
  aHttpServer := TSQLHttpServer.Create('8080', [aRestServer]);
end;

procedure TMyRestServer.Hello(Ctxt: TSQLRestServerURIContext);
var
  data: RawUtf8;
begin
  data := StringFromFile('c:\sample_2mb.json');
  Ctxt.Returns(data) //not compressed, but chunked
  //Ctxt.Returns('{"hallo": "world"}'); //compressed
end;

Thanks for the quick reply.  Would something like this be possible using the wrappers available in SynOpenSSL?

I fully agree that there are advantages to a purely logical design.

However, the requirements of this task specify that a human readable file be created which can be stored offsite on a backup medium with no access to electricity.  That leaves me with either a .sql or a .json file.  Due to the custom nature of the specifications, just creating a sql dump file is not ideal.  So I've decided to create a json file per table which contains the table's field definitions (GetFields), index defs (GetIndexes), and data (FetchAllAsJSON(false)).

Restoring the structure seems very doable (SqlCreate, SQLAddIndex).  I'm not too sure about restoring the data.

Looks perfect.  Thanks!

Does anybody have an example of using TSQLRestServerAuthenticationHttpBasic that works?  I need to use basic authentication and look up the username and password from my own list (not orm).

I changed it like this and it seamed to work.  Not sure how different versions should be handled.  It is conceivable that someone might be using libodbc.so.2 and not the libodbc.so.1 that worked for me.

constructor TODBCLib.Create;
var P: PPointer;
    i: integer;
    dllname: string;
begin
  {$IFDEF LINUX}
    dllname := 'libodbc.so.1';
  {$ELSE}
    dllname := 'odbc32.dll';
  {$ENDIF}
  fHandle := SafeLoadLibrary(dllname);
  if fHandle=0 then
    raise EODBCException.Create('Unable to find ODBC Client Interface (' + dllname + ')');
  P := @@AllocEnv;
  for i := 0 to High(ODBC_ENTRIES) do begin
    P^ := GetProcAddress(fHandle,ODBC_ENTRIES[i]);
    if P^=nil then begin
      FreeLibrary(fHandle);
      fHandle := 0;
      raise EODBCException.CreateFmt('Invalid ' + dllname + ': missing %s',[ODBC_ENTRIES[i]]);
    end;
    inc(P);
  end;
end;   

I've added a simple parameter which would allow this function to return datasets as datasets, ie with fieldnames and rows:

function TDocVariantData.ToNonExpandedJSON(const SplitRows: boolean = false): RawUTF8;
var fields: TRawUTF8DynArray;
    fieldsCount: integer;
    W: TTextWriter;
    r,f: integer;
    row: PDocVariantData;
begin
  if VKind<>dvArray then begin
    result := '';
    exit;
  end;
  if VCount=0 then begin
    result := '[]';
    exit;
  end;
  with _Safe(VValue[0])^ do
    if VKind=dvObject then begin
      fields := VName;
      fieldsCount := VCount;
    end else
      fieldsCount := 0;
  if fieldsCount=0 then
    raise EDocVariant.Create('ToNonExpandedJSON: Value[0] is not an object');
  W := DefaultTextWriterJSONClass.CreateOwnedStream;
  try
    if SplitRows then
      W.Add('{"fieldCount":%,"rowCount":%,"fields":[',[fieldsCount,VCount])
    else
      W.Add('{"fieldCount":%,"rowCount":%,"values":[',[fieldsCount,VCount]);
    for f := 0 to fieldsCount-1 do begin
      W.Add('"');
      W.AddJSONEscape(pointer(fields[f]));
      W.Add('"',',');
    end;
    if SplitRows then
    begin
      W.CancelLastComma;
      W.Add('],"values":[',[]);
    end;
    for r := 0 to VCount-1 do begin
      if SplitRows then
        W.Add('[',[]);
      row := _Safe(VValue[r]);
      if (r>0) and ((row^.VKind<>dvObject) or (row^.VCount<>fieldsCount)) then
        raise EDocVariant.CreateUTF8('ToNonExpandedJSON: Value[%] not object',[r]);
      for f := 0 to fieldsCount-1 do
        if (r>0) and not IdemPropNameU(row^.VName[f],fields[f]) then
          raise EDocVariant.CreateUTF8('ToNonExpandedJSON: Value[%] field=% expected=%',
            [r,row^.VName[f],fields[f]]) else begin
          W.AddVariant(row^.VValue[f],twJSONEscape);
          W.Add(',');
        end;
      if SplitRows then
      begin
        W.CancelLastComma;
        W.Add('],',[]);
      end;
    end;
    W.CancelLastComma;
    W.Add(']','}');
    W.SetText(result);
  finally
    W.Free;
  end;
end;

It will return the nonexpanded json like this:

{
	fieldCount: 3,
	rowCount: 2,
	fields: 
	    [ 
		"DateData", "IntData", "StrData"
  	    ],
	values: 
	    [	
		[  "2016-03-14T13:23:25",  1,  "Hello"  ],
		[  "2016-03-14T13:23:45",  2,  "Bye"  ]
  	    ]
}

@mpv: Yes that might be the case when working on own projects, but I provide data for third parties.  They expect some standards, such as that a record only contain data for 1 record and not all the fields of all the records as one big list.

It feels a bit clumsy, but this should copy the data to allow exporting to the same json format as when expanded=false is used.  Does anyone have suggestions as to how this can be improved?

function CollapseJSON(const data: RawUTF8): RawUTF8;
var
  Source, Target: TDocVariantData;
  TargetValues, valueList : variant;
  fieldCnt, rowCnt, I, J: integer;
begin
  Source.InitJSON(data);
  Target.InitFast;
  TargetValues := _ArrFast([]);
  fieldCnt := 0;
  rowCnt := Source.Count;

  if rowCnt > 0 then
  begin
    valueList := Source.Value[0];
    fieldCnt := DocVariantData(valueList)^.Count;
    Target.Value['fieldCount'] := fieldCnt;
    for I := 0 to fieldCnt -1 do
      DocVariantData(TargetValues)^.AddItem(DocVariantData(valueList)^.Names[i]);

    for I := 0 to rowCnt -1 do
    begin
      valueList := Source.Value[i];
      for J := 0 to fieldCnt -1 do
        DocVariantData(TargetValues)^.AddItem(DocVariantData(valueList)^.Value[J]);
    end;
  end;
  Target.AddValue('values', TargetValues);
  Target.Value['rowCount'] := rowCnt;

  Result := Target.ToJSON;
end;

function RecordToCollapsedJSON(const data; const RecTypeInfo: pointer): RawUTF8;
var
  Source: TDocVariantData;
begin
  Source.InitFromTypeInfo(data, RecTypeInfo, true, []);
  result := CollapseJSON(Source.ToJSON);
end;

Using the initial example, this can then be used like so:

  result := RecordToCollapsedJSON(mytype, TypeInfo(TMyTypeArray));

Hi ab, I would like to know something similar to the above question.

I use TSQLDBConnectionProperties.Execute to return ISQLDBRows and then send ISQLDBRows.FetchAllAsJSON to my 3rd party clients.

There are 2 scenarios where I need to change the data before sending it as a result, and I want to ensure that the same format as FetchAllAsJSON is always used:
  1) I must step through the data and change the content of some fields (such as hide sensitive values)
  2) I must remove some records from the set or add additional records to the result set.

What would be the preferred way of doing this?  The queries are very complicated and the data require additional work before being sent as a JSON result.  Do I send the ISQLDBRows to a memory table, or is there a way of directly making changes to the resulting dataset?

Am I doing something wrong?

@ab  Would it be possible to use this method to serve files without explicitly calling the Files function?  Similarly, is it possible to have an empty root?

So that for example this url: http://localhost:8080/root/files/myfile.html
would become: http://localhost:8080/myfile.html

I assume that one way of using the files method as default would be to register it as an error handler for Error 400s, which would allow me to use custom pages for errors such as 500, 404 etc

Any advice?

Usually I would agree that a username and password is enough.  But sometimes it is important to ensure that 3rd party Applications can not use some parts of your code, regardless of whether they have the admin username and password.

Thanks, I'll take a look at Fingerprintjs.  I suppose if Fingerprintjs can give me the real url of the request page (Referer in header can be spoofed) I can make an assumption on serverside and either allow it or not.
Already using https.  I'm still worried that any 3rd party developer can open my client and with Firebug get the url of for example myapp/login_funcs.js and just use the same (instead of publicapi/login_funcs.js) to get access to the full api.

You are right that code can be changed live in browers.  By readonly I mean that an administrator can't override the JS with old / custom versions without either replacing the exe (full set) or rerouting the urls.  Being able to release a full set of code together as one version is fantastic.

@daniel: I fully agree about client side, opening firebug and making changes to loaded code is childsplay.  But the issue here is server side when you distribute software to sites that you do not administer yourself.  In this scenario, you want the js files to always be at the same version as the exe (updated together) and not give the local site admins the option to permanently change the js files on server side to impact an sla. 

Admins are supposed to be the responsible ones, but that does not change reality.  When you deal with partial upgrades or custom changes done by someone else's admins and it impacts your application's stability, you want to package as much together as possible.

Thanks, this looks exactly like what I need.  Were there any recent changes to UTF8ToString and ReturnBlob?  The compiler doesn't expect the fn parameter as well in these 2 functions.

Perfect! Thanks!  Will test the change tomorrow after downloading the nightly build.

Looks like the parsing is very fragile.  Adding more parameters to the function highlights that some exact format is required, even though it succeeds in browser parsing.

Changing the function to:

function TStaff.FilterStaff(const filters: TFilterList; Limit: integer): TFilterList;

and the datastring to

var dataString = '{"filters":[{"name":"name1","value":"value1","comparetype":">"},{"name":"name2","value":"value2","comparetype":"="}], "Limit":"100"}';

gives the 406 same issue again, but the json validates successfully on sites such as jsonschema.

Using this format, works, provided that the correct type (integer vs string) is passed.  Note that this json doesnt validate on sites such as jsonschema.net:

var dataString = '{"filters":[{"name":"name1","value":"value1","comparetype":">"},{"name":"name2","value":"value2","comparetype":"="}]}, {"Limit":"100"}';

This works for integer Limit parameter:

var dataString = '{"filters":[{"name":"name1","value":"value1","comparetype":">"},{"name":"name2","value":"value2","comparetype":"="}]}, {"Limit":100}';