Bug when returning base64 rawblob fields in json

dcoun · 2022-08-07 08:44:05

In the following REST response from Mormot2 everything seems ok and even with Postman nothing seems to be a problem. Mormot2 as a client consumes it ok.

{
    "result": [
        {
            "RowID": 1,
            "user": "jpapa",
            "pass": "",
            "dedo": "￰4EfSGzZt5QLJGW1rQlMIfjxAsj0RxnlOHqYguRB8upNmh674R5F4hxNHirHgYMDoliW9mxVTQt1ArUTieKXzTqkHKIrakbeIYV2AajqlOHSacOP9H4N+/D7asfAFFwJAB/raCA5Fo/AvWPzK5ddxyF99U+RHh+BcPhido5z+jMzOE/9Jmib6lR6pxHAV29wJuszq2RvvYcvKFD4CQxXNQGe8zgdjaZct3oZR4ENjIjkB8bIIjf+pKA2Af2YzHM8eigTxV6PJjlHDvCBAxLWnk2H1GExkdvwS",
            "act": 1
        },
        true
    ]
}

A flutter client does not like the above response and can not decode the base64 field
From Postman I save as file the REST response of Mormot2 and I open it with a binary viewer. Now the file contents can be shown as the following (check the beginning of the Rawblob encoded field):

{"result":[{"RowID":1,"user":"jpapa","pass":"","dedo":"οΏ°4EfSGzZt5QLJGW1rQlMIfjxAsj0RxnlOHqYguRB8upNmh674R5F4hxNHirHgYMDoliW9mxVTQt1ArUTieKXzTqkHKIrakbeIYV2AajqlOHSacOP9H4N+/D7asfAFFwJAB/raCA5Fo/AvWPzK5ddxyF99U+RHh+BcPhido5z+jMzOE/9Jmib6lR6pxHAV29wJuszq2RvvYcvKFD4CQxXNQGe8zgdjaZct3oZR4ENjIjkB8bIIjf+pKA2Af2YzHM8eigTxV6PJjlHDvCBAxLWnk2H1GExkdvwS","act":1},true]}

Three bytes not acceptable for base64 encoded data are in the beginning (EF BF B0) and it is always there whatever record I am requesting. Do we need this BOM as we expect to have only base64 content? If needed, why not in other text values also?

The above response is provided by the following code in mormot2:

function Tintercode.Getuser(const id: int64; out userobj: TormDBuser):boolean;
begin result:=DBsrv.Orm.Retrieve('RowID=:('+inttostr(id)+'):',userobj,'RowID,user,dedo,act');  end;

It seems a pointer problem that provides the base64 encoded content

Last edited by dcoun (2022-08-07 08:59:20)

ab · 2022-08-07 12:35:49

This is the "base 64 with magic" format, allowing the framework to detect that a buffer is in fact base64 encoded.
This is as documented - see JSON_BASE64_MAGIC_C and the "magic"' parameter or functions.

As simple workaround, for a non-mORMot client, you can just use a RawUtf8 parameter and make the BinToBase64() call (from mormot.core.buffers.pas) in your code.

dcoun · 2022-08-07 12:53:58

Apologies for not noticing it in the documentation. To be honest I check the TsqlRawBlob and it is not mentioned there. My thought is that this violates the standard for JSON.
If it is needed only to have to be base64 decoded for RawBlob fields and to be recognized by the framework, every content from REST before entered to a RawBlob field must anyway be base64 decoded or am I missing something?
I was afraid that this could be a kind of feature, but please help me understand the need for that.
I will follow your suggestion anyway and thank you

Last edited by dcoun (2022-08-07 13:01:57)

ab · 2022-08-12 09:11:20

It does not violate the JSON standard.
It is a perfect valid UTF-8 encoded string, and JSON does not make an request about base-64 encoding.

It is needed internally by the framework when processing the JSON emitted by the DB drivers, to quickly and with no error possible identify what is inside a JSON string, if it is some text or some binary.
There is also a "magic marker" for ISO-8601 dates, for the very same reason.

In fact, it is the lack of an official binary format in JSON which made this.
For instance, in BSON we have binary and date formats, so such an hack is not needed.

mORMot Open Source

#1 2022-08-07 08:44:05

Bug when returning base64 rawblob fields in json

#2 2022-08-07 12:35:49

Re: Bug when returning base64 rawblob fields in json

#3 2022-08-07 12:53:58

Re: Bug when returning base64 rawblob fields in json

#4 2022-08-12 09:11:20

Re: Bug when returning base64 rawblob fields in json

Board footer