JSON encoding/decoding of special (null) chars

igors233 · 2017-02-19 04:31:51

I've been trying to use TSynDictionary to store some Variant values containing binary data (small files), problem is that null chars are not handled, they are dropped when SynDictionary is saved with SaveToJSON or SaveToBinary.
I've checked RFC4627 and control chars are allowed (U+0000 through U+001F) if they're properly escaped.
Problem is not inside TSynDictionary but with general JSON loading, string with binary data is correctly saved to Variant and loaded but it's not correctly saved or loaded as JSON.

For example:

const
  BIN_JSON = '{"Field1":"Value1_1", "Field2":"Value\u0000\u00012_1"}';
var
  Temp: TDocVariantData;
  s, s2: RawUTF8;
begin
  s := 'Test'#0#1#2'1234';
  v := RawUTF8ToVariant(s);
  s2 := VariantToUTF8(v);
  if s = s2 then
    OutputDebugString('OK');
  s2 := VariantSaveJSON(v);
  if s = s2 then
    OutputDebugString('OK');

  v := RawUTF8ToVariant(s);
  AddToDisplay(VariantSaveJSON(v));
  Temp.InitJSON(BIN_JSON);
  s := Temp.ToJSON;
  if s = BIN_JSON then
    OutputDebugString('OK');
end;

Last edited by igors233 (2017-02-23 15:35:35)

ab · 2017-02-19 07:15:23

#0 is not allowed, since in practice it is not handled by most string libraries as end of string.
Your content is not text, it is in fact binary: use hexadecimal or base64 encoding.

igors233 · 2017-02-19 14:53:09

ab wrote:

#0 is not allowed, since in practice it is not handled by most string libraries as end of string.
Your content is not text, it is in fact binary: use hexadecimal or base64 encoding.

#1 is also not accepted and some other chars below 15 but they are valid JSON chars according to RFC. I have a workaround, so just wanted to point out that JSON handling is not fully to the specs...
If it's as designed, that's OK, just please update the docs for that part.

ab · 2017-02-19 15:35:10

#1 and #2 are accepted, and handled are expected.
There are specific regression tests for control chars JSON escape.

Your test code is broken.
Here is some correct test code:

procedure Test;
const
  BIN_JSON = '{"Field1":"Value1_1","Field2":"Value\u0001\u00022_1"}';
var
  Temp: TDocVariantData;
  s, s2: RawUTF8;
  v: variant;
begin
  s := 'Test'#1#2'1234';
  v := RawUTF8ToVariant(s);
  s2 := VariantToUTF8(v);
  if s = s2 then
    writeln('OK'); // no JSON conversion, just to text conversion
  s2 := VariantSaveJSON(v);
  if s2 = s then
    writeln('FAILED');
  if s2 = '"Test\u0001\u00021234"' then
    writeln('OK'); // this is the expected JSON escape

  Temp.InitJSON(BIN_JSON);
  s := Temp.ToJSON;
  if s = BIN_JSON then
    writeln('OK');
end;

igors233 · 2017-02-20 02:17:47

> #1 and #2 are accepted, and handled are expected.
> There are specific regression tests for control chars JSON escape.
> Your test code is broken.
> Here is some correct test code:

You're right, sorry for the mixup. I've missread my test where #0 isn't handled and thought #1 wasn't as well.
It would be good to update docs (if not already), to mention that SaveJSON/LoadJSON doesn't support null chars (binary data).

ab · 2017-02-20 08:23:53

I've enhanced documentation details about JSON serialization.

See https://synopse.info/fossil/info/cb41eada97

mORMot Open Source

#1 2017-02-19 04:31:51

JSON encoding/decoding of special (null) chars

#2 2017-02-19 07:15:23

Re: JSON encoding/decoding of special (null) chars

#3 2017-02-19 14:53:09

Re: JSON encoding/decoding of special (null) chars

#4 2017-02-19 15:35:10

Re: JSON encoding/decoding of special (null) chars

#5 2017-02-20 02:17:47

Re: JSON encoding/decoding of special (null) chars

#6 2017-02-20 08:23:53

Re: JSON encoding/decoding of special (null) chars

Board footer