mORMot Open Source

sokohigh

Member

Registered: 2017-02-27

Posts: 3

Perhaps an error in JSON serialization

Dear Arnaud!

I guess I found an error in SynCommons.TTextWriter.AddAnyAnsiBuffer (see red coloured lines)

procedure TTextWriter.AddAnyAnsiBuffer(P: PAnsiChar; Len: integer;
  Escape: TTextWriterKind; CodePage: Integer);
var B: PUTF8Char;
begin
  if Len>0 then
  case CodePage of
  CP_UTF8, CP_RAWBYTESTRING:
    Add(PUTF8Char(P),Len,Escape);  // direct write of RawUTF8/RawByteString content
  CP_UTF16:
    AddW(PWord(P),0,Escape); // direct write of UTF-16 content
  CP_SQLRAWBLOB: begin
    AddNoJSONEscape(@PByteArray(@JSON_BASE64_MAGIC_QUOTE_VAR)[1],3);
    WrBase64(P,Len,false);
  end;
  else begin
    // first handle trailing 7 bit ASCII chars, by quad
    B := pointer(P);
    if Len>=4 then
      repeat
        if PCardinal(P)^ and $80808080<>0 then
          break; // break on first non ASCII quad
        inc(P,4);
        dec(Len,4);
      until Len<4;
    if (Len>0) and (P^<#128)  then
      repeat
        inc(P);
        dec(Len);
      until (Len=0) or (P^>=#127);
    Add(B,P-B,Escape);
    if Len=0 then
      exit;
    // rely on explicit conversion for all remaining ASCII characters
    TSynAnsiConvert.Engine(CodePage).InternalAppendUTF8(P,Len,self,Escape);
  end;
  end;
end;

and if we look at where Add method leads we will see this:

procedure TTextWriter.AddJSONEscape(P: Pointer; Len: PtrInt);
var i,c: integer;
label noesc;
begin
  if P=nil then
    exit;
  if Len=0 then
    Len := MaxInt;

thus, if first symbol > #127 then we will have two copies of source string in the result...

Actually in my case this leads to incorrect JSON RawUtf8 values, produced from Russian (cp1251) strings:
destination values consist of two parts: source 1251 symbols + utf8 encoded part.

Sorry for my English.

Sincerely, Dmitry.

Last edited by sokohigh (2017-02-27 17:15:04)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,661

Website

Re: Perhaps an error in JSON serialization

Please try https://synopse.info/fossil/info/afa2a689d5

Very sorry for the issue...

sokohigh

Member

Registered: 2017-02-27

Posts: 3

Re: Perhaps an error in JSON serialization

Arnaud, thank you!

sokohigh

Member

Registered: 2017-02-27

Posts: 3

Re: Perhaps an error in JSON serialization

Arnaud, how about this:

      until (Len=0) or (P^>=#127);

?

may be

      until (Len=0) or (P^>=#128);

?