mORMot Open Source

igors233

Member

Registered: 2012-09-10

Posts: 241

Problem with serializing TSynDictionary and TDynArray

I've encountered bug within TSynDictionary and tracked it to TDynArray, if values stored are variants containing doubles, they are saved correctly to JSON but not loaded from JSON.

var
  dict: TSynDictionary;
  v, v2: Variant;
  Key: RawUTF8;
  Value: Double;
  s: RawByteString;
  dArr: TDynArray;
  Arr: TVariantDynArray;
begin
  Dict := TSynDictionary.Create(TypeInfo(TRawUTF8DynArray), TypeInfo(TVariantDynArray), True);

  Key := 'Test';
  Value := Now;
  v := _Obj(['Time', Value], [dvoReturnNullForUnknownProperty, dvoAllowDoubleValue]);
  Dict.Add(Key, v);

  s := Dict.SaveToJSON;        // DocVariantData Time is stored well
  Dict.LoadFromJSON(s);       // But it's read as string

  Dict.FindAndCopy(Key, v2);

  Value := VariantToDoubleDef(v.Time, 0);    
  WriteLn(FloatToStr(Value));

  Value := VariantToDoubleDef(v2.Time, 0);   // Will return 0 since Time is now string value
  WriteLn(FloatToStr(Value));

  // Same problem demonstrated with TDynArray
  dArr.Init(TypeInfo(TVariantDynArray), Arr);
  dArr.Add(v);

  s := dArr.SaveToJSON;
  dArr.LoadFromJSON(PUtf8Char(s));

  dArr.Peek(v2);
  Value := VariantToDoubleDef(v2.Time, 0);
  WriteLn(FloatToStr(Value));

This is happening due to changes on how double values are stored/read in JSON, for TDOcVariantData that could be controlled with dvoAllowDoubleValue but there's no such options for TSynDictionary and TDynArray (possibly others using VariantLoadJSON).
As is now, TDynArray during JSON load calls VariantLoadJSON and passes JSON_OPTIONS[true] so there's no way to correcly load Double value anymore.
As far as I could see there are no settings to handle that so I had to make a local copy of SynCommons and change JSON_OPTIONS to include dvoAllowDouble which is really not a best option.
I believe many others could be affected with this problem and it would be a make few changes to make loading customizable. For example with:
- introduce paramatar to TSynDictionary/TDynArray to allowDoubles
- change JSON_OPTIONS from const to var and allow program to change default values when appropriate.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Problem with serializing TSynDictionary and TDynArray

I have added a new CustomVariantOptions parameter to TSynDictionary.LoadFromJSON.
See https://synopse.info/fossil/info/083b9bc9a1

Thanks for the report.

igors233

Member

Registered: 2012-09-10

Posts: 241

Re: Problem with serializing TSynDictionary and TDynArray

Great thanks!

zhyhero

Member

Registered: 2018-01-10

Posts: 10

Re: Problem with serializing TSynDictionary and TDynArray

Hi,I got this ERROR with Demo 1.

Building Project01.dproj (Debug, Win32)
brcc32 command line for "Project01.vrc"
dcc32 command line for "Project01.dpr"
[dcc32 Error] SynCommons.pas(60169): E2034 Too many actual parameters
[dcc32 Fatal Error] Project01.dpr(86): F2063 Could not compile used unit 'SynCommons.pas'

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Problem with serializing TSynDictionary and TDynArray

@zhyhero
With which compiler?
I am not able to reproduce the issue here.

Vitaly

Member

From: UAE

Registered: 2017-01-31

Posts: 168

Website

Re: Problem with serializing TSynDictionary and TDynArray

I have the same with all my projects with the latest mORMot commit.
Delphi 10.3.3

function TSynDictionary.LoadFromJSON(JSON: PUTF8Char; EnsureNoKeyCollision: boolean{$ifndef NOVARIANTS};
  CustomVariantOptions: PDocVariantOptions{$endif}): boolean;
var k,v: RawUTF8;
begin
  result := false;
  if not JSONObjectAsJSONArrays(JSON,k,v) then
    exit;
  fSafe.Lock;
  try
    if fKeys.LoadFromJSON(pointer(k),nil{$ifndef NOVARIANTS},CustomVariantOptions{$endif})<>nil then                     <------- here
      if fValues.LoadFromJSON(pointer(v),nil{$ifndef NOVARIANTS},CustomVariantOptions{$endif})<>nil then
...

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Problem with serializing TSynDictionary and TDynArray

Please check https://synopse.info/fossil/info/803c764bf1

Vitaly

Member

From: UAE

Registered: 2017-01-31

Posts: 168

Website

Re: Problem with serializing TSynDictionary and TDynArray

Yes, everything is compiling smoothly again. Thanks!