mORMot Open Source

zed

Member

From: Belarus

Registered: 2015-02-26

Posts: 105

Bug with parsing JSON array of double in TDocVariantData

Test, to reproduce a bug:

uses
  mORMot,
  SynCommons,
  SysUtils;

procedure DoTest;
const
  // some random geographic coordinates
  VJSON_In: array [0..1] of RawUTF8 = (
    '{"ArrayOfDouble":[45.2,55.9]}',
    '{"ArrayOfDouble":[45.264823,55.914998]}'
  );
var
  I: Integer;
  VDoc: TDocVariantData;
  VJSON_Out: RawUTF8;
begin
  for I := 0 to Length(VJSON_In) - 1 do begin
    VDoc.InitJSON(VJSON_In[I],[dvoValueCopiedByReference]);
    VJSON_Out := VDoc.ToJSON();
    if VJSON_Out <> VJSON_In[I] then begin
      raise Exception.CreateFmt('Test %d failed!', [I + 1]);
    end;
  end;
end;

First test is Ok, second - Failed, because double values became a strings:

'{"ArrayOfDouble":["45.264823","55.914998"]}'

TSQLRestStorageMongoDB.DocFromJSON() uses method doc.InitJSON() and it corrupt MongoDB json request, so Doubles inserts to DB as Strings.

It seems, that this bug exists at least 3 month, but discovered only now.

Last edited by zed (2017-02-01 17:47:40)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,649

Website

Re: Bug with parsing JSON array of double in TDocVariantData

It is not a bug.
You need to explicitly set the dvoAllowDoubleValue option.

As documented:

- by default, only integer/Int64/currency number values are allowed, unless
dvoAllowDoubleValue is set and 32-bit floating-point conversion is tried,
with potential loss of precision during the conversion

Which is exactly what you observed.

See also https://synopse.info/files/html/Synopse … l#TITLE_42

PBa

Member

From: Austria

Registered: 2017-01-04

Posts: 18

Re: Bug with parsing JSON array of double in TDocVariantData

IMHO the reason for this issue is the same as for the issue described in https://synopse.info/forum/viewtopic.php?id=3739 and was most probably introduced with {3129} [aa35ac110a].

If the TDocVariantOptions does not include [dvoAllowDoubleValue], then JSON encoding/decoding will return any double value with more than 4 digits behind the decimal separator as a string value instead of a double value.

My idea was to include [dvoAllowDoubleValue] in your line

VDoc.InitJSON(VJSON_In[I],[dvoValueCopiedByReference]);

like

VDoc.InitJSON(VJSON_In[I],[dvoValueCopiedByReference, dvoAllowDoubleValue]);

but unfortunately this does not work as expected. It seems that VDoc.InitJSON does not consider an included [dvoAllowDoubleValue] option in opposite to e.g. VariantLoadJSON.

Unfortunately I didn't succeed to find out how to circumvent this issue when using VDoc.InitJSON as in your example. Perhaps somebody else in the forum may help.

Paul

zed

Member

From: Belarus

Registered: 2015-02-26

Posts: 105

Re: Bug with parsing JSON array of double in TDocVariantData

ab
You change old behavior and this broke all old code. Include your own code in TSQLRestStorageMongoDB.DocFromJSON(). You tell me to use dvoAllowDoubleValue option, but this doesn't work, I tried!

My main problem is that now I can't insert double values into MongoDB, because your code in TSQLRestStorageMongoDB.DocFromJSON() is broken and have a bug. Please, fix it.

Last edited by zed (2017-02-02 07:30:25)

zed

Member

From: Belarus

Registered: 2015-02-26

Posts: 105

Re: Bug with parsing JSON array of double in TDocVariantData

I propose return old behavior (process double values by default) and revert option dvoAllowDoubleValue to dvoDontAllowDoubleValue and use it in all places where you need strings.

zed

Member

From: Belarus

Registered: 2015-02-26

Posts: 105

Re: Bug with parsing JSON array of double in TDocVariantData

Oh, I found this breaking change commit: https://github.com/synopse/mORMot/commi … 3aec5db617 is it really so necessary to make this change? Have you some bugs without it? Because with it we definitely have a bugs. I hope you will revert it and set param AllowVarDouble to True by default.

And I think that is not obvious behavior, processing numbers with 4 digits after dot as a doubles, but with 5 digits as a strings. By default, behavior must be consistent.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,649

Website

Re: Bug with parsing JSON array of double in TDocVariantData

I'm not sure reverting to double may be the best solution, since double values may loose precision during JSON serialization process.

But there was some bugs.
I've fixed several identified issues, especially in InitJSON and for MongoDB.
See https://synopse.info/fossil/info/68271abc82
Sorry for the inconvenience.

zed

Member

From: Belarus

Registered: 2015-02-26

Posts: 105

Re: Bug with parsing JSON array of double in TDocVariantData

Yes, this commit fix my issue with MongoDB. Thanks!