TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

Paul · 2017-01-04 15:05:46

I'm new to mORMot as well as I've been reading the documentation and the posts of this forums for more than a year now and I'm fascinated what this framework does provide (very well done!!).

Installing mORMot was no big issue and the TestSQL3.exe is running smoothly but for one test scenario:
...
1.2. Low level types:
...
! - Encode decode JSON: 100 / 282,875 FAILED 133.68ms
...
...
Using mORMot 1.18.3310 ERTL FTS3
Running on Windows 7 64bit SP1 (6.1.7601) with code page 1252
TSQLite3LibraryStatic 3.16.0 with internal MM
Generated with: Delphi 7 compiler

Time elapsed for all tests: 83.30s
Tests performed at 04.01.2017 15:44:35

Total assertions failed for all test suits: 100 / 26,702,774
! Some tests FAILED: please correct the code.

I tracked this failure down to SynSelfTests.pas lines
6151: r := Random;
and
6197: CheckSame(Va,r);

In case when the decimal separator is set to "," instead of "." then line 6197 raises the failure as e.g. r = 0,0123456789 ("," as decimal separator) but Va = '0.0123456789'.

In case when the decimal separator is set to "." then line 6197 doesn't fail as r = 0.0123456789 ("." as decimal separator) and Va = '0.0123456789'

The same issue arises when using Delphi RAD Studio Berlin 10.1

As I'm usually using "," as decimal separator (and changing the decimal separator to "." is not an option) my question is how to circumvent this failure?

Thank you very much!
Paul

ab · 2017-01-04 15:35:18

I will investigate and report here...

PBa · 2017-01-05 13:11:55

As it seems that my account "Paul" as well as my first post prior to AB's yesterday's answer has obviously been deleted, I just want to re-add my question for completeness:

Using the actual mORMot version TestSQL3.exe fails in EncodeDecodeJSON if system decimal separator is set to "," instead of "."
...
1.2. Low level types:
...
! - Encode decode JSON: 100 / 283,293 FAILED 4.26s
...
Using mORMot 1.18.3310 ERTL FTS3
Running on Windows 7 64bit SP1 (6.1.7601) with code page 1252
TSQLite3LibraryStatic 3.16.0 with internal MM
Generated with: Delphi 7 compiler

Time elapsed for all tests: 89.32s
Tests performed at 04.01.2017 12:48:02

Total assertions failed for all test suits: 100 / 26,703,849
! Some tests FAILED: please correct the code.

The same is true when using Delphi 10.1 Berlin compiler.

I've tracked down this issue to lines 6151 and 6194 in SysSelfTests.pas:

6151: r := Random;
...
6194: CheckSame(Va,r);

In case system decimal separator is set to ",": e.g. r = 0,0123456789 ("," as decimal separator) but Va = '0.0123456789' ("." as decimal separator) => line 6194 fails.
In case system decimal separator is set to ".": e.g. r = 0.0123456789 ("." as decimal separator) and Va = '0.0123456789' ("." as decimal separator) => line 6194 doesn't fail.

As the usual layout in Austria and German is using "," as decimal separator, changing the system decimal separator to "." is not an option. Thus my question was how to circumvent this failure?

Thank you very much!
Paul

PBa · 2017-01-21 19:00:18

Dear ab,

today I had time to search for the reason of this bug.
This bug was introduced with {3129} [aa35ac110a] and can be fixed as follows:

In SynSelfTests.pas (version {3343}) please add a new line 5737:

{$ifndef NOVARIANTS}
var Va, Vb: Variant;
    VaAllowDblVal: TDocVariantOptions; // <== added by PBa
    c: currency;
{$endif}

Initialize this variable e.g. in line 6165:

{$endif}
  VaAllowDblVal := [dvoAllowDoubleValue]; // <== added by PBa
  for i := 1 to 100 do begin
    a := Random(maxInt);

and change line 6209:

      Check(Va=a);
      P := VariantLoadJSON(Va,P,nil,@VaAllowDblVal); // <== ",nil,@VaAllowDblVal" added by PBa
      Check(P<>nil);
      CheckSame(Va,r);

Va is now a representation of a double value (Va = 0.0123456789 with English layout where system decimal separator is set to "." and Va = 0,0123456789 with German layout where system decimal separator is set to ",", respectively)
instead of a string representation (Va = '0.0123456789' as stated in my post 2017-01-05) where

CheckSame(Va,r);

will succeed only if decimal separator is "." but will fail if decimal separator is ",".

Please check if you can add these lines to trunk...

Thereafter TestSQL3.exe runs smoothly without any failed assertions (tested with Delphi 7 and Delphi Berlin 10.1).

Thank you very much!
Paul

ab · 2017-01-21 19:33:09

This is how variant conversion do work, I'm afraid.
We use internally only JSON number representation, with '.' as decimal separator.
Writing "Va = 0,0123456789" does not make sense, since Va contains a double value, and is displayed with a decimal separator by the IDE debugger.
Always use explicit conversions to/from variant, to avoid such issues.

PBa · 2017-01-21 20:35:00

Dear ab,
Perhaps I have described my finding in a misunderstanding manner, sorry.
I'm with you that you use internally only JSON number representation with '.' as decimal separator.
But in current implementation {3343} the line

P := VariantLoadJSON(Va,P);

doesn't return Va as a double but as a string value (displayed by the IDE debugger as '0.012345') as internally the variable TryCustomVariants of function VariantLoadJSON is set to nil.
Thus

CheckSame(Va,r);

reads as
a) in case of system decimal separator is set to '.': Checksame('0.012345', 0.012345); which succeeds
b) in case of system decimal separator is set to ',': Checksame('0.012345', 0,012345); which fails
as Checksame compares a string with a double value.

After adding the suggested changes in my previous post

P := VariantLoadJSON(Va,P,nil,@VaAllowDblVal);

sets the internal variable TryCustomVariants to [dvoAllowDoubleValue] and therefore this line returns Va as a double value (displayed by the IDE debugger as 0.012345 (in case of system decimal separator is set to '.') and as 0,012345 (in case of system decimal separator is set to ',')
Thus

CheckSame(Va,r);

reads now as
a) in case of system decimal separator is set to '.': Checksame(0.012345, 0.012345); which succeeds
b) in case of system decimal separator is set to ',': Checksame(0,012345, 0,012345); which succeeds, too.
as Checksame now compares a double value with a double value.

Thank you!
Paul

Last edited by PBa (2017-01-21 20:44:01)

PBa · 2017-01-29 18:39:21

ab wrote:

This is how variant conversion do work, I'm afraid.
We use internally only JSON number representation, with '.' as decimal separator.
Writing "Va = 0,0123456789" does not make sense, since Va contains a double value, and is displayed with a decimal separator by the IDE debugger.

Dear ab,
I'm with you that Va may be displayed with a decimal separator by your IDE debugger: I assume that your Windows system is set to a language which uses an English number format using '.' as a decimal separator. Then TestSQL3.exe JSON Encode Decode test suite does succeed.

ab wrote:

Always use explicit conversions to/from variant, to avoid such issues.

Sorry, this sentence I do not understand - I'm not talking about my own coding but about mORMot Testsuite "TestSQL3.exe" which - in my opinion exhibits a regression when using a Windows system where the system language is set to e.g. German which uses the German number format (which uses ',' as system decimal separator by default).

I've done a further review and made some screenshots to allow reproducing this issue: https://dl.dropboxusercontent.com/u/691 … eIssue.pdf

If you compile the latest Mormot self test suite 1.18.3360, set a breakpoint in line 6209 of SynSelfTests.pas

Checksame(Va,r);

and start TestSQL3.exe within your IDE you will see that Va does NOT contain the expected DOUBLE value but a STRING value. VarType(Va) returns 256 which correlates to a STRING content. Despite the fact that Va contains a STRING instead of the expected DOUBLE value and r is a DOUBLE value, Checksame(Va,r) does succeed if your Windows system is set to an English number format.
In case your Windows system is set to German number format, Checksame(Va,r) does fail.

After adding the suggested lines of code (as suggested in my post dated 2017-01-21 20:00:18) Va now does contain the expected DOUBLE value (and VarType(Va) would return 5 instead of 256).

With these changes TestSQL3.exe does succeed in any case independently of the number format settings of your Windows system.

Please check if you can reproduce and fix this issue in SynSelfTests.pas.
Thank you very much! And sorry for my English.
Paul

PBa · 2017-02-01 20:00:33

Dear ab,
this seems to be the same issue as your answer in https://synopse.info/forum/viewtopic.ph … 219#p23219

Could you please also add [dvoAllowDoubleValue] to the test case in SynSelfTests.pas as described above.

Thank you!
Paul

PBa · 2017-02-03 06:44:58

Dear ab,

yesterday you've fixed some [dvoAllowDoubleValue] issues in https://synopse.info/fossil/info/68271abc8213256f.
I've tested this commit but it does not solve my issue above, as in SynSelftests.pas you check JSON Encoding/Decoding of a double value without using [dvoAllowDoubleValue] which results in failing (at least if you use a German number format).
Please fix it, too (I've suggested some code in my posting of 21 Jan in this topic).

Thank you very much!
Paul

Last edited by PBa (2017-02-03 06:45:49)

ab · 2017-02-03 17:24:56

What about https://synopse.info/fossil/info/0e06277c02 ?

PBa · 2017-02-04 09:01:24

ab,
I've tested it with Delphi 7 and Delphi 10.1 Berlin compiler:
All tests passed successfully for both compilers, also when using German number format.

Thank you for fixing!

Paul

mORMot Open Source

#1 2017-01-04 15:05:46

TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

#2 2017-01-04 15:35:18

Re: TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

#3 2017-01-05 13:11:55

Re: TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

#4 2017-01-21 19:00:18

Re: TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

#5 2017-01-21 19:33:09

Re: TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

#6 2017-01-21 20:35:00

Re: TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

#7 2017-01-29 18:39:21

Re: TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

#8 2017-02-01 20:00:33

Re: TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

#9 2017-02-03 06:44:58

Re: TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

#10 2017-02-03 17:24:56

Re: TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

#11 2017-02-04 09:01:24

Re: TestSQL3.exe JSON Encode Decode fails when decimal separator is ","

Board footer