about UNICODE of DataSetToJSON in SynVirtualDataSet

profh · 2014-09-22 15:42:36

hi,ab

i think that the following code of function DataSetToJSON in SynVirtualDataSet.pas

          W.AddAnsiString({$ifdef UNICODE}AsAnsiString{$else}AsString{$endif},
            twJSONEscape);

should be

          {$ifdef UNICODE}
            W.AddAnsiString(AsAnsiString,twJSONEscape);
          {$else}
            W.AddString(AsString);
          {$endif}

ab · 2014-09-22 18:35:36

Why?
It is an ansistring field, right?

profh · 2014-09-22 23:20:22

it is a ftString, ftFixedChar or ftMemo one, like this:

        ftString, ftFixedChar, ftMemo: begin
          W.Add('"');
          W.AddAnsiString({$ifdef UNICODE}AsAnsiString{$else}AsString{$endif},
            twJSONEscape);
          W.Add('"');
        end;

if define UNICODE, should use W.AddAnsiString for AsAnsiString, else W.AddString for AsString, as follow:

        ftString, ftFixedChar, ftMemo: begin
          W.Add('"');
          {$ifdef UNICODE}
            W.AddAnsiString(AsAnsiString,twJSONEscape);
          {$else}
            W.AddString(AsString);
          {$endif}
          W.Add('"');
        end;

ab · 2014-09-23 07:16:43

It is IMHO pointless.

profh · 2014-09-23 08:47:06

but it is meaningful for me .

if i changed the code above, i always got the right data including chinese words from database, or sometimes broken data.

ab · 2014-09-23 09:08:07

Which version of Delphi are you using?
Your modification would not do anything with Unicode version of Delphi.
And for pre-Unicode version of Delphi, W.AddString() is expected to write UTF-8 content directly, so it would break the content.
What is you DB charset? If it is UTF-8, it should not be a ftString, but ftWideString.

The generated JSON is expected to be UTF-8 encoded!

profh · 2014-09-23 09:58:31

i use delphi7 and mysql, DB charset is UTF-8.

ab · 2014-09-23 11:24:20

So you should define the field as ftWideString, and not ftString.

profh · 2014-09-23 23:43:11

My field data type is varchar(255) with utf8-bin charset in mysql, and i use TZReadOnlyQuery of Zeoslib to query data from mysql, then call DataSetToJSON to get JSON data, so i don't know where to define the field as ftWideString, and not ftString.

Should i define UNICODE in Synopse.inc?

Thanks.

Last edited by profh (2014-09-24 00:35:04)

ab · 2014-09-24 08:55:23

No, do not change UNICODE in Synopse.inc - it would break the whole code.

By design, ftString should store the data in AnsiString, with the default system code page.
There may be an option of Zeos to handle the data as Unicode.

EgonHugeist · 2014-09-24 22:44:16

Endeed there is an option: TZConnection.ConstrolsCodePage = cCP_UTF16. But using cCP_UTF8 should be best fit for mORMot afaik!
Note using cCP_UTF16 is not possible for current Zeos + D7/D2005 -> ftWideString persits as enum but TField.AsWideString isn't supported and a TWideMemo-Field is missing on Delphi db.pas.

Hack without warranty:

you'll find {$IF defined(ZEOS_TEST_ONLY) and defined(TEST_ZFIELDS)} in ZAbstractRODataset.pas. So if you add these
I'm not 100% ready with current TZFields. But TZWideStringField should work as well for D7 too. Another issue is a WideMemo field -> not ready by my selves.

Have my problems to understand what exactly the propblems are. What is expected if mORMot does fill it's JSON contents for D7-D2007? I still start from the premisse UTF8?

moctes · 2014-10-24 00:35:55

Hi,

I'm bumping this thread because I have an issue with DataSetToJSON and I thought this was the best place.

First, some background: I have a project made with mORMot and Delphi 2007 which is running fine on production, basically it gets data from another application server (via some RPC calls), the former server return the data as memory datasets (TDataset descendant) these dataset are getting the records from a Firebird 2.5 database and the mORMot application server is mainly exposing the datasets as JSON using DataSetToJSON and everything is running as expected.

Now to the problem, I have been asked to port the project to Delphi XE3 and it compiles fine with no error whatsoever but a minor glitch on some strings, e.g.:

GARANTÍA

The former text is on a field of type string (ftString), debugging and setting a breakpoint on line 497 (SynVirtualDataset.pas) :

          W.AddAnsiString({$ifdef UNICODE}AsAnsiString{$else}AsString{$endif},
            twJSONEscape);

I can see the "AsString" or "AsAnsiString" give me the correct string, but after "W.AddAnsiString(..." the string changes to :

GARANTÃA

ATM the only way I can see the right char on the JSON output is if I change the former code to :

          W.AddShort(AsAnsiString);  // or W.AddShort(AsString)

But then I could lose data on some long strings, so I am looking for a definitive solution.

Any hints?

The environment:

Synopse framework: 1.18.381
wSeven_64 with code page 1252
SQlite3 engine: 3.8.6
Delphi XE3 compiler

Edit: The codepage on the firebird database is: ISO_8859_1, but I'm not making direct queries from mORMot, the data is queried from other software and I'm only consuming the returned datasets.

Last edited by moctes (2014-10-24 00:42:49)

profh · 2015-04-04 09:17:19

i just use XE7 from Delphi7, and the chinese words from database are also abnormal with function DataSetToJSON in SynVirtualDataSet.pas.

ab · 2015-04-04 09:26:00

Be aware that DataSetToJSON() returns an UTF-8 string content!

So the returned RawUTF8 content should be converted back to string with UTF8ToString(), if you want to use strings.
If you want some JSON, use directly RawUTF8, since JSON is meant to be UTF-8 encoded, as a transmission format.

Please make sure you are not making a confusion between UTF-8 and UTF-16 encodings.

profh · 2015-04-05 04:00:07

everything is ok under delphi 7.
under delphi XE7, function DataSetToJSON got the expected result also if not including chinese words.

i found that under delphi XE7 and delphi 7, function stringtoutf8 got different result with the same source string including chinese words.

ab · 2015-04-05 06:26:41

profh wrote:

i found that under delphi XE7 and delphi 7, function stringtoutf8 got different result with the same source string including chinese words.

StringToUTF8() should work the same under Delphi XE7 and Delphi 7, it is used in production since years, and has deep regression tests.

I'm afraid you are making some confusion.
"string" is not the same in Delphi 7 and XE7.

What is "different result"?
Do you have some code to reproduce?
Do StringToUTF8(UTF8ToString()) fails to convert some text?
Are you using files input and output files to check the content (do not trust the IDE debugger)?

profh · 2015-04-06 14:58:57

perhaps it is the difference between delphi XE7 and delphi 7, i try to understand it.

profh · 2015-04-06 15:21:41

i got it, in Delphi XE7, just as follow:

procedure TForm1.utf8ansiClick(Sender: TObject);
var
  sl:TStringlist;
  aa:tbytes;
begin
  sl:=TStringlist.Create;
  sl.LoadFromFile('c:\1.txt');
  Memo1.Lines.Add(sl.Text);
  aa := TEnCoding.UTF8.GetBytes(sl.Text);
  e1.Text := StringOf(aa);
  e2.Text := TEnCoding.utf8.GetString(aa);
  ......
end;

thanks,ab.

Last edited by profh (2015-04-06 15:22:34)

mORMot Open Source

#1 2014-09-22 15:42:36

about UNICODE of DataSetToJSON in SynVirtualDataSet

#2 2014-09-22 18:35:36

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#3 2014-09-22 23:20:22

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#4 2014-09-23 07:16:43

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#5 2014-09-23 08:47:06

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#6 2014-09-23 09:08:07

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#7 2014-09-23 09:58:31

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#8 2014-09-23 11:24:20

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#9 2014-09-23 23:43:11

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#10 2014-09-24 08:55:23

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#11 2014-09-24 22:44:16

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#12 2014-10-24 00:35:55

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#13 2015-04-04 09:17:19

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#14 2015-04-04 09:26:00

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#15 2015-04-05 04:00:07

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#16 2015-04-05 06:26:41

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#17 2015-04-06 14:58:57

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

#18 2015-04-06 15:21:41

Re: about UNICODE of DataSetToJSON in SynVirtualDataSet

Board footer