mORMot Open Source

flo22

Member

From: austria

Registered: 2015-10-15

Posts: 5

variant contains unencoded utf8 characters if serialized as json, xe5

Hi,
I assign values to my variant like this:

TDocVariant.new(v);
v.add(stringToUtf8('name'),stringToUtf8('someGermanNameContainingÖÄÜ'));

then I output:
Utf8ToString(VariantSaveJson(v)); //contains unencoded characters

but If I output:
Utf8ToString(v.name); //all characters correct

Furthermore in my TSQLRecord - Class I have some calculated RawUtf8 fields for better searching:

type TSQLRecordCustomer=Class(TSQLRecord)
  fVariantCustomer: Variant;
  fName: RawUtf8;
...
published
  property VariantCustomer: Variant read fVariantCustomer write fVariantCustomer;
  property Name: RawUtf8 read fName;
... 
procedure TSQLRecordCustomer.ComputeFieldsBeforeWrite(aRest: TSQLRest; aOccasion: TSQLEvent);
...
    if fVariantCustomer.Exists('Name') then
        fName := fVariantCustomer.Name;
...      
      
Now if I browse my sqlite db the utf8 encoding in all of my calculated fields is correct,
but in the VariantCustomer column the json contains unencoded characters.

Additionally I experienced that if use transtyping as TDocVariantData like this:

TDocVariantData(v).AddValue(StringToUTF8('name'), StringToUTF8('someGermanNameContainingÖÄÜ'));

in both cases:  Utf8ToString(VariantSaveJson(v)) and Utf8ToString(v.name) the encoding is incorrect.

Am I missing something essential here? Thanks' in advance.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,260

Website

Re: variant contains unencoded utf8 characters if serialized as json, xe5

Which version of Delphi are you using?

StringToUtf8('someGermanNameContainingÖÄÜ') is assigned to a variant parameter, so there is an hidden conversion here, which sounds to be wrong.

Similarly, fName := fVariantCustomer.Name does make a conversion.

Try to you use either:
- plain 'someGermanNameContainingÖÄÜ' (under Unicode version of Delphi) to store a variant UnicodeString
- RawUTF8ToVariant(stringToUtf8('someGermanNameContainingÖÄÜ'))

flo22

Member

From: austria

Registered: 2015-10-15

Posts: 5

Re: variant contains unencoded utf8 characters if serialized as json, xe5

Thank you for the prompt response,

I tried to use
TDocVariantData(v).AddValue('name', 'someGermanNameContainingÖÄÜ'); without the stringToUtf8();

and

TDocVariantData(v).AddItem(RawUTF8ToVariant(stringToUtf8('someGermanNameContainingÖÄÜ')));

v.name returns a correctly encoded string now
but the problem with VariantSaveJson(v) remains and my VariantCustomer column does still contain unencoded chars.

I use delphi XE5

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,260

Website

Re: variant contains unencoded utf8 characters if serialized as json, xe5

I'm not able to reproduce the issue here.

flo22

Member

From: austria

Registered: 2015-10-15

Posts: 5

Re: variant contains unencoded utf8 characters if serialized as json, xe5

Your input, that fName := fVariantCustomer.Name is actually a conversion pointed me towards the solution.

I incorrectly read the utf8-text from a .csv and assumed that my retrieved utf8-encoding was correct
because it showed correctly on dislplay-output and in my RawUtf8-db-fields which I set using fName := fVariantCustomer.Name.
This conversion must have fixed my encoding somehow and therefore it did display correctly.
So, in my case the bug was actually sitting in between the chair and the keyboard

Thank you very much for your help!