#1 2021-04-14 18:28:57

marcinb
Member
Registered: 2014-04-08
Posts: 11

Non ASCII characters in TSQLDBStatement.RowDocVariant

Hello

I have legacy firebird database  with code page WIN1250. Still on mORMot 1.18.
What should I do to get whole string ('usługa' instead of 'us') when getting column values via RowDocVariant method. Should I set some additional Zeos connection properties?
I've checked that TSQLDBStatement.FetchAllAsJSON method returns "unmodified" string...

Thanks in advance
Marcin

Last edited by marcinb (2021-04-14 18:39:40)

Offline

#2 2021-04-14 21:05:09

Vitaly
Member
From: UAE
Registered: 2017-01-31
Posts: 168
Website

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

I assume there shouldn't be any problem in reading from WIN1250 FB DB. TDocVariant from RowDocVariant should contain a full RawUTF8 string. Could it be that you got the loss afterward? maybe due to direct string assigning

Offline

#3 2021-04-15 08:02:02

marcinb
Member
Registered: 2014-04-08
Posts: 11

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

Hi

I did some digging (debugging) and found guilty routine: it's UTF8ToWideChar from ZEncode unit called by UTF8ToSynUnicode from SynCommons.
I'm on Delphi 10.4, but finally I'd like to compile this project in FPC on linux.

TIA
Marcin

Offline

#4 2021-04-15 09:42:11

Vitaly
Member
From: UAE
Registered: 2017-01-31
Posts: 168
Website

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

I'm also using 10.4.2. I specifically made a test with the direct select statement execution and RowDocVariant through Zeos (WIN1250 DB).
In my case UTF8ToSynUnicode from SynCommons calls UTF8ToWideChar from SynCommons. And it seems working ok... - checked the resulting variant value from the debugger.
I used FB 2.5.9, although I doubt that it matters. Btw, I checked both, with USE_SYNCOMMONS and without it. Probably I misunderstand something, sorry.

Offline

#5 2021-04-15 13:45:47

marcinb
Member
Registered: 2014-04-08
Posts: 11

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

Can someone help me here, please

For unknow reason my project uses UTF8ToWideChar from ZEncoding instead of this one from SynCommons. It might be some settings I'm not aware of

TIA
Marcin

Last edited by marcinb (2021-04-15 13:59:29)

Offline

#6 2021-04-15 16:37:02

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,546
Website

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

DId you try to disable or enable USE_SYNCOMMONS ?

Did you try a full build?

Did you try to update to the latest mORMot and Zeos version?

Offline

#7 2021-04-15 18:32:48

marcinb
Member
Registered: 2014-04-08
Posts: 11

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

Hello Arnaud
Latest mORMot (downloaded today) and Zeos 7.2.10 with latest patches (of 8th of April)
function UTF8ToWideChar from SynCommons returns 'us' string instead of 'usługa' (note the 3rd character in "usługa")

Any clues?

TIA
Marcin

Offline

#8 2021-04-16 07:15:00

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,546
Website

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

I guess the database charset is not the one setup for the connection.

Offline

#9 2021-04-16 14:44:33

Vitaly
Member
From: UAE
Registered: 2017-01-31
Posts: 168
Website

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

marcinb wrote:

Zeos 7.2.10 with latest patches (of 8th of April)

I'm using the trunk Zeos version. You could try it, probably not all fixes go to patches for a stable Zeos.

https://sourceforge.net/p/zeoslib/code- … ree/trunk/ or https://github.com/frones/ZeosLib

Offline

#10 2021-04-16 14:58:17

marcinb
Member
Registered: 2014-04-08
Posts: 11

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

I have checked same with UTF8 database and it is ok.
So following Arnaud's 'guessing' I'll try to set proper charset for db connection and see where I'll end up.

Offline

#11 2021-04-20 03:55:25

EgonHugeist
Member
From: Germany
Registered: 2013-02-15
Posts: 190

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

@marcinb

what's the database characterset? If it's characterset "none"? If so you'll propably need an extra advice.
Michael

Offline

#12 2021-04-20 07:05:50

marcinb
Member
Registered: 2014-04-08
Posts: 11

Re: Non ASCII characters in TSQLDBStatement.RowDocVariant

Hello

Actually two things did the trick
- update of mORMot and Zeos
- explicilty specifying required characterset for the connection.

I have to perform more tests, but so far it looks good

Thanks for all suggestions
Marcin

Offline

Board footer

Powered by FluxBB