#1 2016-01-13 08:23:33

hnb
Member
Registered: 2015-06-15
Posts: 290

mORMot and ZEOS, RawUTF8 double encoded

All ftUTF8 (RawUTF8) parameters inside TSQLDBZEOSStatement.ExecutePrepared (SynDBZEOS) under FPC for mORMot and ZEOS are double encoded.

That means that the string for example:

'Ryż pełnoziarn. 300g'

in Delphi is encoded and executed as 'Ry'#197#188' pe'#197#130'noziarn. 300g'
but in FPC is encoded and executed as 'Ry'#196#185#196#189' pe'#196#185#226#128#154'noziarn. 300g' (!!!)

that is because VData (for example in fStatement.SetUTF8String(i+FirstDbcIndex,VData) in SynDBZEOS) has codepage 0 (instead of CP_NONE or CP_UTF8).

I have bad feeling about this :\ - bad conversion can exist in many places of framework sad

Patch for this example attached...
https://drive.google.com/file/d/0B4PZhd … sp=sharing

maybe using/declaring RawByteString = type AnsiString in SynCommon for FPC is bad idea. FPC has own RawByteString = type AnsiString(CP_NONE).

best regards,
Maciej Izak

Last edited by hnb (2016-01-13 08:30:45)


best regards,
Maciej Izak

Offline

#2 2016-01-13 08:53:03

EgonHugeist
Member
From: Germany
Registered: 2013-02-15
Posts: 190

Re: mORMot and ZEOS, RawUTF8 double encoded

A good point!

Since FPC 2.7 the official codepage agnostic has been introduced.

Looking @ SynCommons all (RawUnicode, RawUTF8, WinAnsiString, RawByteString) are wrongly declared as AnsiString for all non-Unicode targets inlcuding FPC2.7+

Instead of Setting the codepages Arnaud should check FPC-Version or add a global property based define like in Synopse.inc

Offline

#3 2016-01-13 09:12:49

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded

Small tip for Arnaud:

You can use FPC_FULLVERSION to check more preciously FPC version. For example:

{$if FPC_FULLVERSION < 20701} // FPC version lower than 2.7.1

best regards,
Maciej Izak

Offline

#4 2016-01-13 14:51:56

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,686
Website

Re: mORMot and ZEOS, RawUTF8 double encoded

http://wiki.freepascal.org/FPC_Unicode_support is the reference point.

I'm working on it.

I would add a HASCODEPAGE conditional identifier, which would support UnicodeString and AnsiString(##) syntax introduced by FPC 2.7+
The UNICODE conditional would be specific for the case where string=UnicodeString, i.e. under Delphi 2009+ and FPC 3 in {$mode delphiunicode}

Offline

#5 2016-01-13 17:36:19

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded

No need for HASCODEPAGE, there is define: FPC_HAS_CPSTRING and additionality for FPC exist FPC_HAS_UNICODESTRING.


best regards,
Maciej Izak

Offline

#6 2016-01-13 17:37:36

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,686
Website

Re: mORMot and ZEOS, RawUTF8 double encoded

HASCODEPAGE is mORMot-specific, and would be defined for both FPC 2.7+ and Delphi 2009+.

Offline

#7 2016-01-13 17:43:06

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded

Right, anyway you can use FPC_HAS_CPSTRING to define HASCODEPAGE  instead of using FPC_FULLVERSION. That was my hidden message/intention in previous message wink


best regards,
Maciej Izak

Offline

#8 2016-01-13 18:15:02

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,686
Website

Re: mORMot and ZEOS, RawUTF8 double encoded

This is what I just did in http://synopse.info/fossil/info/a4999ede0d

Thanks for the feedback!

Debugging is not fully finished with FPC yet... any input is welcome!

Offline

#9 2016-01-13 19:43:18

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded

Thanks for modifications, but mentioned in first message problem still occurs... sad


best regards,
Maciej Izak

Offline

#10 2016-01-13 20:33:38

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded

My bad. On the server engine side all was fine, only my "test case" fails, because lack of {$CODEPAGE UTF8} for string:

fooRec.fooField := 'Ry'#197#188' pe'#197#130'noziarn. 300g';

for now mORMot really works for FPC smile!

Note to myself -> remember about {$CODEPAGE UTF8} for strings in code like 'Ry'#197#188' pe'#197#130'noziarn. 300g'

Last edited by hnb (2016-01-13 20:49:43)


best regards,
Maciej Izak

Offline

#11 2016-01-13 20:37:57

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,686
Website

Re: mORMot and ZEOS, RawUTF8 double encoded

{$CODEPAGE UTF8} is made in SynCommons.inc for everything source code of the framework...

Offline

#12 2016-01-14 09:47:41

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded

Small bug "SynCommons.pas(23432,10) Error: overloaded functions have the same parameter list" for ARM Android target.

function "Trim(const S: RawUTF8): RawUTF8;" is implemented twice.


best regards,
Maciej Izak

Offline

#13 2016-01-14 19:52:59

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,686
Website

Re: mORMot and ZEOS, RawUTF8 double encoded

Offline

#14 2016-01-14 20:27:33

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded

Yup, all is fine. Thanks!


best regards,
Maciej Izak

Offline

#15 2016-04-04 10:53:49

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded

I see another potential lack in InitSynCommonsConversionTables in SynCommons.pas. Maybe is worth to set DefaultSystemCodepage to CP_UTF8 for HASCODEPAGE?


best regards,
Maciej Izak

Offline

#16 2016-04-04 11:42:55

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,686
Website

Re: mORMot and ZEOS, RawUTF8 double encoded

On Windows, I'm almost sure all this DefaultSystemCodepage overriding won't work, since the API code page of Windows is fixed by the system - see e.g. GetACP - and UTF-8 is not a valid code page for Windows!
If we override DefaultSystemCodepage, we have to ensure that the Windows API "Wide" (i.e. all *W functions, not *A functions) are called...

Offline

#17 2016-04-04 12:12:06

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded

Calling *W version is recommended for FPC 3.0 (described as tips for Lazarus but usefully for whole FPC):

http://wiki.freepascal.org/Better_Unico … indows_API

maybe this can help us:

http://wiki.freepascal.org/Lazarus_with … UTF-8_mode


best regards,
Maciej Izak

Offline

#18 2016-04-04 12:16:57

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded


best regards,
Maciej Izak

Offline

#19 2016-05-03 08:25:00

hnb
Member
Registered: 2015-06-15
Posts: 290

Re: mORMot and ZEOS, RawUTF8 double encoded

New important bug for RawByteString is fixed:

http://bugs.freepascal.org/view.php?id=30082

fixed in FPC trunk r33597


best regards,
Maciej Izak

Offline

#20 2016-05-03 08:37:08

AOG
Member
Registered: 2014-02-24
Posts: 490

Re: mORMot and ZEOS, RawUTF8 double encoded

Yep !
Because of this bug, I asked for an update of the RTTI branch.
Unfortunately, Steve updated to 33592. So I asked again, for at least 33597 !

Offline

#21 2016-05-03 10:13:16

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,686
Website

Re: mORMot and ZEOS, RawUTF8 double encoded

Oups.... this is indeed a monstrous issue..

Offline

#22 2016-05-03 14:22:34

AOG
Member
Registered: 2014-02-24
Posts: 490

Re: mORMot and ZEOS, RawUTF8 double encoded

Steve has updated the interfacertti branch to trunk (rev 33634) !!
Now this change can also be accepeted for this branch:

property SizeIn: {$ifdef FPC}qword{$else}cardinal{$endif} read FStrm.total_in;
property SizeOut: {$ifdef FPC}qword{$else}cardinal{$endif} read FStrm.total_out;

And on some other places, RawByteString can now be used, the same as valid for Delphi.

Offline

#23 2016-05-03 14:38:17

AOG
Member
Registered: 2014-02-24
Posts: 490

Re: mORMot and ZEOS, RawUTF8 double encoded

At the moment, the new interfacertti branch does not compile on my system (Win8.1).
I would welcome reports !

Offline

#24 2016-05-03 18:23:43

AOG
Member
Registered: 2014-02-24
Posts: 490

Re: mORMot and ZEOS, RawUTF8 double encoded

The new interfacertti branch is ok now !
Again a big step forwards to enjoy the full mORMot power with FPC !!

Offline

#25 2016-05-08 19:05:40

shobits1
Member
Registered: 2016-02-10
Posts: 15

Re: mORMot and ZEOS, RawUTF8 double encoded

AOG: I can't compile the FPC with interfacertti branch; I always get this error

Start compiling package rtl-objpas for target i386-win32.
       Compiling rtl-objpas\BuildUnit_rtl_objpas.pp
       Compiling .\rtl-objpas\src\inc\strutils.pp
       Compiling .\rtl-objpas\src\inc\widestrutils.pp
       Compiling .\rtl-objpas\src\win\varutils.pp
       Compiling .\rtl-objpas\src\inc\convutils.pp
       Compiling .\rtl-objpas\src\inc\dateutils.pp
       Compiling .\rtl-objpas\src\inc\stdconvs.pp
       Compiling .\rtl-objpas\src\inc\variants.pp
The installer encountered the following error:
Compilation of "BuildUnit_rtl_objpas.pp" failed
make[2]: *** [smart] Error 1
make[2]: Leaving directory `C:/development/lazarus_RTTI.BRANCH/fpc/3.1.1/packages'
make[1]: *** [packages_smart] Error 2
make[1]: Leaving directory `C:/development/lazarus_RTTI.BRANCH/fpc/3.1.1'
make: *** [build-stamp.i386-win32] Error 2
make: Leaving directory `C:/development/lazarus_RTTI.BRANCH/fpc/3.1.1'

fpclazup: info: FPC: Running fpc make all install failed with exit code 2
Details:
fpclazup: ERROR: Error running BuildModuleCustom for module FPC
fpclazup: info: Error running fpcup. Technical details: error executing sequence fpc; line: 5, param: FPC
fpclazup: info: Error running fpcup. Technical details: error executing sequence Default; line: 3, param: fpc
Fpclazup failed.

Can you confirm, please.

PS: I'm using fpclazup.

Offline

#26 2016-05-09 07:14:39

AOG
Member
Registered: 2014-02-24
Posts: 490

Re: mORMot and ZEOS, RawUTF8 double encoded

@shobits1

Can confirm !

For now, just use something like

fpclazup --installdir="C:/yourdirectory" --fpcURL="trunk" --lazURL="trunk" --verbose  --getfullrepo --fpcPATCH="fpctrunkrtti.patch"

The trunk rtti patch can also be found on the fpclazup github.
This will give you trunk + RTTI needed by mORMot.

Offline

#27 2016-05-09 19:10:56

shobits1
Member
Registered: 2016-02-10
Posts: 15

Re: mORMot and ZEOS, RawUTF8 double encoded

thanks, I use similar setup - fpc trunk + rtti patch + lazarus trunk - and it works fine; but I was asking about the interfacertti branch.

Can confirm !

I meant to say: can you confirm that interfacertti branch compiles or not (since you reported it's not compiling and then it's fixed).

Offline

Board footer

Powered by FluxBB