mORMot and ZEOS, RawUTF8 double encoded

hnb · 2016-01-13 08:23:33

All ftUTF8 (RawUTF8) parameters inside TSQLDBZEOSStatement.ExecutePrepared (SynDBZEOS) under FPC for mORMot and ZEOS are double encoded.

That means that the string for example:

'Ryż pełnoziarn. 300g'

in Delphi is encoded and executed as 'Ry'#197#188' pe'#197#130'noziarn. 300g'
but in FPC is encoded and executed as 'Ry'#196#185#196#189' pe'#196#185#226#128#154'noziarn. 300g' (!!!)

that is because VData (for example in fStatement.SetUTF8String(i+FirstDbcIndex,VData) in SynDBZEOS) has codepage 0 (instead of CP_NONE or CP_UTF8).

I have bad feeling about this :\ - bad conversion can exist in many places of framework

Patch for this example attached...
https://drive.google.com/file/d/0B4PZhd … sp=sharing

maybe using/declaring RawByteString = type AnsiString in SynCommon for FPC is bad idea. FPC has own RawByteString = type AnsiString(CP_NONE).

best regards,
Maciej Izak

Last edited by hnb (2016-01-13 08:30:45)

EgonHugeist · 2016-01-13 08:53:03

A good point!

Since FPC 2.7 the official codepage agnostic has been introduced.

Looking @ SynCommons all (RawUnicode, RawUTF8, WinAnsiString, RawByteString) are wrongly declared as AnsiString for all non-Unicode targets inlcuding FPC2.7+

Instead of Setting the codepages Arnaud should check FPC-Version or add a global property based define like in Synopse.inc

hnb · 2016-01-13 09:12:49

Small tip for Arnaud:

You can use FPC_FULLVERSION to check more preciously FPC version. For example:

{$if FPC_FULLVERSION < 20701} // FPC version lower than 2.7.1

ab · 2016-01-13 14:51:56

http://wiki.freepascal.org/FPC_Unicode_support is the reference point.

I'm working on it.

I would add a HASCODEPAGE conditional identifier, which would support UnicodeString and AnsiString(##) syntax introduced by FPC 2.7+
The UNICODE conditional would be specific for the case where string=UnicodeString, i.e. under Delphi 2009+ and FPC 3 in {$mode delphiunicode}

hnb · 2016-01-13 17:36:19

No need for HASCODEPAGE, there is define: FPC_HAS_CPSTRING and additionality for FPC exist FPC_HAS_UNICODESTRING.

ab · 2016-01-13 17:37:36

HASCODEPAGE is mORMot-specific, and would be defined for both FPC 2.7+ and Delphi 2009+.

hnb · 2016-01-13 17:43:06

Right, anyway you can use FPC_HAS_CPSTRING to define HASCODEPAGE instead of using FPC_FULLVERSION. That was my hidden message/intention in previous message

ab · 2016-01-13 18:15:02

This is what I just did in http://synopse.info/fossil/info/a4999ede0d

Thanks for the feedback!

Debugging is not fully finished with FPC yet... any input is welcome!

hnb · 2016-01-13 19:43:18

Thanks for modifications, but mentioned in first message problem still occurs...

hnb · 2016-01-13 20:33:38

My bad. On the server engine side all was fine, only my "test case" fails, because lack of {$CODEPAGE UTF8} for string:

fooRec.fooField := 'Ry'#197#188' pe'#197#130'noziarn. 300g';

for now mORMot really works for FPC !

Note to myself -> remember about {$CODEPAGE UTF8} for strings in code like 'Ry'#197#188' pe'#197#130'noziarn. 300g'

Last edited by hnb (2016-01-13 20:49:43)

ab · 2016-01-13 20:37:57

{$CODEPAGE UTF8} is made in SynCommons.inc for everything source code of the framework...

hnb · 2016-01-14 09:47:41

Small bug "SynCommons.pas(23432,10) Error: overloaded functions have the same parameter list" for ARM Android target.

function "Trim(const S: RawUTF8): RawUTF8;" is implemented twice.

ab · 2016-01-14 19:52:59

Should be fixed by http://synopse.info/fossil/info/410d8344b1

hnb · 2016-01-14 20:27:33

Yup, all is fine. Thanks!

hnb · 2016-04-04 10:53:49

I see another potential lack in InitSynCommonsConversionTables in SynCommons.pas. Maybe is worth to set DefaultSystemCodepage to CP_UTF8 for HASCODEPAGE?

ab · 2016-04-04 11:42:55

On Windows, I'm almost sure all this DefaultSystemCodepage overriding won't work, since the API code page of Windows is fixed by the system - see e.g. GetACP - and UTF-8 is not a valid code page for Windows!
If we override DefaultSystemCodepage, we have to ensure that the Windows API "Wide" (i.e. all *W functions, not *A functions) are called...

hnb · 2016-04-04 12:12:06

Calling *W version is recommended for FPC 3.0 (described as tips for Lazarus but usefully for whole FPC):

http://wiki.freepascal.org/Better_Unico … indows_API

maybe this can help us:

http://wiki.freepascal.org/Lazarus_with … UTF-8_mode

hnb · 2016-04-04 12:16:57

I think is worth to track these FPC threads:

http://free-pascal-general.1045716.n5.n … 24746.html
http://free-pascal-general.1045716.n5.n … 24747.html

hnb · 2016-05-03 08:25:00

New important bug for RawByteString is fixed:

http://bugs.freepascal.org/view.php?id=30082

fixed in FPC trunk r33597

AOG · 2016-05-03 08:37:08

Yep !
Because of this bug, I asked for an update of the RTTI branch.
Unfortunately, Steve updated to 33592. So I asked again, for at least 33597 !

ab · 2016-05-03 10:13:16

Oups.... this is indeed a monstrous issue..

AOG · 2016-05-03 14:22:34

Steve has updated the interfacertti branch to trunk (rev 33634) !!
Now this change can also be accepeted for this branch:

property SizeIn: {$ifdef FPC}qword{$else}cardinal{$endif} read FStrm.total_in;
property SizeOut: {$ifdef FPC}qword{$else}cardinal{$endif} read FStrm.total_out;

And on some other places, RawByteString can now be used, the same as valid for Delphi.

AOG · 2016-05-03 14:38:17

At the moment, the new interfacertti branch does not compile on my system (Win8.1).
I would welcome reports !

AOG · 2016-05-03 18:23:43

The new interfacertti branch is ok now !
Again a big step forwards to enjoy the full mORMot power with FPC !!

shobits1 · 2016-05-08 19:05:40

AOG: I can't compile the FPC with interfacertti branch; I always get this error

Start compiling package rtl-objpas for target i386-win32.
       Compiling rtl-objpas\BuildUnit_rtl_objpas.pp
       Compiling .\rtl-objpas\src\inc\strutils.pp
       Compiling .\rtl-objpas\src\inc\widestrutils.pp
       Compiling .\rtl-objpas\src\win\varutils.pp
       Compiling .\rtl-objpas\src\inc\convutils.pp
       Compiling .\rtl-objpas\src\inc\dateutils.pp
       Compiling .\rtl-objpas\src\inc\stdconvs.pp
       Compiling .\rtl-objpas\src\inc\variants.pp
The installer encountered the following error:
Compilation of "BuildUnit_rtl_objpas.pp" failed
make[2]: *** [smart] Error 1
make[2]: Leaving directory `C:/development/lazarus_RTTI.BRANCH/fpc/3.1.1/packages'
make[1]: *** [packages_smart] Error 2
make[1]: Leaving directory `C:/development/lazarus_RTTI.BRANCH/fpc/3.1.1'
make: *** [build-stamp.i386-win32] Error 2
make: Leaving directory `C:/development/lazarus_RTTI.BRANCH/fpc/3.1.1'

fpclazup: info: FPC: Running fpc make all install failed with exit code 2
Details:
fpclazup: ERROR: Error running BuildModuleCustom for module FPC
fpclazup: info: Error running fpcup. Technical details: error executing sequence fpc; line: 5, param: FPC
fpclazup: info: Error running fpcup. Technical details: error executing sequence Default; line: 3, param: fpc
Fpclazup failed.

Can you confirm, please.

PS: I'm using fpclazup.

AOG · 2016-05-09 07:14:39

@shobits1

Can confirm !

For now, just use something like

fpclazup --installdir="C:/yourdirectory" --fpcURL="trunk" --lazURL="trunk" --verbose  --getfullrepo --fpcPATCH="fpctrunkrtti.patch"

The trunk rtti patch can also be found on the fpclazup github.
This will give you trunk + RTTI needed by mORMot.

shobits1 · 2016-05-09 19:10:56

thanks, I use similar setup - fpc trunk + rtti patch + lazarus trunk - and it works fine; but I was asking about the interfacertti branch.

Can confirm !

I meant to say: can you confirm that interfacertti branch compiles or not (since you reported it's not compiling and then it's fixed).

mORMot Open Source

#1 2016-01-13 08:23:33

mORMot and ZEOS, RawUTF8 double encoded

#2 2016-01-13 08:53:03

Re: mORMot and ZEOS, RawUTF8 double encoded

#3 2016-01-13 09:12:49

Re: mORMot and ZEOS, RawUTF8 double encoded

#4 2016-01-13 14:51:56

Re: mORMot and ZEOS, RawUTF8 double encoded

#5 2016-01-13 17:36:19

Re: mORMot and ZEOS, RawUTF8 double encoded

#6 2016-01-13 17:37:36

Re: mORMot and ZEOS, RawUTF8 double encoded

#7 2016-01-13 17:43:06

Re: mORMot and ZEOS, RawUTF8 double encoded

#8 2016-01-13 18:15:02

Re: mORMot and ZEOS, RawUTF8 double encoded

#9 2016-01-13 19:43:18

Re: mORMot and ZEOS, RawUTF8 double encoded

#10 2016-01-13 20:33:38

Re: mORMot and ZEOS, RawUTF8 double encoded

#11 2016-01-13 20:37:57

Re: mORMot and ZEOS, RawUTF8 double encoded

#12 2016-01-14 09:47:41

Re: mORMot and ZEOS, RawUTF8 double encoded

#13 2016-01-14 19:52:59

Re: mORMot and ZEOS, RawUTF8 double encoded

#14 2016-01-14 20:27:33

Re: mORMot and ZEOS, RawUTF8 double encoded

#15 2016-04-04 10:53:49

Re: mORMot and ZEOS, RawUTF8 double encoded

#16 2016-04-04 11:42:55

Re: mORMot and ZEOS, RawUTF8 double encoded

#17 2016-04-04 12:12:06

Re: mORMot and ZEOS, RawUTF8 double encoded

#18 2016-04-04 12:16:57

Re: mORMot and ZEOS, RawUTF8 double encoded

#19 2016-05-03 08:25:00

Re: mORMot and ZEOS, RawUTF8 double encoded

#20 2016-05-03 08:37:08

Re: mORMot and ZEOS, RawUTF8 double encoded

#21 2016-05-03 10:13:16

Re: mORMot and ZEOS, RawUTF8 double encoded

#22 2016-05-03 14:22:34

Re: mORMot and ZEOS, RawUTF8 double encoded

#23 2016-05-03 14:38:17

Re: mORMot and ZEOS, RawUTF8 double encoded

#24 2016-05-03 18:23:43

Re: mORMot and ZEOS, RawUTF8 double encoded

#25 2016-05-08 19:05:40

Re: mORMot and ZEOS, RawUTF8 double encoded

#26 2016-05-09 07:14:39

Re: mORMot and ZEOS, RawUTF8 double encoded

#27 2016-05-09 19:10:56

Re: mORMot and ZEOS, RawUTF8 double encoded

Board footer