Different codepage than UTF8 for SynDBZeos

hnb · 2015-10-13 13:49:13

Hi,

can we add to SynDBZeos URI "codepage" property? I need to deal with many old/big databases encoded in LATIN2 (without possibility to converting to UTF8). AFAIK, there is "controls_cp=UTF8" that will convert all received data to UTF8.

ab · 2015-10-13 15:53:13

Zeos should handle the ISO/IEC 8859-2 -> UTF-8 conversion on its side, AFAIK.

hnb · 2015-10-13 16:26:22

ab wrote:

Zeos should handle the ISO/IEC 8859-2 -> UTF-8 conversion on its side, AFAIK.

to setup conversion i need to be able to set "codepage" to "LATIN2" instead of "UTF8", but now "codepage=UTF8" is hard-coded. I am working with many already existing databases (PostgreSQL). I have few scenarios:

1. strings are stored as UTF8 and in Zeos "codepage=LATIN2" (I know it is ugly :\ but it was created because:
A. there was bug, inside previous Zeos versions for preparing SQL query with national chars.
B. costumers requirements.
2. strings are stored as LATIN2 and in Zeos "codepage=LATIN2"
3. strings are stored as UTF8 and in Zeos "codepage=UTF8"

to deal with some nonstandard databases I need to be able to set inside SynDBZeos "codepage" to "LATIN2". Only when I am able to set "codepage" property, I can use mORMot with all "historical" dependencies.

ab · 2015-10-13 17:07:51

I guess there is a dedicated setting.

Please ask on ZDBC forum...

danielkuettner · 2015-10-13 17:49:20

You can derive your own class and set the options for Zeos as needed:

TMySQLDBZEOSConnectionProperties = class(TSQLDBZEOSConnectionProperties)
  protected
    //fStatementParams: TStrings;
  public
    constructor Create(const aServerName, aDatabaseName, aUserID, aPassWord: RawUTF8); override;
    destructor Destroy; override;
  end;

...

constructor TMySQLDBZEOSConnectionProperties.Create(const aServerName,
  aDatabaseName, aUserID, aPassWord: RawUTF8);
begin
  inherited;
  fStatementParams.Add('cachedlob=false');
  fStatementParams.Add('isc_tpb_read_committed');
  fUseCache:= false;
  ConnectionTimeOutMinutes:= 100;
  fURL.Properties.Values['codepage']:= 'ISO8859-13';
end;

hnb · 2015-10-13 19:46:17

@danielkuettner - thanks for hint. That can be also fixed by my small patch.

@ab - Maybe my previous message is unclear

You are right, there is a dedicated setting but I can't set this setting at current version of mORMot. I need to be able set "codepage".

Patch is attached:

https://drive.google.com/file/d/0B4PZhd … sp=sharing

I hope that this patch will be accepted

Last edited by hnb (2015-10-13 19:49:32)

hnb · 2015-10-14 07:07:06

Something stops working after my patch on client side. I need to do more research.

EgonHugeist · 2015-10-14 08:39:42

@hnb

From my POV there is no reason to change the CCS. Libpg.dll perfektly converts Latin2 to UTF8 and vice versa.

Note: We did some performance changes on the String getters: since mORMot is a UTF8 based framework we (Arnaud & Me ) did decide to change the IZResultSet-Getters to GetPAnsiChar getters for some RDBM like PG, Oracle, SQLite, MySQL.
So what you're trying to do may leads to pain on composing the JSON contents.
You should also change the Getter to GetUTF8String() for the PG-Protocol which than kills the performance(First alloc mem for a String, move Data from libpg to String, move content to JSONWriter, free mem) which is skipped by ZDBC using the Pointer-Getters...

My advice: Clean bad encoded strings once in your DB and keep the mORMot/ZDBC code as is..

Cheers, Michael
ZeosDevTeam

hnb · 2015-10-14 18:33:37

@EgonHugeist

Thanks for additional info. Cleaning bad encoded strings is impossible in few scenarios (n-tier DB replication, more than 500 tables + a lot of records, big databases used non stop). But for few clients I can do this .

mORMot Open Source

#1 2015-10-13 13:49:13

Different codepage than UTF8 for SynDBZeos

#2 2015-10-13 15:53:13

Re: Different codepage than UTF8 for SynDBZeos

#3 2015-10-13 16:26:22

Re: Different codepage than UTF8 for SynDBZeos

#4 2015-10-13 17:07:51

Re: Different codepage than UTF8 for SynDBZeos

#5 2015-10-13 17:49:20

Re: Different codepage than UTF8 for SynDBZeos

#6 2015-10-13 19:46:17

Re: Different codepage than UTF8 for SynDBZeos

#7 2015-10-14 07:07:06

Re: Different codepage than UTF8 for SynDBZeos

#8 2015-10-14 08:39:42

Re: Different codepage than UTF8 for SynDBZeos

#9 2015-10-14 18:33:37

Re: Different codepage than UTF8 for SynDBZeos

Board footer