Conversion problem with FromVarString

EMartin · 2015-06-25 15:08:43

@ab, I have problem when converting un JSON array to dataset, the column with accented characters are truncated. Debugging I found a line in SynDB.pas.TSQLDBProxyStatementAbstract.IntHeaderProcess

    with PSQLDBColumnProperty(fColumn.AddAndMakeUniqueName(FromVarString(Data)))^ do begin
    ...
    end;

calling FromVarString in SynCommons.pas:

function FromVarString(var Source: PByte): RawUTF8;
var Len: PtrUInt;
begin
  Len := FromVarUInt32(Source); // --> THIS RETURN BAD LEN, ONLY ASCII VALUE OF THE FIRST CHAR
  SetString(Result,PAnsiChar(Source),Len);
  inc(Source,Len);
end;

With no accented characters the first character is eaten and the rest of text is OK, but when there is an accented character the text is truncated ugly.

Steps to reproduce (simulating SynDB use):

procedure TForm1.btnFromVarStringClick(Sender: TObject);
var
  lTmp: RawByteString;
  lTmp1: PByte;
  lTmp2: RawUTF8;
begin
  lTmp := '"simple text, text with no accent, other simple text"';
  lTmp1 := Pointer(lTmp);
  lTmp2 := FromVarString(lTmp1);
  ShowMessage(UTF8ToString(lTmp2));
  lTmp := '"simple text, text with accént, other simple text"';
  lTmp1 := Pointer(lTmp);
  lTmp2 := FromVarString(lTmp1);
  ShowMessage(UTF8ToString(lTmp2));
end;

The sequence begin in TSynBinaryDataset.From when TSQLDBProxyStatementRandomAccess is created.

Can you see this problem ?

Thanks in advance.

EMartin

EMartin · 2015-06-25 15:23:24

Development environment:

Delphi 7 Professional
Windows 7 32 bits running in VirtualBox
Firebird 2.5.3

The data is stored correctly in the database using my TSynRestDataset (TClientDataset not using SynDB connections also RESTful).

EMartin

ab · 2015-06-25 19:17:54

??
FromVarString() takes not a string, but a memory buffer previously encoded via ToVarString(), or - in the case of SynDB, via TFileBufferWriter.Write() method.

I've added an explicit regression test about FromVarString/ToVarString with accentued characters, and it works as expected.
See http://synopse.info/fossil/info/e15bf94af7

So your code won't demonstrate anything, since you use ToVarString() in a wrong way, I guess.

EMartin · 2015-06-25 21:06:51

I created a table in Firebird:

CREATE TABLE NEW_TABLE (
    TEXT  VARCHAR(50)
);

and I inserted a value:

INSERT INTO NEW_TABLE (TEXT) VALUES ('simple text, with accént, other simple text');

and a TForm with TDBGrid and a TDataSource I see the truncated text:

procedure TForm1.FormCreate(Sender: TObject);
begin
  fTestSynDBDS := TSynDBDataset.Create(Self);
  fTestSynDBDS.Name := 'rdsTest';                  
  ...
end;

// and executing the query:

procedure TForm1.btnLocalClick(Sender: TObject);
var
  lConnProp: TSQLDBConnectionProperties;
begin
  lConnProp := TSQLDBConnectionProperties.CreateFromJSON('{"DatabaseName":"<DBNAME>","DBMS":7,"Kind":"TSQLDBZEOSConnectionProperties","ServerName":"firebird-2.5://LocalHost","User":"<USERNAME>","Password":"<PASSWORD>"}',);
  try
    fTestSynDBDS.Connection := lConnProp;
    fTestSynDBDS.CommandText := 'select * from NEW_TABLE';
    fTestSynDBDS.Open;
    ds1.Dataset := fTestSynDBDS; // ds1 is a TDataSource linked to TDBGrid
  finally
    lConnProp.Free;
  end;
end;

the wrong text is displayed: "simple text, with acc".

I hope I have been clearer.

Thanks.

EMartin.

ab · 2015-06-26 06:36:11

I just tried with sample 17: no problem with the accents using TSynDBDataset on a SQLite3 table.

So I guess the problem is not in TSynDBDataset, but how your data is stored (codepage, and so on) with your Firebird database.

EMartin · 2015-06-26 10:49:24

I tested defining the field with code page UTF8 and ISO8859_1 and same results, I'll keep investigating. From the debugging up to the line:

with PSQLDBColumnProperty(fColumn.AddAndMakeUniqueName(FromVarString(Data)))^ do begin
    ...
end;

in SynDB.pas.TSQLDBProxyStatementAbstract.IntHeaderProcess the "Data" variable is displayed correctly and after the function VarFromString the text is truncated. But may be the code page is the problem.

Thanks.

EMartin

EMartin · 2015-06-26 12:02:04

And yes, there is a problem with the code page. For reproduce the problem with Sample 17, I opened the people.json file with PSPad and I changed the format from UTF8 to ANSI and when I run Sample 17 all names with no ASCII characters are truncated.

Can you confirm this ?

Thanks.

EMartin

ab · 2015-06-26 12:16:00

All internal process should be in UTF-8.

If you change the JSON content not to be in UTF-8, text would be truncated, for sure.
This is as designed: if you give incorrect and aberrant input, the framework won't fix it magically for you!

From the SynDB point of view, all text content is expected to be UTF-8.
I guess this is not a problem at FromVarString() level, but BEFORE it.
Perhaps the field names are not returned as UTF-8 by SynDBZeos.

EMartin · 2015-06-26 12:44:30

OK, but using RESTful the returned is OK as well as in the line PSQLDBColumnProperty(fColumn.AddAndMakeUniqueName(FromVarString(Data)))^, the variable Data contain all OK.

I'll keep investigate how connect to Firebird to return the data as UTF8.

Thanks.

EMartin.

EMartin · 2015-06-26 15:14:54

For information only, I solved the problem changing the default ZEOS codepage:

procedure TForm1.btnLocalClick(Sender: TObject);
var
  lConnProp: TSQLDBConnectionProperties;
begin
  lConnProp := TSQLDBConnectionProperties.CreateFromJSON('{"DatabaseName":"<DBNAME>","DBMS":7,"Kind":"TSQLDBZEOSConnectionProperties","ServerName":"firebird-2.5://LocalHost","User":"<USERNAME>","Password":"<PASSWORD>"}',);
  try
    TSQLDBZEOSConnectionProperties(lConnProp).ZeosURL.Properties.Values['codepage'] := 'ISO8859_1'; // replace default UTF8
    lConnProp.ConnectionTimeOutMinutes := 10; // set a connection timeout because otherwise IsOutDated return false
    lConnProp.ClearConnectionPool; // remove the connection used for internal statements in SynDB and contain the old code page.
    fTestSynDBDS.Connection := lConnProp;
    fTestSynDBDS.CommandText := 'select * from NEW_TABLE';
    fTestSynDBDS.Open;
    ds1.Dataset := fTestSynDBDS; // ds1 is a TDataSource linked to TDBGrid
  finally
    lConnProp.Free;
  end;
end;

EMartin.

mORMot Open Source

#1 2015-06-25 15:08:43

Conversion problem with FromVarString

#2 2015-06-25 15:23:24

Re: Conversion problem with FromVarString

#3 2015-06-25 19:17:54

Re: Conversion problem with FromVarString

#4 2015-06-25 21:06:51

Re: Conversion problem with FromVarString

#5 2015-06-26 06:36:11

Re: Conversion problem with FromVarString

#6 2015-06-26 10:49:24

Re: Conversion problem with FromVarString

#7 2015-06-26 12:02:04

Re: Conversion problem with FromVarString

#8 2015-06-26 12:16:00

Re: Conversion problem with FromVarString

#9 2015-06-26 12:44:30

Re: Conversion problem with FromVarString

#10 2015-06-26 15:14:54

Re: Conversion problem with FromVarString

Board footer