Synopse SQlite3 Unit Tests

Leander007 · 2011-06-07 12:20:33

TestSQL3 tests
sources=2011-06-07 [fa3685333d], compiler=Delphi 2009, CodePage=1250

1.6. Synopse PDF:
! - TPdfDocument: 2 / 4 FAILED
! - TPdfDocumentGDI: 1 / 3 FAILED
Total failed: 3 / 7 - Synopse PDF FAILED

2.2. File based:
! - Direct access: 2 / 10,132 FAILED
! - TSQLRestClientDB: 2,002 / 589,109 FAILED
Total failed: 2,004 / 618,271 - File based FAILED

2.3. File based WAL:
! - Direct access: 2 / 10,132 FAILED
! - TSQLRestClientDB: 2,002 / 589,109 FAILED
Total failed: 2,004 / 618,271 - File based WAL FAILED

2.4. Memory based:
! - Direct access: 1 / 10,131 FAILED
! - TSQLRestClientDB: 2,002 / 615,382 FAILED
Total failed: 2,003 / 644,543 - Memory based FAILED

Synopse framework used: 1.13
SQlite3 engine used: 3.7.6.3
Generated with: Delphi 2009 compiler

-----
If we look one failing test per time:
It seems that in PDF test the check about setting different fonts should always fail,
because we never set Helvetica font (always forcing 'arial'?) and it looks like the code:

Canvas.SetFont('arial',10,[]);

should be changed to:

Canvas.SetFont(Name[embed],10,[]);

Nevertheless it fails eather way, because font is never realy set to Helvetica in function TPdfCanvas.SetFont.

Unit SynSelfTest:
625:

procedure TTestSynopsePDF._TPdfDocument 
Name: array[boolean] of PDFString =
    ('Arial','Helvetica');
...
Canvas.SetFont('arial',10,[]); 
Check(Canvas.Page.Font.Name=Name[embed]);
...
Check(Hash32(MS.Memory,MS.Position)=Hash[embed]); //$1780F68C=$A1201D84

ab · 2011-06-07 12:51:38

This pdf test is working with Delphi 7, Delphi 2007 and Delphi 2010 compiler.

The Helvetica is selected automatically when SetFont('arial') is made in the following code:

function TPdfCanvas.SetFont(const AName: RawUTF8; ASize: single; AStyle: TFontStyles;
  ACharSet: integer=-1): TPdfFont;
  (...)
  if (fDoc.FCharSet=ANSI_CHARSET) and fDoc.StandardFontsReplace then begin
    // standard/embedded fonts are WinAnsi only
    for i := low(STAND_FONTS_PDF) to high(STAND_FONTS_PDF) do
      if SameTextU(AName,STAND_FONTS_PDF[i]) or
         SameTextU(AName,STAND_FONTS_WIN[i]) then
        SetEmbeddedFont(STAND_FONTS_PDF[i]);
    if result<>nil then
      exit; // we got a standard/embedded font
  end;

So Canvas.SetFont('arial',10,[]) is not an error, on the contrary, it's a test of the embedded font substitution feature of the unit.

So I guess there is an issue in the accentuated "e acute", when converted into your current code page, via a PDFString.
So I use a WideString here, and set the 'e acute' by hand:

WS := 'Texte accentue n°'+IntToString(i);
PWordArray(WS)^[13] := 233;
Canvas.TextOutW(100,y,pointer(WS));

See http://synopse.info/fossil/info/d5ddda8451

Leander007 · 2011-06-07 13:26:25

I was in doubts about this, if I got it right, because you said that it works in your case (code page, compiler).
I still don't fill this very natural , when reading the code e.g. Set(Something) and Expect(Something_Different).
I hope that you understand my confusion .

I tried with new corrections (d5ddda8451), but PDF tests still fails, because the problem exists in prior lines:

Canvas.SetFont('arial',10,[]); //on second run (embed=true) this should set to Helvetica if I understand correctly
Check(Canvas.Page.Font.Name=Name[embed]); //here we test for Helvetica on second run, but it's always Arial !

Last edited by Leander007 (2011-06-07 13:43:26)

ab · 2011-06-07 13:59:27

Could you step in SetFont method and see why it's not selected?

You should reach the SetEmbeddedFont() sub procedure on second run.

Leander007 · 2011-06-07 14:00:35

When compiling the new code I saw that you forgot to rename TPropInfo.Name to TPropInfo.ShortName in SQlite3i18n unit at lines 2850, 2890 to be compilable.

Leander007 · 2011-06-07 14:10:08

At this "entry" line of TPdfCanvas.SetFont function condition is not met to reach SetEmbeddedFont procedure:

if (fDoc.FCharSet=ANSI_CHARSET) and fDoc.StandardFontsReplace then begin...

fDoc.FCharSet=238
ANSI_CHARSET=0
fDoc.StandardFontsReplace=False

ab · 2011-06-07 16:33:55

Leander007 wrote:

When compiling the new code I saw that you forgot to rename TPropInfo.Name to TPropInfo.ShortName in SQlite3i18n unit at lines 2850, 2890 to be compilable.

The unit compiles fine by me.
And there is no TPropInfo.Name in lines 2850 and 2890.

I suspect you don't have a valid source code content.
Please refresh ALL files.

ab · 2011-06-07 16:34:54

Leander007 wrote:

fDoc.FCharSet=238

So by design, the test will always fail!
These standard fonts will only work with WinAnsiCharset.
Not a problem, just ignore embed=true loop.
See http://synopse.info/fossil/info/0425cea128

Leander007 · 2011-06-07 17:01:09

ab wrote:

Leander007 wrote:
When compiling the new code I saw that you forgot to rename TPropInfo.Name to TPropInfo.ShortName in SQlite3i18n unit at lines 2850, 2890 to be compilable.
The unit compiles fine by me.
And there is no TPropInfo.Name in lines 2850 and 2890.
I suspect you don't have a valid source code content.
Please refresh ALL files.

I was using mnemonic TPropInfo as type and Name as property, that could be more readable (in contrary to @P^.Name), for which type this has changed.
Whole 2850 line is:
AddOnceDynArray(WinAnsiString(TSQLRecord.CaptionName(@P^.Name)));

But type TPropInfo in SQLite3Commons has no Name property, it has ShortName instead and I checked (through the html file viewer) again your latest fossil sources 0425cea128 and the issue still exists.

Leander007 · 2011-06-07 17:21:27

ab wrote:

Leander007 wrote:
fDoc.FCharSet=238
So by design, the test will always fail!
These standard fonts will only work with WinAnsiCharset.
Not a problem, just ignore embed=true loop.
See http://synopse.info/fossil/info/0425cea128

Minor lapsus:
In 0425cea128 you added the code in "embed=false" test.

You meant something like this:

for embed := false to true do begin
      if embed and (CharSet<>ANSI_CHARSET) then
          continue; // StandardFontsReplace will work only with ANSI code page

ab · 2011-06-07 18:05:22

Moreover:

if CharSet<>ANSI_CHARSET then
  break;

See http://synopse.info/fossil/fdiff?v1=548 … 715598ad7b

Leander007 · 2011-06-07 18:29:22

Yes, using break is the shortest one .

With next PDF test I really don't have a clue what could be wrong and how to test it, because I don't know what to expect in intermediate steps.

Unit SynSelfTests

Check on line 792 is OK,
but the check on line 813 fails:

H := Hash32(MS.Memory,MS.Position); //MS.Position=6725, H=$93E059A5 
Check( (H=$93D859A5) or (H=$9FB359A7) ); // Arial,Bold and Tahoma FontBBox

Leander007 · 2011-06-07 20:15:35

All upper tests excluding this last post (TPdfDocumentGDI test) are code page related.
They fail if I change system locale code page to something else as 1252 (in my case 1250).

Only TPdfDocumentGDI test (last check) fails in all circumstances in which I run tests.

ab · 2011-06-08 08:03:29

What if you add those lines at the beginning of TestSQL3.dpr:

begin
  AllocConsole;
  if GetThreadLocale<>$0409 then begin
    SetThreadLocale($0409);
    GetFormatSettings;
  end;
  assert(GetACP=1252,'Those unitary tests expect Win1252 character set');
  // will create around 130 MB of log file, if executed
  if false then
  with TSQLLog.Family do begin
    Level := LOG_VERBOSE;
    HighResolutionTimeStamp := true;
  end;
  // testing is performed by some dedicated classes defined in the above units
  with TTestSynopseSQLite3Framework.Create('Synopse SQLite3 Framework Automated tests') do
  try
  ...

Leander007 · 2011-06-08 09:49:42

No this does not work, I think that there is no way to change system code page without restarting the OS.

Leander007 · 2011-06-08 11:31:18

My correction about code page.
TestSQL3 build in OS with code page set to 1252 runs all tests (excluding last PDF test) just fine in OS with code page set to e.g. 1250.
So the problem is in literals. All must be corrected as you did before with chr(number) in consistent way and all will work.

Last edited by Leander007 (2011-06-08 11:31:46)

ab · 2011-06-08 13:00:10

Leander007 wrote:

So the problem is in literals. All must be corrected as you did before with chr(number) in consistent way and all will work.

Or you should force all source code page to Win1252 in the editor, and it will work, won't it?

Leander007 · 2011-06-08 13:26:32

Hm, next error is strange for me, you will get it much faster than me, what is going on.

Direct access file test failure:
When I look in Test1.json dump file, created with different exe-s (1252,1250), the only difference is in BLOB part:

Unit SQLite3

  R.Bind(1,PAnsiChar(chr(224)+chr(233)+chr(231)+'dsjdsBLOB23'),14); // Bind Blob
(...)
  JS := Demo.ExecuteJSON(Req); // get result in JSON format
  FileFromString(JS,'Test1.json');
  Check(Hash32(JS)=$221D7DBD,'Expected ExecuteJSON result not retrieved');//line 5623

Here are JSON excerpts (Difference is in 2 Bytes in BLOB ?! Are we not forcing this with chr(...) to be correct, where is the problem?):
1250: "Mônet","YennZHNqZHNCTE9CMjM=",1840,1926,19, -> Ye is $59 $65
1252: "Mônet","4OnnZHNqZHNCTE9CMjM=",1840,1926,19, -> 4O is $34 $4F

Ok, with debugger I saw that sqlite3_column_blob api in

W.WrBase64(sqlite3_column_blob(Request,i),
                sqlite3_column_bytes(Request,i),true);

returned for those 3 Bytes next values for 1250:
$61 $E9 $E7 -> 97 233 231 //strange, the first one is incorrect?!

and for 1252:
$E0 $E9 $E7 -> 224 233 231 //all three are correct

Last edited by Leander007 (2011-06-08 13:59:24)

moggco · 2011-06-08 14:44:28

Changed the code

WinAnsiToUtf8('LastName=''M'#244'net'' ORDER BY LastName');

to

      s := WinAnsiToUtf8('LastName=''M'+chr(244)+'net'' ORDER BY LastName');

Yes, change all such char in "WinAnsiToUtf8(...)" from #nnn to chr(nnn).
Then all tests passed.

ab · 2011-06-08 15:12:14

Could you sent the resulting files to me?
webcontact01 at our web site domain name

So I'll check it.
Thanks!

Leander007 · 2011-06-08 22:42:03

Ok, I did not find where is webcontact01, but any way I think the file wont help much, because I already said which bytes are different from expected result and the issue exists prior JSON encoding.
When we read from database we get "97 233 231" sequence instead of "224 233 231". So it must be wrong writing to the database or reading from the database.
I will test it further in this days when I'll have spare time.
Moggco said that "next" test is solved in general way with chr(...) solution.
Strange is that this "direct file test" fails even when we use this approach (at least when used cp 1250, but assuming other coding will fail too).

ab · 2011-06-09 05:20:24

Leander007 wrote:

So it must be wrong writing to the database or reading from the database.
I will test it further in this days when I'll have spare time.

The problem is before calling the framework: the database part is correct.
The issue is not in the framework, but in the test code, when you don't use a Win1252 Ansi charset system.

The BLOB content set via a PAnsiChar(....) constant used the current system charset for conversion, so it failed with chr().
Using an array of byte instead will fix this issue.

I've made some modifications (hope latest try).

For instance:

const // BLOBs are stored as array of byte to avoid any charset conflict
  BlobDali: array[0..3] of byte = (97,233,224,231);
  BlobMonet: array[0..13] of byte = (224,233,231,ord('d'),ord('s'),ord('j'),
        ord('d'),ord('s'),ord('B'),ord('L'),ord('O'),ord('B'),ord('2'),ord('3'));

See http://synopse.info/fossil/info/dc304caa30

Leander007 · 2011-06-09 09:08:46

Yes indeed looks that PAnsiChar casting was guilty for hidden conversion.
Super , now there is only one failing test left and that is the PDF check:

Unit SynSelfTest

813:

H := Hash32(MS.Memory,MS.Position);
Check( (H=$93D859A5) or (H=$9FB359A7) ); // Arial,Bold and Tahoma FontBBox
MS.SaveToFile(ChangeFileExt(paramstr(0),'.pdf'));

Is the date below in resulting pdf correct?

/CreationDate (D:20100610191717)
/ModDate (D:20100610191717)

Is the webcontact01 mail or is web link in some page (that I can send you the resulting pdf for quick diff with yours expected one)?

Last edited by Leander007 (2011-06-09 10:51:53)

ab · 2011-06-10 17:19:50

The only difference is in one of the following values:

/FontBBox [-628 -376 2000 1010 ]

It depends on the current version of the Arial font installed on the system...

So I've made the test totally independent of those values.
See http://synopse.info/fossil/info/a9e7c11aec

Leander007 · 2011-06-10 22:43:48

I checked your latest code (/FontBBox independace), but it does not work, the test still fails on:

Check(H=$2636C99E);

The fontbbox is still "/FontBBox [-628 -376 2000 1010 ]"

ab · 2011-06-11 08:23:04

Leander007 wrote:

The fontbbox is still "/FontBBox [-628 -376 2000 1010 ]"

If take a look at the code, you'll see that the file on disk is not the the content tested.
The /FontBBox is only changed in memory, before hash.

There was a duplicate line in the code:

SetString(s,PAnsiChar(MS.Memory),MS.Position);

I was working too late yesterday.

Fixed in http://synopse.info/fossil/info/05cdaf0b7d

Leander007 · 2011-06-11 12:44:09

I didn't really looked the code, just run the tests and look for results .

Now all tests pass just fine.

ab · 2011-06-12 12:27:12

Leander007 wrote:

Now all tests pass just fine.

Great news!

Thanks for your input!

mORMot Open Source

#1 2011-06-07 12:20:33

Synopse SQlite3 Unit Tests

#2 2011-06-07 12:51:38

Re: Synopse SQlite3 Unit Tests

#3 2011-06-07 13:26:25

Re: Synopse SQlite3 Unit Tests

#4 2011-06-07 13:59:27

Re: Synopse SQlite3 Unit Tests

#5 2011-06-07 14:00:35

Re: Synopse SQlite3 Unit Tests

#6 2011-06-07 14:10:08

Re: Synopse SQlite3 Unit Tests

#7 2011-06-07 16:33:55

Re: Synopse SQlite3 Unit Tests

#8 2011-06-07 16:34:54

Re: Synopse SQlite3 Unit Tests

#9 2011-06-07 17:01:09

Re: Synopse SQlite3 Unit Tests

#10 2011-06-07 17:21:27

Re: Synopse SQlite3 Unit Tests

#11 2011-06-07 18:05:22

Re: Synopse SQlite3 Unit Tests

#12 2011-06-07 18:29:22

Re: Synopse SQlite3 Unit Tests

#13 2011-06-07 20:15:35

Re: Synopse SQlite3 Unit Tests

#14 2011-06-08 08:03:29

Re: Synopse SQlite3 Unit Tests

#15 2011-06-08 09:49:42

Re: Synopse SQlite3 Unit Tests

#16 2011-06-08 11:31:18

Re: Synopse SQlite3 Unit Tests

#17 2011-06-08 13:00:10

Re: Synopse SQlite3 Unit Tests

#18 2011-06-08 13:26:32

Re: Synopse SQlite3 Unit Tests

#19 2011-06-08 14:44:28

Re: Synopse SQlite3 Unit Tests

#20 2011-06-08 15:12:14

Re: Synopse SQlite3 Unit Tests

#21 2011-06-08 22:42:03

Re: Synopse SQlite3 Unit Tests

#22 2011-06-09 05:20:24

Re: Synopse SQlite3 Unit Tests

#23 2011-06-09 09:08:46

Re: Synopse SQlite3 Unit Tests

#24 2011-06-10 17:19:50

Re: Synopse SQlite3 Unit Tests

#25 2011-06-10 22:43:48

Re: Synopse SQlite3 Unit Tests

#26 2011-06-11 08:23:04

Re: Synopse SQlite3 Unit Tests

#27 2011-06-11 12:44:09

Re: Synopse SQlite3 Unit Tests

#28 2011-06-12 12:27:12

Re: Synopse SQlite3 Unit Tests

Board footer