#1 Re: mORMot 2 » Regression test fails on Utf8CompareIOS » 2024-11-05 13:13:39

Regression tests now passes without failed assertions.

ab wrote:

Even if I still have a doubt about the initial demand on this UtF8CompareIOS() function.
It was meant to deal with Chinese characters and sorting, and I am not sure if LOCALE_INVARIANT would not break the search...

UtF8CompareIOS() should be used as rare as possible (i.e. for sorting).

https://learn.microsoft.com/en-us/windo … plications

On Windows it will use CompareStringW() which will fail on some computer generated text (random, base64 encoded, etc.) because of digraphs in some languages, but will be fine for text from natural language conversation.
For example for UtF8CompareIOS() "Anja" = "ANJA","ANJa" = "ANJA" , but "AnJa" <> "ANJA" in my locale.
It's a mess.

ab wrote:

I am more concerned about RandomString(): this function should work without any tweak, because it returns a WinAnsiString content, which can get over #127 as expected.
The tests is using this WinAnsi chars to validate the WinAnsi case conversion of the framework, i.e. UpperCaseU().

Actually it works as expected (returns chars in range $20 - $9F).
My bad, in _UTF() test I saw UpperCase()/LowerCase() instead of UpperCaseU()/LowerCaseU() ad figured (wrongly) that chars shoud be in 7-bit range ($20 - $7F).

Cheers

#2 mORMot 2 » Regression test fails on Utf8CompareIOS » 2024-10-31 09:41:25

Gigo
Replies: 3

Regression tests return errors in

test.core.base.pas

procedure TTestCoreBase._UTF8;
...
    W := WinAnsiString(RandomString(len));
    U := WinAnsiToUtf8(W);
...
    Up := mormot.core.unicode.UpperCase(U);
...
    CheckEqual(Utf8CompareIOS(pointer(U), pointer(Up)), 0);      // fails here
...
end

Running on Windows 10, Croatian Locale (ANSI Code Page 1250, OEM Code Page 852), both 32bit and 64bit using mORMot2 commit 2.3.8840

When comparing strings CompareStringW() considers diacritic symbols (šđčćž ŠĐČĆŽ) as well as digraphs (nj NJ or lj LJ)
On the other hand, we use mormot.core.unicode.UpperCase() which will uppercase only invariant chars <#127
We need to force CompareStringW() to use LOCALE_INVARIANT, otherwise it will return <> 0 for "nJ" = "NJ" on random-generated text.

On Linux, everything runs fine with current source.


My suggestion would be :

mormot.core.test.pas

class function TSynTestCase.RandomString(CharCount: integer): WinAnsiString;
-   PByteArray(result)[i] := 32 + R[i] and 127;                                   // can get over #127
+   PByteArray(result)[i] :=  $20 + R[i] mod 95;

mormot.core.test.pas

class function TSynTestCase.RandomAnsi7(CharCount: integer): RawByteString;
-    PByteArray(result)[i] := 32 + R[i] mod 94;
+    PByteArray(result)[i] := 32 + R[i] mod 95;                                   // tilde #$7E shoud be included (not related to errors from test)

mormot.core.os.pas

function Unicode_CompareString(PW1, PW2: PWideChar; L1, L2: PtrInt;
-  result := CompareStringW(LOCALE_USER_DEFAULT, _CASEFLAG[IgnoreCase], PW1, L1, PW2, L2);
+  result := CompareStringW(LOCALE_INVARIANT, _CASEFLAG[IgnoreCase], PW1, L1, PW2, L2);

#3 Re: mORMot 2 » High CPU load on TWebSocketAsyncServerRest » 2023-03-03 20:58:38

ab wrote:

            fEvent.ResetEvent;     /////////// ADD THIS

Solved. Please commit.


Btw, I was able to reproduce the issue on another computer with Delphi 10.4.1

#4 Re: mORMot 2 » High CPU load on TWebSocketAsyncServerRest » 2023-03-03 14:48:01

- mormot2tests regression tests passes without any regressions.
- Without OpenSSL program works as expected, without heavy CPU load.
- I've tried both OpenSSL 1.1 and OpenSSL3 with same result.
- Same thing happens with new commit.

In TAsyncConnectionsThread.Execute program loops at

  ...
      case fProcess of
      ...
        atpReadPending:
          // secondary threads wait, then read and process pending events
          begin
            fWaitForReadPending := true;
            fEvent.WaitForEver;                              //// Executes and returns
            if Terminated then
              break;
            fWakeUpFromSlowProcess := false;
            while GetNextRead(notif) do
              fOwner.fClients.ProcessRead(self, notif);     //// Executes and returns
            fThreadPollingLastWakeUpTix := 0; // will now need to wakeup
            fOwner.fThreadReadPoll.ReleaseEvent; // atpReadPoll lock above
          end;
      ...

To me, looks like fEvent.Handle should be reset, but it's not, so it loops continuously, banging on CPU.

#5 mORMot 2 » High CPU load on TWebSocketAsyncServerRest » 2023-03-03 12:04:01

Gigo
Replies: 5

Using OpenSSL, Server created with TRestHttpServer.Create with aUse=useBidirAsync param.
When a client disconnects without upgrading to WebSockets, server starts to put load on CPU (threads W root, R0,R1,R2).

On destroy, I get 3 Shutdown unfinished lines :

mormot.net.ws.async.TWebSocketAsyncConnections(02d60700) Shutdown unfinished={"TAsyncConnectionsThread(028c7600)":{Process:"atpReadPending",Index:11,Name:"R11 root"}}
mormot.net.ws.async.TWebSocketAsyncConnections(02d60700) Shutdown unfinished={"TAsyncConnectionsThread(028c6fc0)":{Process:"atpReadPending",Index:1,Name:"R1 root"}}
mormot.net.ws.async.TWebSocketAsyncConnections(02d60700) Shutdown unfinished={"TAsyncConnectionsThread(028c81e0)":{Process:"atpReadPending",Index:30,Name:"R30 root"}}

If client upgrades connection to WebSockets, everything is normal.

Same behaviour on Win32 and Win64 target.
Delphi 10.3.1, using build 2.0.5007

#6 Re: mORMot 2 » Two-way TLS » 2023-02-28 10:05:55

That was quick ! It works. Thanks !

#7 mORMot 2 » Two-way TLS » 2023-02-28 09:12:21

Gigo
Replies: 2

I'd like to be able to drop https connections that does not have trusted client certificate.
Current implementation will accept blank certificates from client.

My proposition would be (at least on OpenSSL) :

- using USE_OPENSSL;FORCE_OPENSSL conditional defines

- in mormot.net.sock add two boolean fields to TNetTlsContext

    TNetTlsContext = record
      ...
        // input: Should server request Two-Way TLS connection                      // added
        TwoWayTls : Boolean;                                                        // added
        // input: Should server verify peer just first time                         // added
        VerifyClientOnce : Boolean;                                                 // added
      ...
    end;

- in mormot.lib.openssl11 modify TOpenSslNetTls.SetupCtx

    procedure TOpenSslNetTls.SetupCtx(var Context: TNetTlsContext; Bind: boolean);
    var
      v, PeerVerifyMode: integer;
    begin
      _PeerVerify := self; // safe and simple context for the callbacks
      if Context.IgnoreCertificateErrors then
        SSL_CTX_set_verify(fCtx, SSL_VERIFY_NONE, nil)
      else
      begin
        if Assigned(Context.OnEachPeerVerify) then
          begin                                                                     // added
            PeerVerifyMode := SSL_VERIFY_PEER;                                      // added
            if Context.TwoWayTls then                                               // added
              PeerVerifyMode := PeerVerifyMode or SSL_VERIFY_FAIL_IF_NO_PEER_CERT;  // added
            if Context.VerifyClientOnce then                                        // added
              PeerVerifyMode := PeerVerifyMode or SSL_VERIFY_CLIENT_ONCE;           // added
            SSL_CTX_set_verify(fCtx, PeerVerifyMode, AfterConnectionPeerVerify)     // added
          end                                                                       // added
          // SSL_CTX_set_verify(fCtx, SSL_VERIFY_PEER, AfterConnectionPeerVerify)   // removed
        else
          SSL_CTX_set_verify(fCtx, SSL_VERIFY_PEER, nil);
        if FileExists(TFileName(Context.CACertificatesFile)) then
          SSL_CTX_load_verify_locations(
            fCtx, pointer(Context.CACertificatesFile), nil)
        else
          SSL_CTX_set_default_verify_paths(fCtx);
      end;
      ...
    end;

Then we can create server like this

    InitNetTlsContext(tls, {server=}true,
      'keys\SrvCert.pem',
      'keys\SrvPrivKey.pem',
      'password',
      'keys\CACerts.pem');
    with tls do
      begin
        //OnPeerValidate := MyPeerValidate;               // I guess this works only on client
        //OnAfterPeerValidate := MyAfterPeerValidate;     // I guess this works only on client
        OnEachPeerVerify := MyEachPeerVerify;
        IgnoreCertificateErrors := False;
        AllowDeprecatedTls := False;
        WithPeerInfo := True;
        TwoWayTls := True;
        VerifyClientOnce := True;
      end;

    HttpServer := TRestHttpServer.Create(AHttpPort, [MyRestServer],
      '+', useBidirAsync, 32, secTLS, '', '', [rsoCompressSynLZ,rsoCompressGzip], @tls);
    HttpServer.WebSocketsEnable(MyRestServer,AWebSocketKey)

On each client connection we have option to accept/drop connection using return value of MyEachPeerVerify()

If this does not break anything, maybe AB could merge this to source.
Currently, I have no idea how to achieve this via SSPI/SChannel.

#8 Re: PDF Engine » Central European characters don't render properly » 2017-09-21 07:28:03

Hi, I'm using same codepage (CP 1250 , Croatian locale)

1. Default TPdfDocument.Create should use correct codepage (ACP codepage in our case cp1250)
2. You can specify codepage in constructor i.e.  TPdfDocument.Create(false,1250);

try something like :

  FPDF := TPdfDocument.Create; //or TPdfDocument.Create(false,1250);
  try
    FPDF.EmbeddedTTF := true;
    //FPDF.EmbeddedWholeTTF := true;
    FPDF.DefaultPaperSize := psA4;
    with FPDF.AddPage do
      begin
        FPDF.Canvas.SetFont(StringToUTF8('Arial'),12, [] );

        PDF.Canvas.TextOut(100,100, #$8A#$D0#$C8#$C6#$8E);  // show characters ŠĐČĆŽ
      end

    FPDF.SaveToFile('Something.pdf')
  finally
    FreeAndNil(FPDF);
  end;

However, I've found some bugs that we need to fix and I thinks it's related to your problem.

-Text that we need to print goes trough TPdfCanvas.ShowText method
-Fonts are chosen via TPdfCanvas.SetFont methos

But in order to write AnsiString #$8A#$D0#$C8#$C6#$8E  (5 capital diacritics in our languages),
document uses 2 fonts : Unicode and WinAnsi (cp1252) not cp1250.
ShowText method tries to determine if any non-WinAnsi chars exist in text and sets font to
Unicode or WinAnsi font. Note that 2 chars from out text exists in WinAnsi cp1252 (#$8A and #$8E).
Those 2 letters are written using TPdfFontTrueType.WinAnsiFont, and #$D0#$C8#$C6 are written
using TPdfFontTrueType.UnicodeFont.

Problem arises when we have last ANSI character that is not in cp1252.
TPdfCanvas.ShowText switches to WinAnsi font but doesn't swith back to Unicode on next ShowText.

Furthermore TPdfCanvas.TextRect method calls TPdfCanvas.TextWidth method to calculate text position,
and fails at TPdfFontWinAnsi.GetAnsiCharWidth because fWinAnsiWidth = nil,
and TPdfCanvas.TextWidth method returns DefaultWidth * number of chars.

QuickFix would be :
  1. add one blank char after text if text does not need to be aligned right
  2. call TPdfCanvas.SetFont before each call to TPdfCanvas.ShowText
 
Of course, it won't help if you are using higher level components like TGDIPages / TRenderPages.


I've made 2 little changes to SynPDF that works for me but would need confirmation from AB :

procedure TPdfCanvas.ShowText(const text: PDFString; NextLine: boolean);
begin
  if (FContents<>nil) and (text<>'') then
    if (fDoc.FCharSet=ANSI_CHARSET) or IsAnsiCompatible(text) then begin
      if FPage.Font.Unicode and (FPage.FFont.FTrueTypeFontsIndex<>0) then
        SetPDFFont(TPdfFontTrueType(FPage.Font).WinAnsiFont,FPage.FontSize);
      FContents.Writer.Add('(').AddEscapeText(pointer(text),FPage.Font).Add(')').
        Add(SHOWTEXTCMD[NextLine])
    end else begin
      if not FPage.Font.Unicode and (FPage.FFont.FTrueTypeFontsIndex<>0) then     //
        if FPage.Font <> TPdfFontTrueType(FPage.Font).UnicodeFont then            // gigo
          SetPDFFont(TPdfFontTrueType(FPage.Font).UnicodeFont,FPage.FontSize);    //
      if FPage.FFont.FTrueTypeFontsIndex<>0 then
        // write TrueType text after conversion to unicode
        FContents.Writer.AddToUnicodeHexText(text,NextLine,self) else
        // this standard font should expect MBCS encoding
        FContents.Writer.Add('<').AddHex(text).Add('>').Add(SHOWTEXTCMD[NextLine]);
    end;
end;

override of TPdfFontWinAnsi.GetAnsiCharWidth :

function TPdfFontTrueType.GetAnsiCharWidth(const AText: PDFString; APos: integer): integer;   // gigo
var
  aWideChar: WideChar;
begin
  if Unicode then
    begin
      aWideChar := #32;
      CurrentAnsiConvert.AnsiBufferToUnicode(@aWideChar,@(AText[APos]),1,True);
      Result := GetWideCharWidth(aWideChar);
    end
  else
    Result := inherited GetAnsiCharWidth(AText,APos);
end;

Best regards.

#9 Re: mORMot 1 » Master/slave replication with multiple tables » 2016-05-11 11:32:06

ab wrote:

Here we use the feature on a single table per model, so perhaps we missed something.

The idea of recordversion is that it is global to all tables of the model, since it is just an increasing number.
Perhaps it should be one number per table...

I agree with one number per table approach.

  - It would enable us to sync multiple tables
  - It would be easier to follow changes to database (when debugging)
  - We could sync some tables in one direction while syncing others in opposite direction without messing up recordversion
    (e.g. "Stock" from HQ to Retail and "Invoices" from Retail to HQ at the same time)

#10 mORMot 1 » TSQLRecord adding TDateTime fields » 2014-09-04 11:50:15

Gigo
Replies: 1

Overloaded TSQLRest.Add with SimpleFields array writes blank dates to database :

type
  TTestRec = class(TSQLRecord)
  private
    FDatePosted : TDateTime;
  published
    property DatePosted : TDateTime read FDatePosted write FDatePosted;
  end;

...

  TestRec := TTestRec.Create;
  try
    TestRec.DatePosted := Now;
    Client.Add(TestRec,true)        // will insert current time
  finally
    TestRec.Free;
  end;
  
...  
  Client.Add(TTestRec,[Now])    // will insert MinDateTime

  Client.Add(TTestRec,[DateTimeToIso8601(Now,true)])    // will insert current time

Seems that TSQLRecord.SimplePropertiesFill (actually VarRecToUTF8()) cannot distinguish TDateTime from Double and TSQLPropInfoRTTIDateTime.SetValue is calling Iso8601ToDateTimePUTF8CharVar with floating-point representation of TDateTime.
Maybe if we pass TSQLRecordClass to TSQLRecord.SimplePropertiesFill as second parameter we could distinguish TDateTime from Double using TSQLRecordClass ?

#11 mORMot 1 » Currency fields, virtual methods » 2013-10-07 14:13:21

Gigo
Replies: 1

Hi Arnaud,
I encountered an issue with Currency fields in mORMot framework, that affects Win32 only.

This definition works fine :

TTestRec = class(TSQLRecord)
private
  FTestMoney : Currency;
published
  property TestMoney : Currency read FTestMoney write FTestMoney;
end;

but this does not :

TTestRec = class(TSQLRecord)
private
  FTestMoney : Currency;
published
  property TestMoney : Currency read GetTestMoney write SetTestMoney;
end;


function TTestRec.GetTestMoney: Currency;
begin
  Result := FTestMoney;
end;

procedure TTestRec.SetTestMoney(const Value: Currency);
begin
  FTestMoney := Value;
end;

With later code mORMot writes garbage to Currency field. If I try to add record again, exception "floating point stack check" is raised.
Float (double) fields works fine.

Here's what I've tried to fix this issue:
  - tried forcing USETYPEINFO / doesn't work
  - tried using older builds (down to 1.16) / doesn't work
  - tried to compile code with D7 instead of XE3 / doesn't work
  - traced source from TSQLRecord.GetJSONValues through TSQLPropInfoRTTIInt32.GetJSONValues and finally down to GetInt64Prop function, and found out that method GetTestMoney is actually called but result variable of GetInt64Prop is not filled with result of GetTestMoney.
  - finally I tied to change this code in mORMot.pas

procedure TSQLPropInfoRTTICurrency.GetJSONValues(Instance: TObject; W: TJSONSerializer);
begin
  // temp fix
  {$IFDEF CPU64}
  W.AddCurr64(GetInt64Prop(Instance,pointer(fPropInfo)));
  {$ELSE}
  W.Add(GetFloatProp(Instance,pointer(fPropInfo)));
  {$ENDIF}
end;

procedure TSQLPropInfoRTTICurrency.GetValueVar(Instance: TObject;
  ToSQL: boolean; var result: RawUTF8; wasSQLString: PBoolean);
begin
  if wasSQLString<>nil then
    wasSQLString^ := false;
  // temp fix
  {$IFDEF CPU64}
  result := Curr64ToStr(GetInt64Prop(Instance,pointer(fPropInfo)));
  {$ELSE}
  result := FloatToStr(GetFloatProp(Instance,pointer(fPropInfo)));
  {$ENDIF}
end;

and it works ! (but I guess with speed penalty)

Cheers

#12 Other components » SynZip: TZipWrite create from existing empty ZIP » 2013-09-24 07:21:45

Gigo
Replies: 1

Hi Arnaud,

I encountered an issue with TZipWrite.CreateFom constructor when I try to add files to empty archive.

This code will fail :

procedure Fail;
var
  ZW : TZipWrite;
begin
  if FileExists('Test.zip') then
    DeleteFile('Test.zip');
  // this will pass
  ZW := TZipWrite.Create('Test.zip');
  if Assigned(ZW) then
    FreeAndNil(ZW);

  // this will raise an exception
  ZW := TZipWrite.Create('Test.zip');
  if Assigned(ZW) then
    FreeAndNil(ZW);
end;

Actually the problem is in TZipRead.Create when searching for ExeOffset variable.
if I change code to :

  // existing code
  ExeOffset := -1;
  for i := ZipStartOffset to Size-5 do
    if PCardinal(@buf[i])^+1=$04034b51 then begin // +1 to avoid finding it in the exe part
      ExeOffset := i;
      break;
    end;
  // added
  if ExeOffset<0 then begin
  for i := ZipStartOffset to Size-5 do
    if PCardinal(@buf[i])^+1=$06054b51 then begin // +1 to avoid finding it in the exe part
      ExeOffset := i;
      break;
    end;
  end;
  // existing code
  if ExeOffset<0 then begin
    Unmap;
    raise Exception.Create('No ZIP found');
    exit;
  end;

then everything works (empty ZIP file does not start with file entry).


Keep up the great work,

Cheers

#13 PDF Engine » Non-ANSI Fonts » 2012-01-27 14:50:35

Gigo
Replies: 1

SynPDF 1.15

I'm trying to write some text using EASTEUROPE_CHARSET (which is defaut on my computer). Output PDF renders fine in Acrobat reader, but Ghostscript has problems with CIDFontType2 font substitution when using non-ANSI chars.
Same problem on both TPdfDocument and TPdfDocumentGDI component. I'm using Delphi7. If I set EmbeddedTTF or PDF1A to true there's no problem, fonts are embedded fine.


example 1:

procedure TForm1.TestPDFDoc;
var
  PDF : TPdfDocument;
begin
  PDF := TPdfDocument.Create;
  try
    PDF.DefaultPaperSize := psA4;
    PDF.AddPage;
    PDF.Canvas.SetFont('Arial',30, []);     // default CP is 1250
    PDF.Canvas.TextOut(100,100, #$8A#$D0#$C8#$C6#$8E);
    PDF.SaveToFile('Test.PDF');              // GS will render first char only
  finally
    PDF.Free;
  end;
end;

example 2:

procedure TForm1.TestPDFDocGDI;
var
  PDF : TPdfDocumentGDI;
begin
  PDF := TPdfDocumentGDI.Create;
  try
    PDF.DefaultPaperSize := psA4;
    PDF.AddPage;
    PDF.VCLCanvas.Font.Name := 'Arial';
    PDF.VCLCanvas.Font.Size := 30;
    PDF.VCLCanvas.Font.Charset := EASTEUROPE_CHARSET;  // same as DEFAULT_CHARSET on my computer
    PDF.VCLCanvas.TextOut(100,100,#$8A#$D0#$C8#$C6#$8E);
    PDF.SaveToFile('TestGDI.PDF');                                    // GS will render first char only
  finally
    PDF.Free;
  end;
end;

For a quick fix I modified two procedures in SynPDF.pas to force embedding font when non-ANSI chars used :

// fixed TPdfDocument issue

procedure TPdfCanvas.ShowText(const text: PDFString; NextLine: boolean);
begin
  if (FContents<>nil) and (text<>'') then
    if (fDoc.FCharSet=ANSI_CHARSET) or IsAnsiCompatible(PAnsiChar(pointer(text))) then begin
      if FPage.Font.Unicode and (FPage.FFont.FTrueTypeFontsIndex<>0) then
        SetPDFFont(TPdfFontTrueType(FPage.Font).WinAnsiFont,FPage.FontSize);
      FContents.Writer.Add('(').AddEscapeText(pointer(text),FPage.Font).Add(')').
        Add(SHOWTEXTCMD[NextLine])
    end else begin
      // gigo
      Doc.EmbeddedTTF := true;
      if FPage.FFont.FTrueTypeFontsIndex<>0 then
        // write TrueType text after conversion to unicode
        FContents.Writer.AddToUnicodeHexText(text,NextLine,self) else
        // this standard font should expect MBCS encoding
        FContents.Writer.Add('<').AddHex(text).Add('>').Add(SHOWTEXTCMD[NextLine]);
    end;
end;

// fixed TPdfDocumentGDI issue

procedure TPdfEnum.TextOut(var R: TEMRExtTextOut);
...
at line 7742
...
begin
  with DC[nDC] do begin
    SetLength(tmp,R.emrtext.nChars+1); // faster than WideString for our purpose
    move(pointer(PtrUInt(@R)+R.emrtext.offString)^,tmp[0],R.emrtext.nChars*2);

    // gigo
    if not IsAnsiCompatible(PWideChar(tmp)) then
      Canvas.Doc.EmbeddedTTF := true;

    // guess the font size
    if font.LogFont.lfHeight<0 then
      ASize := -font.LogFont.lfHeight*Canvas.fFactorY else
      ASize := font.spec.cell*Canvas.fFactorY;
    ...

The question is : Am I on right track here, or should I just avoid Ghostscript for rendering ? If this is SynPDF issue, Id like to see these (or proper) changes in future SynPDF versions.

Board footer

Powered by FluxBB