You are not logged in.
Pages: 1
Regression tests now passes without failed assertions.
Even if I still have a doubt about the initial demand on this UtF8CompareIOS() function.
It was meant to deal with Chinese characters and sorting, and I am not sure if LOCALE_INVARIANT would not break the search...
UtF8CompareIOS() should be used as rare as possible (i.e. for sorting).
https://learn.microsoft.com/en-us/windo … plications
On Windows it will use CompareStringW() which will fail on some computer generated text (random, base64 encoded, etc.) because of digraphs in some languages, but will be fine for text from natural language conversation. 
For example for UtF8CompareIOS() "Anja" = "ANJA","ANJa" = "ANJA" , but "AnJa" <> "ANJA" in my locale.
It's a mess.
I am more concerned about RandomString(): this function should work without any tweak, because it returns a WinAnsiString content, which can get over #127 as expected.
The tests is using this WinAnsi chars to validate the WinAnsi case conversion of the framework, i.e. UpperCaseU().
Actually it works as expected (returns chars in range $20 - $9F). 
My bad, in _UTF() test I saw UpperCase()/LowerCase() instead of UpperCaseU()/LowerCaseU() ad figured (wrongly) that chars shoud be in 7-bit range ($20 - $7F).
Cheers
Regression tests return errors in
test.core.base.pas
procedure TTestCoreBase._UTF8;
...
    W := WinAnsiString(RandomString(len));
    U := WinAnsiToUtf8(W);
...
    Up := mormot.core.unicode.UpperCase(U);
...
    CheckEqual(Utf8CompareIOS(pointer(U), pointer(Up)), 0);      // fails here
...
endRunning on Windows 10, Croatian Locale (ANSI Code Page 1250, OEM Code Page 852), both 32bit and 64bit using mORMot2 commit 2.3.8840
When comparing strings CompareStringW() considers diacritic symbols (šđčćž ŠĐČĆŽ) as well as digraphs (nj NJ or lj LJ)
On the other hand, we use mormot.core.unicode.UpperCase() which will uppercase only invariant chars <#127
We need to force CompareStringW() to use LOCALE_INVARIANT, otherwise it will return <> 0 for "nJ" = "NJ" on random-generated text.
On Linux, everything runs fine with current source.
My suggestion would be :
mormot.core.test.pas
class function TSynTestCase.RandomString(CharCount: integer): WinAnsiString;
-   PByteArray(result)[i] := 32 + R[i] and 127;                                   // can get over #127
+   PByteArray(result)[i] :=  $20 + R[i] mod 95;mormot.core.test.pas
class function TSynTestCase.RandomAnsi7(CharCount: integer): RawByteString;
-    PByteArray(result)[i] := 32 + R[i] mod 94;
+    PByteArray(result)[i] := 32 + R[i] mod 95;                                   // tilde #$7E shoud be included (not related to errors from test)mormot.core.os.pas
function Unicode_CompareString(PW1, PW2: PWideChar; L1, L2: PtrInt;
-  result := CompareStringW(LOCALE_USER_DEFAULT, _CASEFLAG[IgnoreCase], PW1, L1, PW2, L2);
+  result := CompareStringW(LOCALE_INVARIANT, _CASEFLAG[IgnoreCase], PW1, L1, PW2, L2);fEvent.ResetEvent; /////////// ADD THIS
Solved. Please commit.
Btw, I was able to reproduce the issue on another computer with Delphi 10.4.1
- mormot2tests regression tests passes without any regressions.
- Without OpenSSL program works as expected, without heavy CPU load.
- I've tried both OpenSSL 1.1 and OpenSSL3 with same result.
- Same thing happens with new commit.
In TAsyncConnectionsThread.Execute program loops at
  ...
      case fProcess of
      ...
        atpReadPending:
          // secondary threads wait, then read and process pending events
          begin
            fWaitForReadPending := true;
            fEvent.WaitForEver;                              //// Executes and returns
            if Terminated then
              break;
            fWakeUpFromSlowProcess := false;
            while GetNextRead(notif) do
              fOwner.fClients.ProcessRead(self, notif);     //// Executes and returns
            fThreadPollingLastWakeUpTix := 0; // will now need to wakeup
            fOwner.fThreadReadPoll.ReleaseEvent; // atpReadPoll lock above
          end;
      ...To me, looks like fEvent.Handle should be reset, but it's not, so it loops continuously, banging on CPU.
Using OpenSSL, Server created with TRestHttpServer.Create with aUse=useBidirAsync param.
When a client disconnects without upgrading to WebSockets, server starts to put load on CPU (threads W root, R0,R1,R2).
On destroy, I get 3 Shutdown unfinished lines :
mormot.net.ws.async.TWebSocketAsyncConnections(02d60700) Shutdown unfinished={"TAsyncConnectionsThread(028c7600)":{Process:"atpReadPending",Index:11,Name:"R11 root"}}
mormot.net.ws.async.TWebSocketAsyncConnections(02d60700) Shutdown unfinished={"TAsyncConnectionsThread(028c6fc0)":{Process:"atpReadPending",Index:1,Name:"R1 root"}}
mormot.net.ws.async.TWebSocketAsyncConnections(02d60700) Shutdown unfinished={"TAsyncConnectionsThread(028c81e0)":{Process:"atpReadPending",Index:30,Name:"R30 root"}}
If client upgrades connection to WebSockets, everything is normal.
Same behaviour on Win32 and Win64 target.
Delphi 10.3.1, using build 2.0.5007
That was quick ! It works. Thanks !
I'd like to be able to drop https connections that does not have trusted client certificate.
Current implementation will accept blank certificates from client.
My proposition would be (at least on OpenSSL) :
- using USE_OPENSSL;FORCE_OPENSSL conditional defines
- in mormot.net.sock add two boolean fields to TNetTlsContext
    TNetTlsContext = record
      ...
        // input: Should server request Two-Way TLS connection                      // added
        TwoWayTls : Boolean;                                                        // added
        // input: Should server verify peer just first time                         // added
        VerifyClientOnce : Boolean;                                                 // added
      ...
    end;- in mormot.lib.openssl11 modify TOpenSslNetTls.SetupCtx
    procedure TOpenSslNetTls.SetupCtx(var Context: TNetTlsContext; Bind: boolean);
    var
      v, PeerVerifyMode: integer;
    begin
      _PeerVerify := self; // safe and simple context for the callbacks
      if Context.IgnoreCertificateErrors then
        SSL_CTX_set_verify(fCtx, SSL_VERIFY_NONE, nil)
      else
      begin
        if Assigned(Context.OnEachPeerVerify) then
          begin                                                                     // added
            PeerVerifyMode := SSL_VERIFY_PEER;                                      // added
            if Context.TwoWayTls then                                               // added
              PeerVerifyMode := PeerVerifyMode or SSL_VERIFY_FAIL_IF_NO_PEER_CERT;  // added
            if Context.VerifyClientOnce then                                        // added
              PeerVerifyMode := PeerVerifyMode or SSL_VERIFY_CLIENT_ONCE;           // added
            SSL_CTX_set_verify(fCtx, PeerVerifyMode, AfterConnectionPeerVerify)     // added
          end                                                                       // added
          // SSL_CTX_set_verify(fCtx, SSL_VERIFY_PEER, AfterConnectionPeerVerify)   // removed
        else
          SSL_CTX_set_verify(fCtx, SSL_VERIFY_PEER, nil);
        if FileExists(TFileName(Context.CACertificatesFile)) then
          SSL_CTX_load_verify_locations(
            fCtx, pointer(Context.CACertificatesFile), nil)
        else
          SSL_CTX_set_default_verify_paths(fCtx);
      end;
      ...
    end;Then we can create server like this
    InitNetTlsContext(tls, {server=}true,
      'keys\SrvCert.pem',
      'keys\SrvPrivKey.pem',
      'password',
      'keys\CACerts.pem');
    with tls do
      begin
        //OnPeerValidate := MyPeerValidate;               // I guess this works only on client
        //OnAfterPeerValidate := MyAfterPeerValidate;     // I guess this works only on client
        OnEachPeerVerify := MyEachPeerVerify;
        IgnoreCertificateErrors := False;
        AllowDeprecatedTls := False;
        WithPeerInfo := True;
        TwoWayTls := True;
        VerifyClientOnce := True;
      end;
    HttpServer := TRestHttpServer.Create(AHttpPort, [MyRestServer],
      '+', useBidirAsync, 32, secTLS, '', '', [rsoCompressSynLZ,rsoCompressGzip], @tls);
    HttpServer.WebSocketsEnable(MyRestServer,AWebSocketKey)On each client connection we have option to accept/drop connection using return value of MyEachPeerVerify()
If this does not break anything, maybe AB could merge this to source.
Currently, I have no idea how to achieve this via SSPI/SChannel.
Hi, I'm using same codepage (CP 1250 , Croatian locale)
1. Default TPdfDocument.Create should use correct codepage (ACP codepage in our case cp1250)
2. You can specify codepage in constructor i.e.  TPdfDocument.Create(false,1250);
try something like :
  FPDF := TPdfDocument.Create; //or TPdfDocument.Create(false,1250);
  try
    FPDF.EmbeddedTTF := true;
    //FPDF.EmbeddedWholeTTF := true;
    FPDF.DefaultPaperSize := psA4;
    with FPDF.AddPage do
      begin
        FPDF.Canvas.SetFont(StringToUTF8('Arial'),12, [] );
        PDF.Canvas.TextOut(100,100, #$8A#$D0#$C8#$C6#$8E);  // show characters ŠĐČĆŽ
      end
    FPDF.SaveToFile('Something.pdf')
  finally
    FreeAndNil(FPDF);
  end;However, I've found some bugs that we need to fix and I thinks it's related to your problem.
-Text that we need to print goes trough TPdfCanvas.ShowText method
-Fonts are chosen via TPdfCanvas.SetFont methos
But in order to write AnsiString #$8A#$D0#$C8#$C6#$8E  (5 capital diacritics in our languages),
document uses 2 fonts : Unicode and WinAnsi (cp1252) not cp1250.
ShowText method tries to determine if any non-WinAnsi chars exist in text and sets font to
Unicode or WinAnsi font. Note that 2 chars from out text exists in WinAnsi cp1252 (#$8A and #$8E).
Those 2 letters are written using TPdfFontTrueType.WinAnsiFont, and #$D0#$C8#$C6 are written
using TPdfFontTrueType.UnicodeFont.
Problem arises when we have last ANSI character that is not in cp1252.
TPdfCanvas.ShowText switches to WinAnsi font but doesn't swith back to Unicode on next ShowText.
Furthermore TPdfCanvas.TextRect method calls TPdfCanvas.TextWidth method to calculate text position,
and fails at TPdfFontWinAnsi.GetAnsiCharWidth because fWinAnsiWidth = nil,
and TPdfCanvas.TextWidth method returns DefaultWidth * number of chars.
QuickFix would be :
  1. add one blank char after text if text does not need to be aligned right
  2. call TPdfCanvas.SetFont before each call to TPdfCanvas.ShowText
  
Of course, it won't help if you are using higher level components like TGDIPages / TRenderPages.
I've made 2 little changes to SynPDF that works for me but would need confirmation from AB :
procedure TPdfCanvas.ShowText(const text: PDFString; NextLine: boolean);
begin
  if (FContents<>nil) and (text<>'') then
    if (fDoc.FCharSet=ANSI_CHARSET) or IsAnsiCompatible(text) then begin
      if FPage.Font.Unicode and (FPage.FFont.FTrueTypeFontsIndex<>0) then
        SetPDFFont(TPdfFontTrueType(FPage.Font).WinAnsiFont,FPage.FontSize);
      FContents.Writer.Add('(').AddEscapeText(pointer(text),FPage.Font).Add(')').
        Add(SHOWTEXTCMD[NextLine])
    end else begin
      if not FPage.Font.Unicode and (FPage.FFont.FTrueTypeFontsIndex<>0) then     //
        if FPage.Font <> TPdfFontTrueType(FPage.Font).UnicodeFont then            // gigo
          SetPDFFont(TPdfFontTrueType(FPage.Font).UnicodeFont,FPage.FontSize);    //
      if FPage.FFont.FTrueTypeFontsIndex<>0 then
        // write TrueType text after conversion to unicode
        FContents.Writer.AddToUnicodeHexText(text,NextLine,self) else
        // this standard font should expect MBCS encoding
        FContents.Writer.Add('<').AddHex(text).Add('>').Add(SHOWTEXTCMD[NextLine]);
    end;
end;override of TPdfFontWinAnsi.GetAnsiCharWidth :
function TPdfFontTrueType.GetAnsiCharWidth(const AText: PDFString; APos: integer): integer;   // gigo
var
  aWideChar: WideChar;
begin
  if Unicode then
    begin
      aWideChar := #32;
      CurrentAnsiConvert.AnsiBufferToUnicode(@aWideChar,@(AText[APos]),1,True);
      Result := GetWideCharWidth(aWideChar);
    end
  else
    Result := inherited GetAnsiCharWidth(AText,APos);
end;Best regards.
Here we use the feature on a single table per model, so perhaps we missed something.
The idea of recordversion is that it is global to all tables of the model, since it is just an increasing number.
Perhaps it should be one number per table...
I agree with one number per table approach.
  - It would enable us to sync multiple tables
  - It would be easier to follow changes to database (when debugging)
  - We could sync some tables in one direction while syncing others in opposite direction without messing up recordversion
    (e.g. "Stock" from HQ to Retail and "Invoices" from Retail to HQ at the same time)
Overloaded TSQLRest.Add with SimpleFields array writes blank dates to database :
type
  TTestRec = class(TSQLRecord)
  private
    FDatePosted : TDateTime;
  published
    property DatePosted : TDateTime read FDatePosted write FDatePosted;
  end;
...
  TestRec := TTestRec.Create;
  try
    TestRec.DatePosted := Now;
    Client.Add(TestRec,true)        // will insert current time
  finally
    TestRec.Free;
  end;
  
...  
  Client.Add(TTestRec,[Now])    // will insert MinDateTime
  Client.Add(TTestRec,[DateTimeToIso8601(Now,true)])    // will insert current timeSeems that TSQLRecord.SimplePropertiesFill (actually VarRecToUTF8()) cannot distinguish TDateTime from Double and TSQLPropInfoRTTIDateTime.SetValue is calling Iso8601ToDateTimePUTF8CharVar with floating-point representation of TDateTime.
Maybe if we pass TSQLRecordClass to TSQLRecord.SimplePropertiesFill as second parameter we could distinguish TDateTime from Double using TSQLRecordClass ?
Hi Arnaud,
I encountered an issue with Currency fields in mORMot framework, that affects Win32 only. 
This definition works fine :
TTestRec = class(TSQLRecord)
private
  FTestMoney : Currency;
published
  property TestMoney : Currency read FTestMoney write FTestMoney;
end;but this does not :
TTestRec = class(TSQLRecord)
private
  FTestMoney : Currency;
published
  property TestMoney : Currency read GetTestMoney write SetTestMoney;
end;
function TTestRec.GetTestMoney: Currency;
begin
  Result := FTestMoney;
end;
procedure TTestRec.SetTestMoney(const Value: Currency);
begin
  FTestMoney := Value;
end;With later code mORMot writes garbage to Currency field. If I try to add record again, exception "floating point stack check" is raised.
Float (double) fields works fine.
Here's what I've tried to fix this issue:
  - tried forcing USETYPEINFO / doesn't work
  - tried using older builds (down to 1.16) / doesn't work
  - tried to compile code with D7 instead of XE3 / doesn't work
  - traced source from TSQLRecord.GetJSONValues through TSQLPropInfoRTTIInt32.GetJSONValues and finally down to GetInt64Prop function, and found out that method GetTestMoney is actually called but result variable of GetInt64Prop is not filled with result of GetTestMoney.
  - finally I tied to change this code in mORMot.pas
procedure TSQLPropInfoRTTICurrency.GetJSONValues(Instance: TObject; W: TJSONSerializer);
begin
  // temp fix
  {$IFDEF CPU64}
  W.AddCurr64(GetInt64Prop(Instance,pointer(fPropInfo)));
  {$ELSE}
  W.Add(GetFloatProp(Instance,pointer(fPropInfo)));
  {$ENDIF}
end;
procedure TSQLPropInfoRTTICurrency.GetValueVar(Instance: TObject;
  ToSQL: boolean; var result: RawUTF8; wasSQLString: PBoolean);
begin
  if wasSQLString<>nil then
    wasSQLString^ := false;
  // temp fix
  {$IFDEF CPU64}
  result := Curr64ToStr(GetInt64Prop(Instance,pointer(fPropInfo)));
  {$ELSE}
  result := FloatToStr(GetFloatProp(Instance,pointer(fPropInfo)));
  {$ENDIF}
end;and it works ! (but I guess with speed penalty)
Cheers
Hi Arnaud,
I encountered an issue with TZipWrite.CreateFom constructor when I try to add files to empty archive.
This code will fail :
procedure Fail;
var
  ZW : TZipWrite;
begin
  if FileExists('Test.zip') then
    DeleteFile('Test.zip');
  // this will pass
  ZW := TZipWrite.Create('Test.zip');
  if Assigned(ZW) then
    FreeAndNil(ZW);
  // this will raise an exception
  ZW := TZipWrite.Create('Test.zip');
  if Assigned(ZW) then
    FreeAndNil(ZW);
end;Actually the problem is in TZipRead.Create when searching for ExeOffset variable.
if I change code to :
  // existing code
  ExeOffset := -1;
  for i := ZipStartOffset to Size-5 do
    if PCardinal(@buf[i])^+1=$04034b51 then begin // +1 to avoid finding it in the exe part
      ExeOffset := i;
      break;
    end;
  // added
  if ExeOffset<0 then begin
  for i := ZipStartOffset to Size-5 do
    if PCardinal(@buf[i])^+1=$06054b51 then begin // +1 to avoid finding it in the exe part
      ExeOffset := i;
      break;
    end;
  end;
  // existing code
  if ExeOffset<0 then begin
    Unmap;
    raise Exception.Create('No ZIP found');
    exit;
  end;then everything works (empty ZIP file does not start with file entry).
Keep up the great work,
Cheers
SynPDF 1.15
I'm trying to write some text using EASTEUROPE_CHARSET (which is defaut on my computer). Output PDF renders fine in Acrobat reader, but Ghostscript has problems with CIDFontType2 font substitution when using non-ANSI chars.
Same problem on both TPdfDocument and TPdfDocumentGDI component. I'm using Delphi7. If I set EmbeddedTTF or PDF1A to true there's no problem, fonts are embedded fine.
example 1:
procedure TForm1.TestPDFDoc;
var
  PDF : TPdfDocument;
begin
  PDF := TPdfDocument.Create;
  try
    PDF.DefaultPaperSize := psA4;
    PDF.AddPage;
    PDF.Canvas.SetFont('Arial',30, []);     // default CP is 1250
    PDF.Canvas.TextOut(100,100, #$8A#$D0#$C8#$C6#$8E);
    PDF.SaveToFile('Test.PDF');              // GS will render first char only
  finally
    PDF.Free;
  end;
end;example 2:
procedure TForm1.TestPDFDocGDI;
var
  PDF : TPdfDocumentGDI;
begin
  PDF := TPdfDocumentGDI.Create;
  try
    PDF.DefaultPaperSize := psA4;
    PDF.AddPage;
    PDF.VCLCanvas.Font.Name := 'Arial';
    PDF.VCLCanvas.Font.Size := 30;
    PDF.VCLCanvas.Font.Charset := EASTEUROPE_CHARSET;  // same as DEFAULT_CHARSET on my computer
    PDF.VCLCanvas.TextOut(100,100,#$8A#$D0#$C8#$C6#$8E);
    PDF.SaveToFile('TestGDI.PDF');                                    // GS will render first char only
  finally
    PDF.Free;
  end;
end;For a quick fix I modified two procedures in SynPDF.pas to force embedding font when non-ANSI chars used :
// fixed TPdfDocument issue
procedure TPdfCanvas.ShowText(const text: PDFString; NextLine: boolean);
begin
  if (FContents<>nil) and (text<>'') then
    if (fDoc.FCharSet=ANSI_CHARSET) or IsAnsiCompatible(PAnsiChar(pointer(text))) then begin
      if FPage.Font.Unicode and (FPage.FFont.FTrueTypeFontsIndex<>0) then
        SetPDFFont(TPdfFontTrueType(FPage.Font).WinAnsiFont,FPage.FontSize);
      FContents.Writer.Add('(').AddEscapeText(pointer(text),FPage.Font).Add(')').
        Add(SHOWTEXTCMD[NextLine])
    end else begin
      // gigo
      Doc.EmbeddedTTF := true;
      if FPage.FFont.FTrueTypeFontsIndex<>0 then
        // write TrueType text after conversion to unicode
        FContents.Writer.AddToUnicodeHexText(text,NextLine,self) else
        // this standard font should expect MBCS encoding
        FContents.Writer.Add('<').AddHex(text).Add('>').Add(SHOWTEXTCMD[NextLine]);
    end;
end;// fixed TPdfDocumentGDI issue
procedure TPdfEnum.TextOut(var R: TEMRExtTextOut);
...
at line 7742
...
begin
  with DC[nDC] do begin
    SetLength(tmp,R.emrtext.nChars+1); // faster than WideString for our purpose
    move(pointer(PtrUInt(@R)+R.emrtext.offString)^,tmp[0],R.emrtext.nChars*2);
    // gigo
    if not IsAnsiCompatible(PWideChar(tmp)) then
      Canvas.Doc.EmbeddedTTF := true;
    // guess the font size
    if font.LogFont.lfHeight<0 then
      ASize := -font.LogFont.lfHeight*Canvas.fFactorY else
      ASize := font.spec.cell*Canvas.fFactorY;
    ...The question is : Am I on right track here, or should I just avoid Ghostscript for rendering ? If this is SynPDF issue, Id like to see these (or proper) changes in future SynPDF versions.
Pages: 1