You are not logged in.
Pages: 1
Regression tests now passes without failed assertions.
Even if I still have a doubt about the initial demand on this UtF8CompareIOS() function.
It was meant to deal with Chinese characters and sorting, and I am not sure if LOCALE_INVARIANT would not break the search...
UtF8CompareIOS() should be used as rare as possible (i.e. for sorting).
https://learn.microsoft.com/en-us/windo … plications
On Windows it will use CompareStringW() which will fail on some computer generated text (random, base64 encoded, etc.) because of digraphs in some languages, but will be fine for text from natural language conversation.
For example for UtF8CompareIOS() "Anja" = "ANJA","ANJa" = "ANJA" , but "AnJa" <> "ANJA" in my locale.
It's a mess.
I am more concerned about RandomString(): this function should work without any tweak, because it returns a WinAnsiString content, which can get over #127 as expected.
The tests is using this WinAnsi chars to validate the WinAnsi case conversion of the framework, i.e. UpperCaseU().
Actually it works as expected (returns chars in range $20 - $9F).
My bad, in _UTF() test I saw UpperCase()/LowerCase() instead of UpperCaseU()/LowerCaseU() ad figured (wrongly) that chars shoud be in 7-bit range ($20 - $7F).
Cheers
Regression tests return errors in
test.core.base.pas
procedure TTestCoreBase._UTF8;
...
W := WinAnsiString(RandomString(len));
U := WinAnsiToUtf8(W);
...
Up := mormot.core.unicode.UpperCase(U);
...
CheckEqual(Utf8CompareIOS(pointer(U), pointer(Up)), 0); // fails here
...
end
Running on Windows 10, Croatian Locale (ANSI Code Page 1250, OEM Code Page 852), both 32bit and 64bit using mORMot2 commit 2.3.8840
When comparing strings CompareStringW() considers diacritic symbols (šđčćž ŠĐČĆŽ) as well as digraphs (nj NJ or lj LJ)
On the other hand, we use mormot.core.unicode.UpperCase() which will uppercase only invariant chars <#127
We need to force CompareStringW() to use LOCALE_INVARIANT, otherwise it will return <> 0 for "nJ" = "NJ" on random-generated text.
On Linux, everything runs fine with current source.
My suggestion would be :
mormot.core.test.pas
class function TSynTestCase.RandomString(CharCount: integer): WinAnsiString;
- PByteArray(result)[i] := 32 + R[i] and 127; // can get over #127
+ PByteArray(result)[i] := $20 + R[i] mod 95;
mormot.core.test.pas
class function TSynTestCase.RandomAnsi7(CharCount: integer): RawByteString;
- PByteArray(result)[i] := 32 + R[i] mod 94;
+ PByteArray(result)[i] := 32 + R[i] mod 95; // tilde #$7E shoud be included (not related to errors from test)
mormot.core.os.pas
function Unicode_CompareString(PW1, PW2: PWideChar; L1, L2: PtrInt;
- result := CompareStringW(LOCALE_USER_DEFAULT, _CASEFLAG[IgnoreCase], PW1, L1, PW2, L2);
+ result := CompareStringW(LOCALE_INVARIANT, _CASEFLAG[IgnoreCase], PW1, L1, PW2, L2);
fEvent.ResetEvent; /////////// ADD THIS
Solved. Please commit.
Btw, I was able to reproduce the issue on another computer with Delphi 10.4.1
- mormot2tests regression tests passes without any regressions.
- Without OpenSSL program works as expected, without heavy CPU load.
- I've tried both OpenSSL 1.1 and OpenSSL3 with same result.
- Same thing happens with new commit.
In TAsyncConnectionsThread.Execute program loops at
...
case fProcess of
...
atpReadPending:
// secondary threads wait, then read and process pending events
begin
fWaitForReadPending := true;
fEvent.WaitForEver; //// Executes and returns
if Terminated then
break;
fWakeUpFromSlowProcess := false;
while GetNextRead(notif) do
fOwner.fClients.ProcessRead(self, notif); //// Executes and returns
fThreadPollingLastWakeUpTix := 0; // will now need to wakeup
fOwner.fThreadReadPoll.ReleaseEvent; // atpReadPoll lock above
end;
...
To me, looks like fEvent.Handle should be reset, but it's not, so it loops continuously, banging on CPU.
Using OpenSSL, Server created with TRestHttpServer.Create with aUse=useBidirAsync param.
When a client disconnects without upgrading to WebSockets, server starts to put load on CPU (threads W root, R0,R1,R2).
On destroy, I get 3 Shutdown unfinished lines :
mormot.net.ws.async.TWebSocketAsyncConnections(02d60700) Shutdown unfinished={"TAsyncConnectionsThread(028c7600)":{Process:"atpReadPending",Index:11,Name:"R11 root"}}
mormot.net.ws.async.TWebSocketAsyncConnections(02d60700) Shutdown unfinished={"TAsyncConnectionsThread(028c6fc0)":{Process:"atpReadPending",Index:1,Name:"R1 root"}}
mormot.net.ws.async.TWebSocketAsyncConnections(02d60700) Shutdown unfinished={"TAsyncConnectionsThread(028c81e0)":{Process:"atpReadPending",Index:30,Name:"R30 root"}}
If client upgrades connection to WebSockets, everything is normal.
Same behaviour on Win32 and Win64 target.
Delphi 10.3.1, using build 2.0.5007
That was quick ! It works. Thanks !
I'd like to be able to drop https connections that does not have trusted client certificate.
Current implementation will accept blank certificates from client.
My proposition would be (at least on OpenSSL) :
- using USE_OPENSSL;FORCE_OPENSSL conditional defines
- in mormot.net.sock add two boolean fields to TNetTlsContext
TNetTlsContext = record
...
// input: Should server request Two-Way TLS connection // added
TwoWayTls : Boolean; // added
// input: Should server verify peer just first time // added
VerifyClientOnce : Boolean; // added
...
end;
- in mormot.lib.openssl11 modify TOpenSslNetTls.SetupCtx
procedure TOpenSslNetTls.SetupCtx(var Context: TNetTlsContext; Bind: boolean);
var
v, PeerVerifyMode: integer;
begin
_PeerVerify := self; // safe and simple context for the callbacks
if Context.IgnoreCertificateErrors then
SSL_CTX_set_verify(fCtx, SSL_VERIFY_NONE, nil)
else
begin
if Assigned(Context.OnEachPeerVerify) then
begin // added
PeerVerifyMode := SSL_VERIFY_PEER; // added
if Context.TwoWayTls then // added
PeerVerifyMode := PeerVerifyMode or SSL_VERIFY_FAIL_IF_NO_PEER_CERT; // added
if Context.VerifyClientOnce then // added
PeerVerifyMode := PeerVerifyMode or SSL_VERIFY_CLIENT_ONCE; // added
SSL_CTX_set_verify(fCtx, PeerVerifyMode, AfterConnectionPeerVerify) // added
end // added
// SSL_CTX_set_verify(fCtx, SSL_VERIFY_PEER, AfterConnectionPeerVerify) // removed
else
SSL_CTX_set_verify(fCtx, SSL_VERIFY_PEER, nil);
if FileExists(TFileName(Context.CACertificatesFile)) then
SSL_CTX_load_verify_locations(
fCtx, pointer(Context.CACertificatesFile), nil)
else
SSL_CTX_set_default_verify_paths(fCtx);
end;
...
end;
Then we can create server like this
InitNetTlsContext(tls, {server=}true,
'keys\SrvCert.pem',
'keys\SrvPrivKey.pem',
'password',
'keys\CACerts.pem');
with tls do
begin
//OnPeerValidate := MyPeerValidate; // I guess this works only on client
//OnAfterPeerValidate := MyAfterPeerValidate; // I guess this works only on client
OnEachPeerVerify := MyEachPeerVerify;
IgnoreCertificateErrors := False;
AllowDeprecatedTls := False;
WithPeerInfo := True;
TwoWayTls := True;
VerifyClientOnce := True;
end;
HttpServer := TRestHttpServer.Create(AHttpPort, [MyRestServer],
'+', useBidirAsync, 32, secTLS, '', '', [rsoCompressSynLZ,rsoCompressGzip], @tls);
HttpServer.WebSocketsEnable(MyRestServer,AWebSocketKey)
On each client connection we have option to accept/drop connection using return value of MyEachPeerVerify()
If this does not break anything, maybe AB could merge this to source.
Currently, I have no idea how to achieve this via SSPI/SChannel.
Hi, I'm using same codepage (CP 1250 , Croatian locale)
1. Default TPdfDocument.Create should use correct codepage (ACP codepage in our case cp1250)
2. You can specify codepage in constructor i.e. TPdfDocument.Create(false,1250);
try something like :
FPDF := TPdfDocument.Create; //or TPdfDocument.Create(false,1250);
try
FPDF.EmbeddedTTF := true;
//FPDF.EmbeddedWholeTTF := true;
FPDF.DefaultPaperSize := psA4;
with FPDF.AddPage do
begin
FPDF.Canvas.SetFont(StringToUTF8('Arial'),12, [] );
PDF.Canvas.TextOut(100,100, #$8A#$D0#$C8#$C6#$8E); // show characters ŠĐČĆŽ
end
FPDF.SaveToFile('Something.pdf')
finally
FreeAndNil(FPDF);
end;
However, I've found some bugs that we need to fix and I thinks it's related to your problem.
-Text that we need to print goes trough TPdfCanvas.ShowText method
-Fonts are chosen via TPdfCanvas.SetFont methos
But in order to write AnsiString #$8A#$D0#$C8#$C6#$8E (5 capital diacritics in our languages),
document uses 2 fonts : Unicode and WinAnsi (cp1252) not cp1250.
ShowText method tries to determine if any non-WinAnsi chars exist in text and sets font to
Unicode or WinAnsi font. Note that 2 chars from out text exists in WinAnsi cp1252 (#$8A and #$8E).
Those 2 letters are written using TPdfFontTrueType.WinAnsiFont, and #$D0#$C8#$C6 are written
using TPdfFontTrueType.UnicodeFont.
Problem arises when we have last ANSI character that is not in cp1252.
TPdfCanvas.ShowText switches to WinAnsi font but doesn't swith back to Unicode on next ShowText.
Furthermore TPdfCanvas.TextRect method calls TPdfCanvas.TextWidth method to calculate text position,
and fails at TPdfFontWinAnsi.GetAnsiCharWidth because fWinAnsiWidth = nil,
and TPdfCanvas.TextWidth method returns DefaultWidth * number of chars.
QuickFix would be :
1. add one blank char after text if text does not need to be aligned right
2. call TPdfCanvas.SetFont before each call to TPdfCanvas.ShowText
Of course, it won't help if you are using higher level components like TGDIPages / TRenderPages.
I've made 2 little changes to SynPDF that works for me but would need confirmation from AB :
procedure TPdfCanvas.ShowText(const text: PDFString; NextLine: boolean);
begin
if (FContents<>nil) and (text<>'') then
if (fDoc.FCharSet=ANSI_CHARSET) or IsAnsiCompatible(text) then begin
if FPage.Font.Unicode and (FPage.FFont.FTrueTypeFontsIndex<>0) then
SetPDFFont(TPdfFontTrueType(FPage.Font).WinAnsiFont,FPage.FontSize);
FContents.Writer.Add('(').AddEscapeText(pointer(text),FPage.Font).Add(')').
Add(SHOWTEXTCMD[NextLine])
end else begin
if not FPage.Font.Unicode and (FPage.FFont.FTrueTypeFontsIndex<>0) then //
if FPage.Font <> TPdfFontTrueType(FPage.Font).UnicodeFont then // gigo
SetPDFFont(TPdfFontTrueType(FPage.Font).UnicodeFont,FPage.FontSize); //
if FPage.FFont.FTrueTypeFontsIndex<>0 then
// write TrueType text after conversion to unicode
FContents.Writer.AddToUnicodeHexText(text,NextLine,self) else
// this standard font should expect MBCS encoding
FContents.Writer.Add('<').AddHex(text).Add('>').Add(SHOWTEXTCMD[NextLine]);
end;
end;
override of TPdfFontWinAnsi.GetAnsiCharWidth :
function TPdfFontTrueType.GetAnsiCharWidth(const AText: PDFString; APos: integer): integer; // gigo
var
aWideChar: WideChar;
begin
if Unicode then
begin
aWideChar := #32;
CurrentAnsiConvert.AnsiBufferToUnicode(@aWideChar,@(AText[APos]),1,True);
Result := GetWideCharWidth(aWideChar);
end
else
Result := inherited GetAnsiCharWidth(AText,APos);
end;
Best regards.
Here we use the feature on a single table per model, so perhaps we missed something.
The idea of recordversion is that it is global to all tables of the model, since it is just an increasing number.
Perhaps it should be one number per table...
I agree with one number per table approach.
- It would enable us to sync multiple tables
- It would be easier to follow changes to database (when debugging)
- We could sync some tables in one direction while syncing others in opposite direction without messing up recordversion
(e.g. "Stock" from HQ to Retail and "Invoices" from Retail to HQ at the same time)
Overloaded TSQLRest.Add with SimpleFields array writes blank dates to database :
type
TTestRec = class(TSQLRecord)
private
FDatePosted : TDateTime;
published
property DatePosted : TDateTime read FDatePosted write FDatePosted;
end;
...
TestRec := TTestRec.Create;
try
TestRec.DatePosted := Now;
Client.Add(TestRec,true) // will insert current time
finally
TestRec.Free;
end;
...
Client.Add(TTestRec,[Now]) // will insert MinDateTime
Client.Add(TTestRec,[DateTimeToIso8601(Now,true)]) // will insert current time
Seems that TSQLRecord.SimplePropertiesFill (actually VarRecToUTF8()) cannot distinguish TDateTime from Double and TSQLPropInfoRTTIDateTime.SetValue is calling Iso8601ToDateTimePUTF8CharVar with floating-point representation of TDateTime.
Maybe if we pass TSQLRecordClass to TSQLRecord.SimplePropertiesFill as second parameter we could distinguish TDateTime from Double using TSQLRecordClass ?
Hi Arnaud,
I encountered an issue with Currency fields in mORMot framework, that affects Win32 only.
This definition works fine :
TTestRec = class(TSQLRecord)
private
FTestMoney : Currency;
published
property TestMoney : Currency read FTestMoney write FTestMoney;
end;
but this does not :
TTestRec = class(TSQLRecord)
private
FTestMoney : Currency;
published
property TestMoney : Currency read GetTestMoney write SetTestMoney;
end;
function TTestRec.GetTestMoney: Currency;
begin
Result := FTestMoney;
end;
procedure TTestRec.SetTestMoney(const Value: Currency);
begin
FTestMoney := Value;
end;
With later code mORMot writes garbage to Currency field. If I try to add record again, exception "floating point stack check" is raised.
Float (double) fields works fine.
Here's what I've tried to fix this issue:
- tried forcing USETYPEINFO / doesn't work
- tried using older builds (down to 1.16) / doesn't work
- tried to compile code with D7 instead of XE3 / doesn't work
- traced source from TSQLRecord.GetJSONValues through TSQLPropInfoRTTIInt32.GetJSONValues and finally down to GetInt64Prop function, and found out that method GetTestMoney is actually called but result variable of GetInt64Prop is not filled with result of GetTestMoney.
- finally I tied to change this code in mORMot.pas
procedure TSQLPropInfoRTTICurrency.GetJSONValues(Instance: TObject; W: TJSONSerializer);
begin
// temp fix
{$IFDEF CPU64}
W.AddCurr64(GetInt64Prop(Instance,pointer(fPropInfo)));
{$ELSE}
W.Add(GetFloatProp(Instance,pointer(fPropInfo)));
{$ENDIF}
end;
procedure TSQLPropInfoRTTICurrency.GetValueVar(Instance: TObject;
ToSQL: boolean; var result: RawUTF8; wasSQLString: PBoolean);
begin
if wasSQLString<>nil then
wasSQLString^ := false;
// temp fix
{$IFDEF CPU64}
result := Curr64ToStr(GetInt64Prop(Instance,pointer(fPropInfo)));
{$ELSE}
result := FloatToStr(GetFloatProp(Instance,pointer(fPropInfo)));
{$ENDIF}
end;
and it works ! (but I guess with speed penalty)
Cheers
Hi Arnaud,
I encountered an issue with TZipWrite.CreateFom constructor when I try to add files to empty archive.
This code will fail :
procedure Fail;
var
ZW : TZipWrite;
begin
if FileExists('Test.zip') then
DeleteFile('Test.zip');
// this will pass
ZW := TZipWrite.Create('Test.zip');
if Assigned(ZW) then
FreeAndNil(ZW);
// this will raise an exception
ZW := TZipWrite.Create('Test.zip');
if Assigned(ZW) then
FreeAndNil(ZW);
end;
Actually the problem is in TZipRead.Create when searching for ExeOffset variable.
if I change code to :
// existing code
ExeOffset := -1;
for i := ZipStartOffset to Size-5 do
if PCardinal(@buf[i])^+1=$04034b51 then begin // +1 to avoid finding it in the exe part
ExeOffset := i;
break;
end;
// added
if ExeOffset<0 then begin
for i := ZipStartOffset to Size-5 do
if PCardinal(@buf[i])^+1=$06054b51 then begin // +1 to avoid finding it in the exe part
ExeOffset := i;
break;
end;
end;
// existing code
if ExeOffset<0 then begin
Unmap;
raise Exception.Create('No ZIP found');
exit;
end;
then everything works (empty ZIP file does not start with file entry).
Keep up the great work,
Cheers
SynPDF 1.15
I'm trying to write some text using EASTEUROPE_CHARSET (which is defaut on my computer). Output PDF renders fine in Acrobat reader, but Ghostscript has problems with CIDFontType2 font substitution when using non-ANSI chars.
Same problem on both TPdfDocument and TPdfDocumentGDI component. I'm using Delphi7. If I set EmbeddedTTF or PDF1A to true there's no problem, fonts are embedded fine.
example 1:
procedure TForm1.TestPDFDoc;
var
PDF : TPdfDocument;
begin
PDF := TPdfDocument.Create;
try
PDF.DefaultPaperSize := psA4;
PDF.AddPage;
PDF.Canvas.SetFont('Arial',30, []); // default CP is 1250
PDF.Canvas.TextOut(100,100, #$8A#$D0#$C8#$C6#$8E);
PDF.SaveToFile('Test.PDF'); // GS will render first char only
finally
PDF.Free;
end;
end;
example 2:
procedure TForm1.TestPDFDocGDI;
var
PDF : TPdfDocumentGDI;
begin
PDF := TPdfDocumentGDI.Create;
try
PDF.DefaultPaperSize := psA4;
PDF.AddPage;
PDF.VCLCanvas.Font.Name := 'Arial';
PDF.VCLCanvas.Font.Size := 30;
PDF.VCLCanvas.Font.Charset := EASTEUROPE_CHARSET; // same as DEFAULT_CHARSET on my computer
PDF.VCLCanvas.TextOut(100,100,#$8A#$D0#$C8#$C6#$8E);
PDF.SaveToFile('TestGDI.PDF'); // GS will render first char only
finally
PDF.Free;
end;
end;
For a quick fix I modified two procedures in SynPDF.pas to force embedding font when non-ANSI chars used :
// fixed TPdfDocument issue
procedure TPdfCanvas.ShowText(const text: PDFString; NextLine: boolean);
begin
if (FContents<>nil) and (text<>'') then
if (fDoc.FCharSet=ANSI_CHARSET) or IsAnsiCompatible(PAnsiChar(pointer(text))) then begin
if FPage.Font.Unicode and (FPage.FFont.FTrueTypeFontsIndex<>0) then
SetPDFFont(TPdfFontTrueType(FPage.Font).WinAnsiFont,FPage.FontSize);
FContents.Writer.Add('(').AddEscapeText(pointer(text),FPage.Font).Add(')').
Add(SHOWTEXTCMD[NextLine])
end else begin
// gigo
Doc.EmbeddedTTF := true;
if FPage.FFont.FTrueTypeFontsIndex<>0 then
// write TrueType text after conversion to unicode
FContents.Writer.AddToUnicodeHexText(text,NextLine,self) else
// this standard font should expect MBCS encoding
FContents.Writer.Add('<').AddHex(text).Add('>').Add(SHOWTEXTCMD[NextLine]);
end;
end;
// fixed TPdfDocumentGDI issue
procedure TPdfEnum.TextOut(var R: TEMRExtTextOut);
...
at line 7742
...
begin
with DC[nDC] do begin
SetLength(tmp,R.emrtext.nChars+1); // faster than WideString for our purpose
move(pointer(PtrUInt(@R)+R.emrtext.offString)^,tmp[0],R.emrtext.nChars*2);
// gigo
if not IsAnsiCompatible(PWideChar(tmp)) then
Canvas.Doc.EmbeddedTTF := true;
// guess the font size
if font.LogFont.lfHeight<0 then
ASize := -font.LogFont.lfHeight*Canvas.fFactorY else
ASize := font.spec.cell*Canvas.fFactorY;
...
The question is : Am I on right track here, or should I just avoid Ghostscript for rendering ? If this is SynPDF issue, Id like to see these (or proper) changes in future SynPDF versions.
Pages: 1