You are not logged in.
Pages: 1
Hi AB,
I have been using TZipWrite to add multiple folders to a ZIP but their contents all get merged into the root of the ZIP file.
My solution was to add a 'ZipFolder' parameter to the procedure so you can specify a Foldername to use inside the ZIP file....
procedure TZipWrite.AddFolder(const FolderName: TFileName; const Mask: TFileName;
Recursive: boolean; CompressLevel: integer;
ZipFolder: TFileName = '');
..
..
begin
if ZipFolder <> '' then
ZipFolder:= IncludeTrailingPathDelimiter(ZipFolder);
result := RecursiveAdd(IncludeTrailingPathDelimiter(FolderName), ZipFolder);
end;
Tested with mORMot1. I see mORMot2 is a little different but functionally the same
It is just my personal experience feedback after looking at terrabytes of logs (literally) on 500 mORMot servers working together (in my previous job).
Hhah - yes I can relate to that!
I like the idea of the SynLogViewer though
Cheers ab
I plan to have these additional log files as an option to log some real-time data from various sources. It's coming every few seconds and is good debugging info but is clogging up the regular log file.
I'll give it a try and see how it goes.
Cheers
I am currently writing all my log data to a single log file using
with TSQLLog.Family do
begin { enable logging to file and to console }
TSQLLog.Family.FileExistsAction:= acAppend;
DestinationPath:= ServerDataPath + 'Logs\';
...
then
TSQLLog.Add.Log(sllInfo, 'My Log Entry');
I want to split things up into separate log files - 1x log file for DB Access, 1 file for logic engine etc
I am a confused how the Logging 'Family' works. It mentions in the code comments that this can be done and so looking for some help in how best to do this.
Thanks in advance...
Well at least I feel better knowing it's not just me
I think we can put it down to a compiler bug and know that the fix ab committed is worth having, at leas for Delphi.
Thanks again for your help!
Cheers flydev - it's so weird, predictable on 3 machines for me and I can see that the inlined ElemPtr is returning an invalid pointer. It bugs me when I can't get to the root cause of a problem but I think I'm going to have to give up on this one
Forget this last post - forgot to turn on optimisation around ElemMoveTo. So the problem did exist up until ab's fix. I'll leave it at one of those quirks I have to live with unless I come across anything more...
Well it seems i have found the solution, if not the problem.
I was using an older version of Mormot1 dated may 2022.
Well that was unexpected!
I've tried this program on 3 different machines running different version of Delphi and all 3 are failing in release build.
Could you confirm you're using the older ver of mormot (before ab's fix), and I'm going to see what my 3x installs all have in common - maybe it's something in my config that's doing this.
BTW, I'm actually running a Debug build but with {$O+} and {$O-} around "procedure TDynArray.ElemMoveTo" implementation - still shows the problem but easier to debug.
Well back in the office now (Australia time ) and am also reproducing the problem in Delphi 11.1 and 11.2 too (was using 10.4 earlier).
Not sure if it's my setup, or just down to the usage case of the ElemPtr in certain situations.
I know the above has fixed things but would be interested to know if this test program reproduces the problem for you in a release build?
program MoveFastx87;
{$APPTYPE CONSOLE}
uses
SysUtils, SynCommons;
var
anArray: TInt64DynArray;
aDynArray: TDynArray;
aInt64: Int64;
begin
try
aDynArray.Init(TypeInfo(TInt64DynArray), anArray);
aInt64:= 1;
aDynArray.Add(aInt64);
if aDynArray.Pop(aInt64) then
writeln ('Popped ', aInt64);
except
on E: Exception do
Writeln(E.ClassName, ': ', E.Message);
end;
readln;
end.
Hi ab
Did some investigating over the weekend going through the generated code
The problem is occurring in TDynArray.ElemPtr - and only seems to be when the TDynArray.ElemPtr is inlined, optimised, and I'm guessing it's also depending on something like whether parameters are passed on the stack or by register to the procedure using it - not sure but it doesn't happen in all scenarios.
It's the use of the label and the Goto that are confusing the compiler...
function TDynArray.ElemPtr(index: PtrInt): pointer;
label ok;
var c: PtrUInt;
begin // very efficient code on FPC and modern Delphi
result := pointer(fValue);
if result=nil then
exit;
result := PPointer(result)^;
if result=nil then
exit;
c := PtrUInt(fCountP);
if c<>0 then begin
if PtrUInt(index)<PCardinal(c)^ then { <== only here does 'edi' register get populated with our "index" }
ok: inc(PByte(result),PtrUInt(index)*ElemSize) else { <== assumes "index" is now in 'edi' register}
result := nil
end else
{$ifdef FPC}
if PtrUInt(index)<=PPtrUInt(PtrUInt(result)-_DALEN)^ then
{$else}
if PtrUInt(index)<PPtrUInt(PtrUInt(result)-_DALEN)^ then
{$endif FPC}
goto ok else { <== Goto 'ok' but 'edi' has not been populated with our 'index' }
result := nil;
end;
I guess you would say this is a Delphi Optimisation problem - I expanded out the 'if' statement to not use the Goto and this fixed the problem - slightly more code but just as efficient I think.
function TDynArray.ElemPtr(index: PtrInt): pointer;
var c: PtrUInt;
begin // very efficient code on FPC and modern Delphi
result := pointer(fValue);
if result=nil then
exit;
result := PPointer(result)^;
if result=nil then
exit;
c := PtrUInt(fCountP);
if c<>0 then
begin
if PtrUInt(index)<PCardinal(c)^ then
inc(PByte(result),PtrUInt(index)*ElemSize)
else
result := nil
end
else
{$ifdef FPC}
if PtrUInt(index)<=PPtrUInt(PtrUInt(result)-_DALEN)^ then
{$else}
if PtrUInt(index)<PPtrUInt(PtrUInt(result)-_DALEN)^ then
{$endif FPC}
inc(PByte(result),PtrUInt(index)*ElemSize)
else
result := nil
end;
You could say this is a bit of a 'gotcha' but at the same time I can see the Goto jumping to inside an if statement being a bit of an odd scenario for it
Just for feedback, it seems this issue may have re-appeared in Delphi v11.1
Using a TDynArray to manage an an array of Int64's which I Pop items from, it worked in Debug but Release gives the "EAccessViolation (c0000005) [] at c4c540 SynCommons.MoveX87"
var
lId: int64;
begin
if fMyIDsDynArray.Pop(lID) then ...
but changing to this, works (no FastMove)...
if fMyIDsDynArray.Count > 0 then
begin
lId:= pInt64(fMyIDsDynArray.ElemPtr(0))^;
fMyIDsDynArray.Delete(0);
...
HTH...
I am calling TZipWrite.AddDeflated and passing a filename and it's giving and Assert exception in SynZip.TSynZipCompressor.Seek
It seems the base TStream.CopyFrom routine has changed from D10.4 to D11.0 and now calls TStream.GetSize as part of it's CopyFrom code and now calls TSynZipCompressor.Seek(0, soFromEnd) to determin the stream size
I've changed the TSynZipCompressor code like this...
function TSynZipCompressor.Seek(Offset: Integer; Origin: Word): Longint;
begin
if not FInitialized then
result := 0 else
if (Offset = 0) and (Origin in [soFromCurrent, soFromEnd]) then // <== changed from (Origin = soFromCurrent) then // for TStream.Position
result := FStrm.total_in else begin
result := 0;
assert((Offset = 0) and (Origin = soFromBeginning) and (FStrm.total_in = 0));
end;
end;
Arnaud - is this correct. Should it work?
Hi AB
Had the idea of shifting the DST Start/End times in GetBiasForDateTime if the value is in UTC
function TSynTimeZone.GetBiasForDateTime(const Value: TDateTime;
const TzId: TTimeZoneID; out Bias: integer; out HaveDaylight: boolean;
const DateIsUTC: boolean = false): boolean; <======== NEW ===
var ndx: integer;
d: TSynSystemTime;
tzi: PTimeZoneInfo;
std,dlt: TDateTime;
begin
if (self=nil) or (TzId='') then
ndx := -1 else
if TzID=fLastZone then
ndx := fLastIndex else begin
ndx := fZones.FindHashed(TzID);
fLastZone := TzID;
flastIndex := ndx;
end;
if ndx<0 then begin
Bias := 0;
HaveDayLight := false;
result := TzID='UTC'; // e.g. on XP
exit;
end;
d.FromDate(Value); // faster than DecodeDate
tzi := fZone[ndx].GetTziFor(d.Year);
if tzi.change_time_std.IsZero then begin
HaveDaylight := false;
Bias := tzi.Bias+tzi.bias_std;
end else begin
HaveDaylight := true;
std := tzi.change_time_std.EncodeForTimeChange(d.Year);
dlt := tzi.change_time_dlt.EncodeForTimeChange(d.Year);
// === NEW === shift the DST start and end times to convert to UTC
if DateIsUTC then
begin
std:= ((std*MinsPerDay)+tzi.Bias+tzi.bias_dlt)/MinsPerDay; // Std shifts by the DST bias
dlt:= ((dlt*MinsPerDay)+tzi.Bias+tzi.bias_std)/MinsPerDay; // Dst shifts by the STD bias
end;
if std<dlt then
if (std<=Value) and (Value<dlt) then
Bias := tzi.Bias+tzi.bias_std else
Bias := tzi.Bias+tzi.bias_dlt else
if (dlt<=Value) and (Value<std) then
Bias := tzi.Bias+tzi.bias_dlt else
Bias := tzi.Bias+tzi.bias_std;
end;
result := true;
end;
Then in UtcToLocal when we get the Bias we specify it's a UTC time
function TSynTimeZone.UtcToLocal(const UtcDateTime: TDateTime;
const TzId: TTimeZoneID): TDateTime;
var Bias: integer;
HaveDaylight: boolean;
begin
if (self=nil) or (TzId='') then
result := UtcDateTime else begin
GetBiasForDateTime(UtcDateTime,TzId,Bias,HaveDaylight, true); //<======= NEW specify it's a UTC time ===
result := ((UtcDateTime*MinsPerDay)-Bias)/MinsPerDay;
end;
end;
Tested this on a few timezones and seems to work OK (but I do find working with Timezones, DTS and bias always a bit of a struggle!)
Interestingly I noticed a slight difference testing with "IncMinute" instead of the Multiply/Divide solution. I guess it's a rounding/precision issue but couldn't work out why...
With "std:= ((std*MinsPerDay)+tzi.Bias+tzi.bias_dlt)/MinsPerDay;"
UTC4:00pm = Aus EST 2:00AM
UTC4:01pm = Aus EST 3:01AM
With "std:= incMinute(std, tzi.Bias+tzi.bias_dlt);"
UTC4:00pm = Aus EST 3:00AM
UTC4:01pm = Aus EST 3:01AM
I'm looking at applying the main bias first, then using the result to determin if the DST bias should also be applied. The downside is that it involves a double query of the timezone
Not at the moment. I'll give it some thought over the weekend and see...
The current implementation is fast! I didn't want to compromise speed with a sort of "special case" scenario. A bit of investigating I think....
I never did work out what the problem actually was, but I changed my code so instead of calling
TGDIPages.ExportPDF ('myfilename.pfc', false, false);
I create my own filestream and call
TGDIPages.ExportPDFStream(MyFileStream);
which seems to have fixed the problem.
Why this problem only appeared on 1 or 2 machines I'll never know......
Hi all...
Some feedback on a bug I think I've found with converting UTC to Local with TSynTimeZone when a timezone has Daylight Savings and we're crossing a DST start date.
TSynTimeZone.Default.UtcToLocal is determining whether Daylight Savings should be applied based on the given UTC datetime, but should be determining it based on the final Local DateTime
My example scenario...
My TimeZone is "AUS Eastern Standard Time". It is UTC+10:00 and supports daylight savings
Daylight saving started on Sunday 3rd October at 2am local time
Calling (psudo code)
TSynTimeZone.Default.UtcToLocal("saturday 2nd Oct at 10pm", "AUS Eastern Standard Time")
should calculate the bias with daylight saving applied as the final local time is on Sunday 3rd @ 9am (UTC+11:00 for DaylightSaving) ), but the bias is being calculated based on the original UTC date-time as so is without DaylightSaving. It seems in this 10 hour window the bias+DaylightSaving is being calculated incorrectly.
As soon as the UTC time rolls past 2am on Sunday 3rd October everything is calculated correctly.
I haven't checked but I guess the LocalToUTC would be incorrect as we come out of daylight savings too.
I did a little test app to identify the problem. Here's the results
UTC: 2/10/2021 10:00:00 PM is LOCAL: 3/10/2021 8:00:00 AM (E. Australia Standard Time) <-- This timezone doesn't use DST. I'm using it as a reference
UTC: 2/10/2021 10:00:00 PM is LOCAL: 3/10/2021 8:00:00 AM (AUS Eastern Standard Time) <-- This timezone does have DST. Local date conversion is incorrectUTC: 2/10/2021 10:00:00 AM is LOCAL: 2/10/2021 8:00:00 PM (E. Australia Standard Time)
UTC: 2/10/2021 10:00:00 AM is LOCAL: 2/10/2021 8:00:00 PM (AUS Eastern Standard Time) <-- incorrectUTC: 3/10/2021 12:00:00 PM is LOCAL: 3/10/2021 10:00:00 PM (E. Australia Standard Time)
UTC: 3/10/2021 12:00:00 PM is LOCAL: 3/10/2021 11:00:00 PM (AUS Eastern Standard Time) <-- incorrectUTC: 3/10/2021 1:00:00 AM is LOCAL: 3/10/2021 11:00:00 AM (E. Australia Standard Time)
UTC: 3/10/2021 1:00:00 AM is LOCAL: 3/10/2021 11:00:00 AM (AUS Eastern Standard Time) <-- incorrectUTC: 3/10/2021 2:00:00 AM is LOCAL: 3/10/2021 12:00:00 PM (E. Australia Standard Time)
UTC: 3/10/2021 2:00:00 AM is LOCAL: 3/10/2021 1:00:00 PM (AUS Eastern Standard Time) <-- correct. As soon UTC passes the local timezone DST start time everything is OK
I'm not sure if the above relates to my problem but was just wondering what the Delphi 'Screen' global variable is set to when an application is running as a service with no-one logged into the PC?
May actual problem is that my application is running (as a service) on a number of machines, some execute "TGDIPages.ExportPDFStream" just fine (in 1 or 2 seconds), others take forever (upto 1 or 2 hours) for a 2x page document, and typically I can't reproduce the problem on any of my development machines!
And if I run the same program as a console version on the customers machine it works fine (only the Service application version seems to fail)
Any thoughts?
Thanks for pointing me in the right direction AB
Set-Cookie in the header was being returned my mORMot with "Path=/root".
I'm now using HaProxys "http-response header replace value" ACL to modify this back to the requesters original path, so the final returned header has Set-Cookie with "Path=server1/root" (I think this is the correct way to do this - it's not mORMots job
All good - and no changes to the excellent mORMot
I am configuring an MVC web application behind a reverse proxy that is performing URL Rewrite and am having problems with session checks and cookies.
My reverse proxy (I've tried HaProxy and Apache) is doing a URL Rewrite and removing the first part of the URL, so a URL on the WEB as...
https://MyWebAddress/server1/root/login
gets passed the the mORMot server at the backend after a URL rewrite as
http://backendserverIP/root/login
But then subsequent calls to
CurrentSession.CheckAndRetrieve(@lCookieData,TypeInfo(TCookieData));
are returning a session ID of 0 and CookieData is empty. I also checked the web client and there is no cookie stored in the browser.
If change the proxy so the external URL matches the internal one then all works OK.
I am guessing it is related to the URL mis-match but any suggestions on a solution?
Ahh worked it out...
{{{ or {{& can be used to unescape the output
{{& JSContent "anytext"}}
Hi
I'm trying to add a Mustache Helper that inserts some JavaScript into my HTML page
The JS content contains some double-quotes but the resulting output has these escaped - I want it to just be the straight copy of what I'm sending...
Simple example....
if my HTML Template contains...
{{JSContent "anytext"}}
and my helper is like this...
procedure TMyMVCApplication.JSContent(const Value: variant; out result: variant);
begin
result:= 'this.getText = function(MyAction) {return "Redo";};';
end;
the resulting HTML after the Mustache rendering is...
this.getText = function(MyAction) {return "Redo";};
Is there any way I can stop the helper (or is it the renderer?) ESCaping the quotes?
Thanks ab - working well.
(I noted a comment of the "inc(result)" line in SessionDeleteDeprecated and wasn't sure why too - I'll keep an eye on that thread....)
I think the problem I am having is that checking for deprecated sessions only occurs if there are any currently active sessions.
Problem:
I am checking ClientsCurrent in the TSQLRestServer.OnSessionCreate method...
function TMyApp.DoOnSessionCreate(Sender: TSQLRestServer; Session: TAuthSession; Ctxt: TSQLRestServerURIContext): boolean;
var
lClientsCurrent: integer;
begin
lClientsCurrent:= Sender.Stats.ClientsCurrent;
result:= lClientsCurrent >= fMyMaxConnections;
end;
but the check for inactive sessions only occurs once a session has been created (note TSQLRestServer.SessionAccess only gets called if Ctxt.URISessionSignaturePos <> 0)...
function TSQLRestServer.SessionAccess(Ctxt: TSQLRestServerURIContext): TAuthSession;
...
fSessionsDeprecatedTix := tix; // check deprecated sessions every second
for i := fSessions.Count-1 downto 0 do
if tix>TAuthSession(fSessions.List[i]).TimeOutTix then
SessionDelete(i,nil);
So if I limit MyMaxConnections to say 2, and then both of these clients crash-stop, a third connection attempt will get declined (ClientsCurrent >= fMyMaxConnections), but the old crashed sessions will never get deleted because TSQLRestServer.SessionAccess never gets called as I don't have any real active sessions.
Solution:
I am thinking I can create my own descendant of TSQLRestServer to give me access to the protected session methods and then check for deprecated sessions during OnSessionCreate, or to let the client connect, then check the session count and use a callback to force a disconnect if needed, or is there a better way to do this?
Or have I missed something?
I noticed that if a client crash-stops, the Server.Stats.ClientsCurrent will show an incorrect number (ie it doesn't know the previous connection crashed).
Is there a way to purge old/invalid connections from the Server.Stats ?
Perfect! Cheers pvn0
My question is, is it possible to get the clients IP Address (ie the IP Address the clients web browser is running on, or the external IP Address in case of access over the internet) inside the view code.
Eg, in
procedure TMyMVCApplication.Default(var Scope: variant);
begin
{ Need to show the Default page here }
SetVariantNull(Scope);
end;
Do I have access the the clients IP Address?
Would be interested in other context info too if available.
Ah, sound like OnAuthenticationFailed is the way to go.
Cheers
Hi,
I am trying to limit the number of concurrent client connections to my server by using TSQLRestServer.OnSessionCreate and returning true in order to abort the login attempt.
I also want to set Ctxt.Call^.OutStatus to something other than HTTP_FORBIDDEN (like HTTP_PAYMENT_REQUIRED ) to let the client know it was a concurrent connection limitation rather than an invalid login attempt.
The problem I have is that when I do this, TSQLRestServer.SessionCreate in turn calls Ctxt.AuthenticationFailed(afSessionCreationAborted) which forces the OutStatus to HTTP_FORBIDDEN
procedure TSQLRestServer.SessionCreate(var User: TSQLAuthUser;
Ctxt: TSQLRestServerURIContext; out Session: TAuthSession);
var i: PtrInt;
begin
Session := nil;
if (reOneSessionPerUser in Ctxt.Call^.RestAccessRights^.AllowRemoteExecute) and
(fSessions<>nil) then
for i := 0 to fSessions.Count-1 do
if TAuthSession(fSessions.List[i]).User.fID=User.fID then begin
{$ifdef WITHLOG}
with TAuthSession(fSessions.List[i]) do
Ctxt.Log.Log(sllUserAuth,'User.LogonName=% already connected from %/%',
[User.LogonName,RemoteIP,Ctxt.Call^.LowLevelConnectionID],self);
{$endif}
Ctxt.AuthenticationFailed(afSessionAlreadyStartedForThisUser);
exit; // user already connected
end;
Session := fSessionClass.Create(Ctxt,User);
if Assigned(OnSessionCreate) then
if OnSessionCreate(self,Session,Ctxt) then begin // TRUE aborts session creation
{$ifdef WITHLOG}
Ctxt.Log.Log(sllUserAuth,'Session aborted by OnSessionCreate() callback '+
'for User.LogonName=% (connected from %/%) - clients=%, sessions=%',
[User.LogonName,Session.RemoteIP,Ctxt.Call^.LowLevelConnectionID,
fStats.GetClientsCurrent,fSessions.Count],self);
{$endif}
Ctxt.AuthenticationFailed(afSessionCreationAborted); <===== This forces Ctxt.Call^.OutStatus to HTTP_FORBIDDEN
User := nil;
FreeAndNil(Session);
exit;
end;
User := nil; // will be freed by TAuthSession.Destroy
fSessions.Add(Session);
fStats.ClientConnect;
end;
This is partly feedback for AB (thanks AB ) but also looking for suggestions for a graceful way to fix this...
Hi Arnaud,
Just letting you know of a potential bug in GetDiskPartitionsText (SynTable.pas)
The windows version seems to be ignoring the "withfreespace" parameter...
function GetInfo(var p: TDiskPartition): shortstring;
var av, fr, tot: QWord;
begin
if not withfreespace or not GetDiskInfo(p.mounted,av,fr,tot) then
{$ifdef MSWINDOWS}
FormatShort('%: % (%)',[p.mounted[1],p.name,KB(p.size,nospace)],result) else
FormatShort(F[nospace],[p.mounted[1],p.name,KB(p.size,nospace)],result); <===== doesn't include disk free space values
{$else}
FormatShort('% % (%)',[p.mounted,p.name,KB(p.size,nospace)],result) else
FormatShort(F[nospace],[p.mounted,p.name,KB(fr,nospace),KB(tot,nospace)],result);
{$endif}
end;
I tried changing the MSWINDOWS version to be the same as non-windows ...
FormatShort(F[nospace],[p.mounted,p.name,KB(fr,nospace),KB(tot,nospace)],result);
and all seems to work OK
Was there a fix to the problem with inserting Null blobs into MS SQL?
Still remains the question how to insert Null value into a varbinary(MAX) (=BLOB) field
I'm wanting to store eg TInt64DynArray in a TSQLRecord property but getting the following if the array is empty
! EXC EOleDBException {"Message":"TOleDBConnection: OLEDB Error 80040E14 - (line 1): Implicit conversion from data type nvarchar to varbinary(max) is not allowed. Use the CONVERT function to run this query.\r\n"}
I am looking at serializing/deserializing all my dynamic arrays as JSON into the DB (which I'm happy to do) but is there already an easy way to do this?
(Using v1.18 and TOleDBMSSQL2012ConnectionProperties but have tried others too with same results)
Excellent Del, I have been trying to get mORMot to compile into a package with the same problems as yourself and this seems to have fixed things.
Pages: 1