mORMot 2 log file creation settings

JD · 2024-08-24 09:26:03

Hi there everyone,

I would like my server to generate log files where events are recorded using local time (not UTC), limited to 10MB in size and rotated daily at midnight.

This is my current setting

  LogFamily                         := SQLite3Log.Family;
  LogFamily.Level                   := LOG_VERBOSE;
  LogFamily.PerThreadLog            := ptIdentifiedInOneFile;
  LogFamily.LocalTimestamp          := TRUE;      // Log events in local time
  LogFamily.RotateFileCount         := 1;         // > 0 enables RotateFileSizeKB / RotateFileDailyAtHour
                                                  // = 1 means logs are not compressed
                                                  // File will be rotated by size and hour
  LogFamily.RotateFileSizeKB        := 10*1024;   // rotate by 10 MB logs
  LogFamily.RotateFileDailyAtHour   := 0;         // Midnight
  ApplicationPath                   := Executable.ProgramFilePath;
  LogFamily.DestinationPath         := ExpandFileName(ApplicationPath + 'log\');
  LogFamily.FileExistsAction        := acAppend;

Is this correct? I cannot test the settings natively on Windows for now, but my tests in a Docker container did not yield the desired results.

In addition, I would like to know if it is possible to compress old log files in the ZIP format instead of the default synlz format if necessary.

Thanks a lot for your kind assistance.

Cheers,

JD

ab · 2024-08-25 07:07:23

My guess is that if you use

LogFamily.PerThreadLog            := ptIdentifiedInOneFile;

then rotation is disabled by design.

Because ptIdentifiedInOneFile means one file per thread, so it does not make any sense to rotate those files.

JD · 2024-08-25 12:35:59

Hi there ab,

Thanks for the reply. I finally got it to rotate in the Docker container. I thought that a new log file will be created automatically immediately if it is past midnight because of the RotateFileDailyAtHour setting.

This morning, yesterday's log file was the only one in use, but when I made some requests to the server, it compressed yesterday's log file in the synlz format and created a new log file for today. I did not need to remove LogFamily.PerThreadLog := ptIdentifiedInOneFile; for this to work.

Now the next question is, can the archives be in the Zip format instead of synlz?

Thanks for your assistance.

Cheers,

JD

Last edited by JD (2024-08-25 12:38:08)

ab · 2024-08-25 14:25:55

You can just use EventArchiveZip() from mormot.core.zip:

TSynLog.Family.OnArchive := @EventArchiveZip;

to generate .zip files.
But the .zip files may become quite huge at the end.

Therefore, for log files, I would rather use .gz output, as it is more usual and useful.
Just assign AlgoGZFast() mormot.core.zip.pas into LogCompressAlgo global.

JD · 2024-08-26 10:24:39

ab wrote:

Just assign AlgoGZFast() mormot.core.zip.pas into LogCompressAlgo global.

Thanks for your advice, ab. Much appreciated.

How do I assign AlgoGZFast into LogCompressAlgo? Do I have to modify the mormot.core.zip.pas to do this?

Cheers,

JD

ab · 2024-08-26 10:44:38

JD wrote:

How do I assign AlgoGZFast into LogCompressAlgo?

LogCompressAlgo := AlgoGZFast;

JD · 2024-08-26 11:23:38

Yes, the joke is on me ;o) . I added it to the EventArchiveZip function in the mormot.core.zip source file as follows:

function EventArchiveZip(
  const aOldLogFileName, aDestinationPath: TFileName): boolean;
var
  n: integer;
  z, s: TStream;
  fsize: Int64;
  ftime: TUnixMSTime;
  zipname, decname: TFileName;
begin
  result := false;
  EventArchiveZipSafe.Lock;
  LogCompressAlgo := AlgoGZFast;       // <----- New addition to compress log files in GZ format

  //---- more code follows ----- 

end;

However what this means is that I'll have to remember to add this like every time I update mORMot 2. So is it possible for you to modify the source so that the compression algorithm can be an option to be defined without having to touch mORMot 2 source files?

Thanks a million for your kind assistance.

Cheers,

JD

Coldzer0 · 2024-08-26 12:22:06

You don't have to edit the mormot2 source code.
All you need to do is add "mormot.core.buffers, mormot.core.zip" in your uses.

And at the start of your app or the log init part
"LogCompressAlgo := AlgoGZFast;"

ab · 2024-08-26 12:39:24

No, no, EventArchiveZip() should not be modified.
Your modification breaks everything.

JD · 2024-08-26 15:49:30

@ab and @Coldzer0

Thanks a lot for your advice. I've added your suggestions. I'll keep you posted on the results.

Cheers,

JD

Last edited by JD (2024-08-26 15:49:46)

flydev · 2024-08-27 07:47:28

you can find an example there: https://github.com/synopse/mORMot2/blob … #L160-L174

LogCompressAlgo is defined as global var in mormot.core.buffers, so include both units buffers and zip as @ColdZer0 said.

Last edited by flydev (2024-08-27 07:49:02)

JD · 2024-08-27 13:28:09

@flydev Thanks a lot for the link.

JD · 2024-09-02 07:05:57

UPDATE

After several days, I can now confirm that log compression (synlz, gz and Zip) works. I've tried and tested all 3 options successfully.

In addition, I was pleasantly surprised to discover that zip compression creates log archives grouped into months and years YYYYMM.zip. for example, a new Zip archive has just been created for the current month, September, while the old archive for August is still there with August log files inside it. It is nice to have this option for certain use cases.

Thanks a lot to ab and all the other contributors to mORMot for such a wonderful framework.

Cheers,

JD

Last edited by JD (2024-09-02 07:07:26)

mORMot Open Source

#1 2024-08-24 09:26:03

mORMot 2 log file creation settings

#2 2024-08-25 07:07:23

Re: mORMot 2 log file creation settings

#3 2024-08-25 12:35:59

Re: mORMot 2 log file creation settings

#4 2024-08-25 14:25:55

Re: mORMot 2 log file creation settings

#5 2024-08-26 10:24:39

Re: mORMot 2 log file creation settings

#6 2024-08-26 10:44:38

Re: mORMot 2 log file creation settings

#7 2024-08-26 11:23:38

Re: mORMot 2 log file creation settings

#8 2024-08-26 12:22:06

Re: mORMot 2 log file creation settings

#9 2024-08-26 12:39:24

Re: mORMot 2 log file creation settings

#10 2024-08-26 15:49:30

Re: mORMot 2 log file creation settings

#11 2024-08-27 07:47:28

Re: mORMot 2 log file creation settings

#12 2024-08-27 13:28:09

Re: mORMot 2 log file creation settings

#13 2024-09-02 07:05:57

Re: mORMot 2 log file creation settings

Board footer