Unit mormot.lib.win7zip, support for -ssw switch and compression issue

WG42 · 2023-05-10 15:39:03

Currently I am refactoring my software to make use of mORMot2. I started years ago with TZipFile and moved later to Jedi JCL JclCompression.
mORMot2 works great so far, thank you for that.

1)
A user wrote me an email. Compressing files stops, when trying to add a file which is currently opened by another program.
When using the -ssw switch (I suppose he tried with 7z CLI) everything works fine.
> is it possible to add custom switch support? Or at least the -ssw switch for the moment so opened files can be compressed?

2)
When setting CompressionLevel to 0 (SAVE), CompressionMethod to m7Copy and setting up some more options to create an archiv, this archive gets created as expected in Save/Copy mode.
As soon as you add a password or encrypt headers, CompressionMethod will be set automatically to LZMA.

Last edited by WG42 (2023-05-10 19:00:46)

ab · 2023-05-10 16:42:59

Thanks for the feedback.

Note that there is no "switch" whatsoever in the 7Z API.
Even no simple methods to call - just a complex net of interfaces.

1) The files are opened as fmOpenReadDenyNone = fmOpenRead or fmShareDenyNone
in T7zWriter.GetStream() so they should be opened once currently opened by another program.

2) I have no idea where this behavior comes from.
Is the 7z CLI not doing the same?

WG42 · 2023-05-10 16:59:34

Thank you for your answer.

To 1) I will test that using mORMot2.
The behavior "compression stops when trying to compress a file in use/opened by another program" occurs when using JclCompression I think.

When using 7z GUI normally, everything is working as set in the options. I can set SAVE mode (0), Encrypt headers and a Password. The archive compression method is Copy instead of LZMA.
When using mORMot2, once you set a password or encrypt headers, LZMA is used.

ab · 2023-05-10 17:55:38

In respect to JclCompression, we did fix some bugs during the rewrite as mormot..lib.win7zip.pas - and I hope we did not add any new bug.

Was encrypt + save working fine with JclCompression?

WG42 · 2023-05-10 17:59:49

Encrypt Headers + Save (Level 0) and Copy mode works with JclCompression.

Tested it also with mORMot2:
- FileFormat > fh7z
- SetCompressionLevel > 0
- SetCompressionMethod7z > m7Copy
- SetPassword > password

Everything fine up to here, now add EncryptHeaders7z(True); and LZMA is applied.

Why I use mORMot2 and not JclCompression:
- it's being worked on - thank you so much for that!
- it's from france - me too... sort of
- it's better in my opinion
- many more reasons

Last edited by WG42 (2023-05-10 18:07:10)

tbo · 2023-05-10 18:12:05

ab wrote:

1) The files are opened as fmOpenReadDenyNone = fmOpenRead or fmShareDenyNone
in T7zWriter.GetStream() so they should be opened once currently opened by another program.

If you want to save a folder with function I7zWriter.AddFiles and open a file of this folder before, an exception is thrown:

var fileStream: THandleStream := TFileStreamEx.Create('c:\mORMot2\src\lib\mormot.lib.win7zip.pas', fmOpenRead);
try
  var libWriter: I7zWriter := New7zWriter(fh7z);
  libWriter.AddFiles('c:\mORMot2\src\', 'mORMot2_2023-05-01', '*.pas;*.inc', True);
  libWriter.SaveToFile(MakePath([Executable.ProgramFilePath, 'TestMultiData.7z']));
finally
  fileStream.Free;
end;

The two exceptions are as follows:

ExceptionMessage="TFileStreamEx.Create(c:\mORMot2\src\lib\mormot.lib.win7zip.pas) failed as ERROR_SHARING_VIOLATION"
ExceptionName="EOSException"
ExceptionAddress=763A8FC2
---------------------------
ExceptionMessage="T7zWriter.SaveToStream error 8007000E (Für diesen Vorgang sind nicht genügend Speicherressourcen verfügbar)"
ExceptionName="E7Zip"
ExceptionAddress=763A8FC2

The behavior is normal, unless the file would have been opened with fmOpenRead or fmShareDenyWrite/ or fmShareDenyNone. The question would be rather, couldn't you skip this file without canceling the whole process?

With best regards
Thomas

Last edited by tbo (2023-05-10 18:16:54)

WG42 · 2023-05-10 18:30:38

Maybe the ProgressCallback could result/send something once a file is done compressing or once a file has not been compressed due to an error. Like this in ProgressCallback we could catch these errors for logging.
it would be nice to have a more powerful ProgressCallback event. Is a file done compressing, did it work or not, which file has been compressed (name of it).

Last edited by WG42 (2023-05-10 18:44:42)

tbo · 2023-05-10 18:36:20

WG42 wrote:

Maybe the ProgressCallback could result/send something once a file is done compressing or once a file has not been compressed due to an error. Like this in ProgressCallback we could catch these errors for logging.

Or an additional wish, the function AddFiles would have as return a list of files that could not be packed.

With best regards
Thomas

ab · 2023-05-11 07:52:26

If the file is opened with plain fmOpenRead then no other process would be able to open it - for sure.

Please try https://github.com/synopse/mORMot2/commit/9674e7ee
- the exception should be intercepted for locked files during the process
- AddFiles() will return the list of locked files which were not marked to be added

tbo · 2023-05-11 14:13:44

ab wrote:

Please try https://github.com/synopse/mORMot2/commit/9674e7ee
- the exception should be intercepted for locked files during the process
- AddFiles() will return the list of locked files which were not marked to be added

Commit 5441 (9674e7e) works as expected. Thanks for this. The following test was passed:

var fileStream: THandleStream := Nil;
try
  fileStream := TFileStreamEx.Create('c:\mORMot2\src\lib\mormot.lib.win7zip.pas', fmOpenRead);
  var libWriter: I7zWriter := New7zWriter(fh7z);
  var skippedFiles: TFileNameDynArray := libWriter.AddFiles('c:\mORMot2\src\', '', '*.pas;*.inc', True);
  if Length(skippedFiles) > 0 then
    ShowMessage(Format('Skipped Files: %d', [Length(skippedFiles)]));

  libWriter.SaveToFile(MakePath([Executable.ProgramFilePath, 'TestData.7z']));
finally
  fileStream.Free;
end;

IsFileReadable function added in AddFiles makes no measurable difference in timing.

With best regards
Thomas

Last edited by tbo (2023-05-11 14:23:55)

WG42 · 2023-05-11 15:51:20

Thank you very much. Just wanted to know if this would be possible too

Maybe the ProgressCallback could result/send something once a file is done compressing or once a file has not been compressed due to an error. Like this in ProgressCallback we could catch these errors for logging.
it would be nice to have a more powerful ProgressCallback event. Is a file done compressing, did it work or not, which file has been compressed (name of it).

A more powerful OnProgress event would be so good!

Didn't fully understand about ssw switches and options like that in general. Is it possible to add that too?
Another software written in Delphi offers such an option to pass options like -ssw.

Passing custom parameters to the 7z.dll would also solve some problems users might get.

Last edited by WG42 (2023-05-11 16:51:01)

ab · 2023-05-11 16:44:56

@WG42
It is not possible to "pass custom parameters to 7z.dll" because, as I wrote above, the 7z.dll has no knowledge of the 7z.exe parameters whatsoever.

WG42 · 2023-05-11 16:50:06

Sorry I missunderstood something. In this other software you can just input something like x=20 or mt=4. This input is parsed and set later normally. So... nothing for the library.

Last edited by WG42 (2023-05-11 16:56:51)

WG42 · 2023-05-24 15:35:59

Could you add the skippedFile functionality also to the AddFile() function?

function T7zWriter.AddFile(const Filename: TFileName; const ZipName: RawUtf8): string;
var
  item: T7zItem;
  Handle: THandle;
begin
  Handle := FileOpen(Filename, fmOpenReadDenyNone);
  if not ValidHandle(Handle) then
    exit;
  // result := ValidHandle(Handle);
  // if not result then
  //   exit;

  if IsFileReadable(Filename) then
   begin
    item := T7zItem.Create;
    item.SourceMode := smFile;
    item.FileName := Filename;
    item.ZipName := ZipName;
    GetFileTime(Handle, @item.CreationTime, nil, @item.LastWriteTime);
    item.Size := FileSize(Handle);
    CloseHandle(Handle);
    item.Attributes := GetFileAttributes(pointer(Filename));
    item.IsFolder := item.Attributes and faDirectory <> 0;
    item.IsAnti := false;
    item.Ownership := soOwned;
    item.UpdateItemIndex := -1;
    AddOrReplace(item);
   end
   else
     // notify the caller that this file is locked
     result := Filename;
end;

Maybe there could be a list, not a function result, in which AddFile() and AddFiles() add files to skip.
So when I use 100 times AddFile() (because I have my own list I go through to add files) then I would like to check only 1 files to skip list after these 100 AddFile() calls.

Last edited by WG42 (2023-05-24 16:18:57)

tbo · 2023-05-24 16:15:30

WG42 wrote:

Could you add the skippedFile(s) functionality also to the AddFile() function?

This is not necessary because AddFile() does not include a locked file. See following example:

var fileStream: THandleStream := Nil;
try
  var testFile: TFileName := 'c:\mORMot2\src\lib\mormot.lib.win7zip.pas';
  fileStream := TFileStreamEx.Create(testFile, fmOpenRead);

  var skippedFiles: TFileNameDynArray;
  var libWriter: I7zWriter := New7zWriter(fh7z);
  libWriter.SetCompressionLevel(3);
  if libWriter.AddFile(testFile, StringToUtf8(ExtractFileName(testFile))) then
    libWriter.SaveToFile(MakePath([Executable.ProgramFilePath, 'TestData.7z']))
  else
    AddString(TStringDynArray(skippedFiles), testFile);

  for var fileName: TFileName in libWriter.AddFiles('c:\mORMot2\src\', '', '*.pas;*.inc', True) do
    AddString(TStringDynArray(skippedFiles), fileName);

  ShowMessage(Format('File(s) could not be added! Count: %d', [Length(skippedFiles)]));
finally
  fileStream.Free;
end;

With best regards
Thomas

Last edited by tbo (2023-05-24 16:16:46)

mORMot Open Source

#1 2023-05-10 15:39:03

Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#2 2023-05-10 16:42:59

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#3 2023-05-10 16:59:34

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#4 2023-05-10 17:55:38

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#5 2023-05-10 17:59:49

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#6 2023-05-10 18:12:05

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#7 2023-05-10 18:30:38

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#8 2023-05-10 18:36:20

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#9 2023-05-11 07:52:26

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#10 2023-05-11 14:13:44

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#11 2023-05-11 15:51:20

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#12 2023-05-11 16:44:56

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#13 2023-05-11 16:50:06

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#14 2023-05-24 15:35:59

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

#15 2023-05-24 16:15:30

Re: Unit mormot.lib.win7zip, support for -ssw switch and compression issue

Board footer