mORMot Open Source

amsdev

Member

Registered: 2011-09-16

Posts: 23

SynZip.TZipWrite high memory usage

Hi,

I use TZipWrite to make large zip archive contains about 36k of small files. Result zip file is about 1 gb size.

But while compressing TZipWrite eat about 6.5 GB of RAM !!!! It's to much for my app.

I Use TZipWrite this way:

procedure TSaveThread.SaveMessages;
var
  MsgSource:TMemoryStream;
  Zipper:TZipWrite;
begin
MsgSource:=TMemoryStream.Create;
Zipper:=TZipWrite.Create('SomeFile.zip']);
try
  LocalQuery.SQL.Add('SELECT MsgSource FROM MsgList'); // LocalQuery is ZEOS TZQuery
  LocalQuery.Open;
  while not LocalQuery.EOF do
    begin
    MsgSource.Clear;
    TMemoField(LocalQuery.Fields.Fields[0]).SaveToStream(MsgSource);
    MsgSource.Position:=0;
    Zipper.AddDeflated(GenerateFilePathAndName+'.eml',MsgSource.Memory,MsgSource.Size,9);
    LocalQuery.Next;
    end;
finally
  Zipper.Free;
  MsgSource.Free;
  end;
end;

How to reduce memory usage while creating large zip archive ? 500-700 mb is Ok for me, but not 6.5 gb !

My enviroment: CentOS 7 x64, Lazarus 1.8RC5/FPC 3.0.4RC1, target Linux64

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,272

Website

Re: SynZip.TZipWrite high memory usage

There is no reason why TZipWrite use so much memory.
AddDeflated won't allocate memory, but temporary small buffers, and will directly write to the file stream.

Are you sure it is TZipWrite the culprit?
What do you see when you comment the Zipper.AddDeflated() line?

amsdev

Member

Registered: 2011-09-16

Posts: 23

Re: SynZip.TZipWrite high memory usage

If I comment out Zipper.AddDeflated() - program does not eat memory.

I need to add to archive about 36k of files 1-300kb size (it's archive of email messages stored in memo field in sqlite database).

Then process started (in thread) it's go fast and app not use much memory. After 25-30% done process becomes slower and memory usage goes up. At 90+% it's much slower and 6.5 gb of ram is used.

Heaptrc not show any memory leaks. After process completed and TZipWriter freed - memory unallocated.

amsdev

Member

Registered: 2011-09-16

Posts: 23

Re: SynZip.TZipWrite high memory usage

Looks like this problem related to FPC memory manager and debug mode/heaptrc.

After adding cmem to project uses and recompile program in Release mode problem is gone (mem usage drop from 6.5 gb to 300 mb).

Thanks for help !