mORMot Open Source

BeginEnd;

Member

Registered: 2011-01-08

Posts: 9

SynLZ vs QuickLZ

QuickLZ's site (quicklz.com) says:

QuickLZ is the world's fastest compression library, reaching 308 Mbyte/s per core

how SynLZ compares?

Last edited by BeginEnd; (2012-01-03 15:58:40)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,666

Website

Re: SynLZ vs QuickLZ

SynLZ and QuickLZ are just two diverse implementation of the LZ compression algorithm.
Both use a control word, and in-place hashing.
This is more or less the same algorithm than the one included in GIF (LZW).

IMHO SynLZ is faster, since our hand-tuned assembler version is more optimized than the QuickLZ C version.
But it will depend of course of the kind of data processed, of course. For shorted buffer, it may be faster.
For some plain text file, SynLZ ratio against LZO is faster than QuickLZ.

You can make your own benchmark, if you want to.

Main difference is perhaps the license: QuickLZ need to be paid for inclusion into a commercial product, whereas SynLZ can be used more freely via the MPL license.

Therinor

Member

Registered: 2012-01-20

Posts: 5

Re: SynLZ vs QuickLZ

What about LZ4 ?

http://code.google.com/p/lz4/

It says that's even faster than QuickLZ...

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,666

Website

Re: SynLZ vs QuickLZ

LZ4 to be faster than QuickLZ, it is possible, since QuickLZ is not so optimized in my POV.
LZ4 has a permissive BSD license, so it is like SynLZ for this.

I just made some tests, with a 190 MB file containing pascal sources (most files of our source code repository).
Running on a Core 2 duo PC.

LZ4 compression = 1.25 sec, compressed size = 71 MB, decompression = 0.44 sec
SynLZ compression = 1.09 sec, compressed size = 63 MB, decompression = 0.99 sec
zip (1) compression = 6.44 sec, compressed size = 52 MB, decompression (using our asm optimized SynZip.pas unit) = 1.49 sec
zip (6) compression = 20.07 sec, compressed size = 42 MB, decompression (using our asm optimized SynZip.pas unit) = 1.35 sec
zip (9) compression = 46.14 sec, compressed size = 41 MB, decompression (using our asm optimized SynZip.pas unit) = 1.29 sec

Decompression is slower in SynLZ, due to the algorithm used: it needs to recreate the hash table even at decompression, while it is not needed by LZ4.
Having the hash table at hand allows more patterns to be available, so compression ratio is better, at the expand of a slower speed.

Conclusion:

SynLZ compresses better than LZ4,
SynLZ is faster to compress,
but SynLZ is slower to decompress.

Anyway, SynLZ is much faster than zip, and is a very good candidate for client-server side data compression (as we use in mORMot).
See http://synopse.info/fossil/info/40eb86e9bb

Therinor

Member

Registered: 2012-01-20

Posts: 5

Re: SynLZ vs QuickLZ

Very interesting comparison.

SynLZ compressing better is expected, due to hash table (this gain is btw most sensible for text files).
LZ4 decoding faster is expected, due to the absence of hash table, as you mentioned.

But SynLZ compressing faster than LZ4 is interesting, and in fact not expected.

Is this advantage also applicable to binary files, or specific to text files ?

Another question : What was compared : source codes or binaries ?
If source code : is there a performance penalty in calling C code from Pascal ?

Regards

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,666

Website

Re: SynLZ vs QuickLZ

But SynLZ compressing faster than LZ4 is interesting, and in fact not expected.

In order to compress the data, you'll have to create hashing and maintain a reference table.
This is common to both SynLZ and LZ4, and I suspect that SynLZ assembler is more efficient than LZ4 C source code.

Is this advantage also applicable to binary files, or specific to text files ?

As always with compression, it will depend on the data itself. There is no general rule.
But since the number of patterns recognized at compression is bigger for SynLZ, it can be expected than SynLZ will always compress better than LZ4.

Another question : What was compared : source codes or binaries ?

I use binaries to make the comparison.

If source code : is there a performance penalty in calling C code from Pascal ?

The penalty will be none, if you call the C from pascal. Both are native code.

Therinor

Member

Registered: 2012-01-20

Posts: 5

Re: SynLZ vs QuickLZ

In order to compress the data, you'll have to create hashing and maintain a reference table.
This is common to both SynLZ and LZ4, and I suspect that SynLZ assembler is more efficient than LZ4 C source code.

Yes, right, this is not Pascal vs C, but ASM vs C, and you seem skilled enough to outsmart any compiler out there.
That sounds like a good explanation.

I use binaries to make the comparison.

Mmmh, then maybe I/O could get into the way.
The example I/O routine inside "LZ4demo" (posted at http://code.google.com/p/lz4/) is not really optimized, to say the least.

The penalty will be none, if you call the C from pascal. Both are native code.

Thanks for information. It's a long time since i last programmed using pascal,
i was still under the feeling that mixing both codes is quite troublesome.
Maybe i should have a look into it again,
FreePascal has made some great progresses, and i miss the convenience of Delphi.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,666

Website

Re: SynLZ vs QuickLZ

The example I/O routine inside "LZ4demo" (posted at http://code.google.com/p/lz4/) is not really optimized, to say the least.

You are right about file process - and the timing is far from precise.

We may get diverse results from direct call with a high resolution timer, like the one used for SynLZ benchmark.

Here is my test code:

program lz;

{$APPTYPE CONSOLE}

uses
  FastMM4,
  SysUtils,
  SynZip,
  SynCommons,
  SynLz;

var content,dest,new: RawByteString;
    D: integer;
    Timer: TPrecisionTimer;
begin
  if paramcount<>1 then begin
    writeln(paramstr(0)+' inputfile');
    exit;
  end;
  content := StringFromFile(paramstr(1));
  SetLength(dest,length(content)*2);
  Timer.Start;
  D := SynLZcompress1asm(Pointer(content),length(content),pointer(dest));
  writeln('synlz compress = ',Timer.Stop,' ',KB(Timer.PerSec(length(content))),'/s ',kb(D));
  SetLength(new,Length(dest));
  Timer.Start;
  assert(SynLZdecompress1asm(pointer(dest),D,pointer(new))=length(content));
  writeln('synlz uncompress = ',Timer.Stop,' ',KB(Timer.PerSec(length(content))),'/s');
  SetLength(dest,D);
  FileFromString(dest,changeFileExt(paramstr(1),'.synlz'));
  Timer.Start;
  new := CompressString(content,false,9);
  writeln('zip compress 9 = ',Timer.Stop,' ',KB(Timer.PerSec(length(content))),'/s ',kb(Length(new)));
  Timer.Start;
  dest := UncompressString(new);
  writeln('zip uncompress = ',Timer.Stop,' ',KB(Timer.PerSec(length(content))),'/s ');
  FileFromString(dest,changeFileExt(paramstr(1),'.gz'));
  Assert(dest=content);
end.

Therinor

Member

Registered: 2012-01-20

Posts: 5

Re: SynLZ vs QuickLZ

From the look of it, you have a pretty precise benchmark code.
This should give you some very good feedback on SynLZ behavior, and properly track improvements to your code.

LZ4 is providing something roughly similar, although slightly less precise, through its internal benchmark function, included into bench.c source file.

It can be used with the "-b" command, such as :

LZ4Demo.exe -b samplefile.ext

I guess it should provide something comparable.