About ZLib/SynLZ/Snappy/LZ4

ecofast · 2016-09-05 12:25:26

Hello, I did some test using System.ZLib(XE10 Berlin)、SynLZ、Snappy and LZ4, like https://github.com/Hugie/lz4-delphi did.
And the result shows that LZ4 is ABSOLUTELY the WINNER.

It seems that LZ4 HC has a more quick decompress speed and better compress ratio, but I can not find any edition of Delphi, Could anyone give some help?
Thanks!

ab · 2016-09-05 17:19:50

Not so sure.
See http://synopse.info/forum/viewtopic.php?pid=3374#p3374

And forget ZDelphi Zlib unit. Try SynZip.
Or our SynZLibSSE.pas under FPC/Win64 only by now.

emk · 2016-09-05 20:39:51

I would like to use mORMot Win32 server because many RTL/mORMot functions are asm optimised and thanks to mORMot good arhitecture memory need is low If, I would compile for Win64 I'll loose many of them. What is the solution for compression to use a mobile client (iOS/Android) in conjuction with mORMot method based service? Delphi zlib seems to slow (I dindn't test it), SynLZ it's not crossplatform, Cloudflare zlib is only for FPC/Win64).

1. Please make SynLZ crossplatform or write some Snappy wrapper(crossplatform) and define some "synaessnappy" header; snappy seems to be on the same level with SynLZ.
2. What compression to use for AJAX + Win32. Do you intend to enhance the speed of zlib for Win32?

Thank you.

ecofast · 2016-09-06 04:24:27

ab wrote:

Not so sure.
See http://synopse.info/forum/viewtopic.php?pid=3374#p3374
And forget ZDelphi Zlib unit. Try SynZip.
Or our SynZLibSSE.pas under FPC/Win64 only by now.

Hi ab,
Through the way https://github.com/atelierw/LZ4Delphi did, i modified https://github.com/Hugie/lz4-delphi, add function LZ4_compressHC(const ASource: Pointer; ADestination: Pointer; AInputSize: Integer): Integer; cdecl; external name '_LZ4_compressHC'; in lz4d.lz4 unit, and it works!
Here's my testing result: http://www.cnblogs.com/ecofast/p/5844016.html
I DO love LZ4 HC.

ab · 2016-09-06 05:42:41

Yes, LZ4HC is comparable to UCL or LZMA: slow compression, fast decompression, good ratio.

For our client/server process, we do mainly compression on the server side, so for us, compression speed and compression ratio do matter.
The client decompression speed is not a problem, for server application scaling, in practice.
All those algorithms are much faster to decompress than to download the content!

This was the whole idea of SynLZ: make the best compression speed possible, with the best compression ratio possible, in pure pascal.
Of course, we target server process, with a highly number of concurrent requests...

emk · 2016-09-06 15:40:31

Any plans to support SynLZ on FMX platform? Sometimes I will need to transfer some large blocks to iOS and Android devices.
Thank you.

ab · 2017-08-10 12:08:03

SynLZ support for NextGen compiler is now available in the "CrossPlatform" sub-folder.
See https://github.com/synopse/mORMot/blob/ … mSynLZ.pas

Feeback is welcome, since we don't use Delphi for iOS and Android here!

ab · 2017-08-10 13:07:43

I've also written a new x64 asm optimized version of SynLZ, and profiled the existing x86 asm to be even faster than previously.
For a 100MB text log file, SynLZ is faster than Snappy, and compress better (93% instead of 84%).
For other kind of files, Snappy is slightly faster, but SynLZ compresses better, and most of the time faster.

For Win32:

Win32 Processing DragonFly-devpcm.log = 98.7 MB for 1 times
 Snappy compress in 125.07ms, ratio=84%, 789.3 MB/s
 Snappy uncompress in 70.35ms, 1.3 GB/s
 SynLZ compress in 103.61ms, ratio=93%, 952.8 MB/s
 SynLZ uncompress in 68.71ms, 1.4 GB/s

For Win64:

Win64 Processing DragonFly-devpcm.log = 98.7 MB for 1 times
 Snappy compress in 107.13ms, ratio=84%, 921.5 MB/s
 Snappy uncompress in 61.06ms, 1.5 GB/s
 SynLZ compress in 97.25ms, ratio=93%, 1015.1 MB/s
 SynLZ uncompress in 61.27ms, 1.5 GB/s

Anyway, the main benefit of SynLZ is that it was designed in plain pascal, so it is clearly cross-platform and well integrated with Delphi/FPC (no external .o).

emk · 2017-08-11 14:40:31

Perfect! Didn't try but thank you very much.

Johan · 2017-08-14 10:58:07

I still see a few opportunities for improvement. You use plain x64 instructions for zero init. These can be optimized by using SSE 128 registers.
I'd have to look into it more.

ab · 2017-08-15 07:07:43

The x86 code is for plain x86 with no SSE instructions. This is on purpose.

Cahaya · 2021-08-24 07:45:37

Lzstring same with synlz ?

(decompressFromEncodedURIComponent) any method in mormot support that ?

Thank you

ab · 2021-08-24 10:07:40

No, SynLZ is a genuine algorithm, which has nothing to do with LZString.

Cahaya · 2021-08-26 04:53:24

Is it compatible between lz and lzw algo ?
Is it compatible between lz compression and synlz ?

Thank you.

ab · 2021-08-26 06:44:05

No, SynLZ is a genuine algorithm, which has nothing to do with any other LZ algorithm.

It is LZ-based for its global pattern, but its bit encoding is genuine, and make it so fast to compress, but is unique, therefore incompatible with any other LZ implementations.

mORMot Open Source

#1 2016-09-05 12:25:26

About ZLib/SynLZ/Snappy/LZ4

#2 2016-09-05 17:19:50

Re: About ZLib/SynLZ/Snappy/LZ4

#3 2016-09-05 20:39:51

Re: About ZLib/SynLZ/Snappy/LZ4

#4 2016-09-06 04:24:27

Re: About ZLib/SynLZ/Snappy/LZ4

#5 2016-09-06 05:42:41

Re: About ZLib/SynLZ/Snappy/LZ4

#6 2016-09-06 15:40:31

Re: About ZLib/SynLZ/Snappy/LZ4

#7 2017-08-10 12:08:03

Re: About ZLib/SynLZ/Snappy/LZ4

#8 2017-08-10 13:07:43

Re: About ZLib/SynLZ/Snappy/LZ4

#9 2017-08-11 14:40:31

Re: About ZLib/SynLZ/Snappy/LZ4

#10 2017-08-14 10:58:07

Re: About ZLib/SynLZ/Snappy/LZ4

#11 2017-08-15 07:07:43

Re: About ZLib/SynLZ/Snappy/LZ4

#12 2021-08-24 07:45:37

Re: About ZLib/SynLZ/Snappy/LZ4

#13 2021-08-24 10:07:40

Re: About ZLib/SynLZ/Snappy/LZ4

#14 2021-08-26 04:53:24

Re: About ZLib/SynLZ/Snappy/LZ4

#15 2021-08-26 06:44:05

Re: About ZLib/SynLZ/Snappy/LZ4

Board footer