SNAPPY delphi 64 port

RObyDP · 2016-08-04 23:05:16

Hello, I did a small port of the widely used SNAPPY compression library.
Feel free to check it for errors or enhancements: http://www.dellapasqua.com/snappy64.
Thank you.
Roberto
Btw. does somebody wants to benchmark it with synopse?

ab · 2016-08-05 06:57:56

See http://synopse.info/forum/viewtopic.php?pid=7414#p7414

It is not really a Delphi port of the library, but a Delphi header port - the library is still in C.
So supporting only Win64 is a bit like a limitation.

Under Win32, SynLZ is faster at compression time, I guess.
Under Win64, there is no tuned asm version, so the SynLZ compression timing may not be much faster than Snappy.

ab · 2016-08-05 07:23:24

SynLZ has 3 times faster compression, 10-50% faster decompression, with a better compression ratio...

Here are my benchmark under Win64 (with pure pascal SynLZ):

Snappy compress in 1.65s, size=67827200, 39.1 MB/s
Snappy compress in 1.64s, size=67827200, 39.4 MB/s
Snappy compress in 1.65s, size=67827200, 38.9 MB/s
SynLZ compress in 472.28ms, size=63090450, 127.3 MB/s
SynLZ compress in 470.66ms, size=63090450, 127.8 MB/s
SynLZ compress in 457.81ms, size=63090450, 131.4 MB/s
Snappy uncompress in 468.60ms, size=113066400, 230.1 MB/s
Snappy uncompress in 486.82ms, size=113066400, 221.4 MB/s
Snappy uncompress in 469.47ms, size=113066400, 229.6 MB/s
SynLZ uncompress in 414.99ms, size=113066400, 259.8 MB/s
SynLZ uncompress in 433.20ms, size=113066400, 248.9 MB/s
SynLZ uncompress in 429.03ms, size=113066400, 251.3 MB/s

And under Win32 (with optimized x86 asm):

SynLZ compress in 477.69ms, size=63090450, 125.9 MB/s
SynLZ compress in 466.40ms, size=63090450, 129 MB/s
SynLZ compress in 485.27ms, size=63090450, 123.9 MB/s
SynLZ uncompress in 331.49ms, size=113066400, 325.2 MB/s
SynLZ uncompress in 326.82ms, size=113066400, 329.9 MB/s
SynLZ uncompress in 328.19ms, size=113066400, 328.5 MB/s

Source is in http://pastebin.com/vN4zKhMj

So in short:
- SynLZ has a better compression ration than Snappy (here the input is some huge text file - html version)
- SynLZ is much faster than Snappy - even if here we use the "pure pascal" version under Win64: the optimized x86 asm version is even faster, under Windows or Linux
- SynLZ is cross-platform and cross-compiler

So a tuned Open Source algorithm in Delphi beats Google / C libraries.

edwinsn · 2016-08-05 09:56:08

Well done!

RObyDP · 2016-08-05 10:48:34

I have compiled the C port with a basic "bcc64 -c snappy.c", without options, where I got 0 warnings nor errors.
I have used the PORTABLE_ENDIAN library for some byte order conversion routines.
Those results are surprising, Snappy is "famous" for the speed.
Thanks for your test.
Roberto

RObyDP · 2016-08-05 12:28:17

Look, I forgot to compile with optimizations, here we go

Snappy compress in 2.92s, size=508411112, 165.9 MB/s
SynLZ compress in 3.99s, size=532219716, 127.1 MB/s

I continue to optimize then I'll update the lib.
Roberto

mpv · 2016-08-05 12:29:15

But code written by @AB is indeed "famous" for the speed x64 version of SynLZ is NOT optimized - is better to use a x32 SynLZ for comparison

Last edited by mpv (2016-08-05 12:30:57)

RObyDP · 2016-08-05 12:45:36

SNAPPY is 400% faster than SynLZ on uncompress and offers a better compression ratio

1GB test file, corei7 2.6ghz

COMPRESS
Snappy compress in 2.69s, size=508411112, 179.9 MB/s
SynLZ compress in 3.66s, size=532219716, 138.4 MB/s

UNCOMPRESS
Snappy uncompress in 952.86ms, size=1043862016, 1 GB/s
SynLZ uncompress in 3.84s, size=1043862016, 258.6 MB/s

I have updated the lib in www.dellapasqua.com/snappy64

thanks
Roberto

ab · 2016-08-05 14:04:47

I've seen that you updated the snappy.o file.
I find your 1GB virtual machine content a bit irrelevant for my use case.

Now the updated numbers for a huge text content:

Snappy compress in 701.02ms, size=67827200, 92.2 MB/s
Snappy compress in 783.11ms, size=67827200, 82.6 MB/s
Snappy compress in 737.67ms, size=67827200, 87.6 MB/s
Snappy uncompress in 216.10ms, size=113066400, 498.9 MB/s
Snappy uncompress in 212.54ms, size=113066400, 507.3 MB/s
Snappy uncompress in 192.25ms, size=113066400, 560.8 MB/s
SynLZ compress in 491.48ms, size=63090450, 122.4 MB/s
SynLZ compress in 496.62ms, size=63090450, 121.1 MB/s
SynLZ compress in 510.54ms, size=63090450, 117.8 MB/s
SynLZ uncompress in 455.88ms, size=113066400, 236.5 MB/s
SynLZ uncompress in 494.17ms, size=113066400, 218.2 MB/s
SynLZ uncompress in 443.00ms, size=113066400, 243.4 MB/s

Snappy numbers are now better than before.

I've made tests with a 180 MB JSON file:

Snappy compress in 455.40ms, size=42899374, 89.8 MB/s
Snappy compress in 456.50ms, size=42899374, 89.6 MB/s
Snappy compress in 456.22ms, size=42899374, 89.6 MB/s
Snappy uncompress in 146.44ms, size=189778220, 1.2 GB/s
Snappy uncompress in 145.39ms, size=189778220, 1.2 GB/s
Snappy uncompress in 153.23ms, size=189778220, 1.1 GB/s
SynLZ compress in 259.19ms, size=34086785, 125.4 MB/s
SynLZ compress in 254.69ms, size=34086785, 127.6 MB/s
SynLZ compress in 249.83ms, size=34086785, 130.1 MB/s
SynLZ uncompress in 266.02ms, size=189778220, 680.3 MB/s
SynLZ uncompress in 251.68ms, size=189778220, 719 MB/s
SynLZ uncompress in 258.72ms, size=189778220, 699.5 MB/s

Then with a 3103 bytes JSON file (which maps a typical JSON REST request):

Snappy compress in 867.58ms, size=159700000, 175.5 MB/s
Snappy compress in 870.35ms, size=159700000, 174.9 MB/s
Snappy compress in 874.78ms, size=159700000, 174.1 MB/s
Snappy uncompress in 201.43ms, size=310300000, 1.4 GB/s
Snappy uncompress in 202.68ms, size=310300000, 1.4 GB/s
Snappy uncompress in 205.30ms, size=310300000, 1.4 GB/s
SynLZ compress in 625.52ms, size=152100000, 231.8 MB/s
SynLZ compress in 577.85ms, size=152100000, 251 MB/s
SynLZ compress in 614.07ms, size=152100000, 236.2 MB/s
SynLZ uncompress in 836.17ms, size=310300000, 353.9 MB/s
SynLZ uncompress in 828.57ms, size=310300000, 357.1 MB/s
SynLZ uncompress in 836.36ms, size=310300000, 353.8 MB/s

And for a 95 MB log file:

Snappy compress in 1.18s, size=93947545, 75.2 MB/s
Snappy compress in 1.16s, size=93947545, 77 MB/s
Snappy compress in 1.17s, size=93947545, 76 MB/s
Snappy uncompress in 326.82ms, size=478661220, 1.3 GB/s
Snappy uncompress in 341.65ms, size=478661220, 1.3 GB/s
Snappy uncompress in 321.00ms, size=478661220, 1.3 GB/s
SynLZ compress in 754.95ms, size=64259820, 81.1 MB/s
SynLZ compress in 744.31ms, size=64259820, 82.3 MB/s
SynLZ compress in 742.28ms, size=64259820, 82.5 MB/s
SynLZ uncompress in 603.75ms, size=478661220, 756 MB/s
SynLZ uncompress in 598.76ms, size=478661220, 762.3 MB/s
SynLZ uncompress in 586.40ms, size=478661220, 778.4 MB/s

Here the compression ratio of this very compressible log file is higher for SynLZ (86%) than with Snappy (80%).

So it depends the kind of data you want to compress.
For compression ratio and compression speed, SynLZ is better than Snappy for JSON content.
Of course, uncompression is slower with SynLZ, but it was the very purpose of its algorithm.
For server process, which compresses more than it uncompresses, SynLZ is still better.

And if we use the x86 optimized version under Win32 on the same JSON content:

SynLZ compress in 477.44ms, size=152100000, 303.8 MB/s
SynLZ compress in 489.62ms, size=152100000, 296.2 MB/s
SynLZ compress in 498.82ms, size=152100000, 290.7 MB/s
SynLZ uncompress in 721.99ms, size=310300000, 409.8 MB/s
SynLZ uncompress in 729.40ms, size=310300000, 405.7 MB/s
SynLZ uncompress in 731.60ms, size=310300000, 404.4 MB/s

And x86/Win32 on the log file:

SynLZ compress in 725.73ms, size=64259820, 84.4 MB/s
SynLZ compress in 731.83ms, size=64259820, 83.7 MB/s
SynLZ compress in 737.58ms, size=64259820, 83 MB/s
SynLZ uncompress in 541.42ms, size=478661220, 843.1 MB/s
SynLZ uncompress in 504.21ms, size=478661220, 905.3 MB/s
SynLZ uncompress in 518.81ms, size=478661220, 879.8 MB/s

So we still can be proud of SynLZ.

ab · 2016-08-05 14:24:54

Updated source code of the benchmark sample:
http://pastebin.com/FLXRpWSL

Don't forget that our SynLZ unit run on Rasperry Pi, thanks to FPC!

RObyDP · 2016-08-05 16:29:23

yes, I know mormot is cool, and I love pascal code ;-)

RObyDP · 2016-08-05 16:40:24

With a 160KB text file I got those results:

Snappy compress in 93.51ms, size=10044800, 102.4 MB/s
Snappy uncompress in 23.59ms, size=16285800, 658.2 MB/s

SynLZ compress in 105.62ms, size=9391100, 84.7 MB/s
SynLZ uncompress in 68.73ms, size=16285800, 225.9 MB/s

Seems to me a good algorithm

CIAO!

RObyDP · 2016-08-06 16:36:18

hello again, I have made a look further into Andy port and the source haven't stripped out debug assertions.
Now I have updated the library www.dellapasqua.com/snappy64, here the results:

50KB json magento order
Snappy compress in 28.16ms, size=6814000, 230.7 MB/s
SynLZ compress in 66.88ms, size=6358000, 90.6 MB/s
Snappy uncompress in 16.65ms, size=43102000, 2.4 GB/s
SynLZ uncompress in 47.30ms, size=43102000, 868.9 MB/s

1GB vhd file
Snappy compress in 1.83s, size=508411112, 264.1 MB/s
SynLZ compress in 3.96s, size=532219716, 128.1 MB/s
Snappy uncompress in 931.06ms, size=1043862016, 1 GB/s
SynLZ uncompress in 3.82s, size=1043862016, 260 MB/s

so average is 250MB compress and 1GB decompress
btw. I don't want to make a flame over the marvellous mormot code, just maybe can be usefull having a faster lib for delphi "world"
btw2. mormot and A. are fantastic imho

ab · 2016-08-06 16:55:37

Other platforms are mandatory IMHO for Snappy.

Leslie7 · 2016-08-07 15:46:42

RObyDP

You are focusing on speed, but compression ratio and CPU load are very important too. Can you share results with these included?

The gain in compression speed may easily be lost while transferring less compressed data on the wire. A closer to real life test would be to extend the test case with transferring the data via WAN to a mORMot client.

Currently mORMot is at its best on 32Bit, it is excellent for cheep linux VPS's. Probably most users would be interested in 32Bit solutions.

Last edited by Leslie7 (2016-08-07 21:02:11)

ab · 2016-08-07 21:02:55

Snappy has been optimized for 64-bit x86 processors, AFAIR.

I wonder what will be the 32-bit numbers .

zed · 2016-08-08 07:00:54

RObyDP
I think, if you compile snappy as *.dll with MS Visual Studio and use it from Delphi, you may found that performance is better.

I have some experience with CityHash lib from Google. I compile it as *.obj with bcc32 and as *.dll with MSVS and compare speed for both versions. As you can guess, dll version was faster. I also test pure Pascal version, but it was even slower than *.obj version.

RObyDP · 2016-08-10 15:06:22

Sorry for the delay in this post,

in my last test those are the results:

Json 50KB TMemoryStream file test, core i7 2.6ghz, WIN64:
compression speed at 275MB/sec
decompression speed at 3.3GB/sec
compression ratio 600%

bcc clang 7.20 uses LLVM 3.3 backend and optimize as or slightly better than GCC and VC

RObyDP · 2016-08-10 15:28:16

From stackoverflow:

[...]
A while back I wrote a few garbage collectors to teach myself more about performance optimization in C.
And the results I got is in my mind enough to slightly favor clang.
Especially since garbage collection is mostly about pointer chasing and copying memory.

The results are (numbers in seconds):

+---------------------+-----+-----+
|Type |GCC |Clang|
+---------------------+-----+-----+
|Copying GC |22.46|22.55|
|Copying GC, optimized|22.01|20.22|
|Mark & Sweep | 8.72| 8.38|
|Ref Counting/Cycles |15.14|14.49|
|Ref Counting/Plain | 9.94| 9.32|
+---------------------+-----+-----+

LLVM seems slighly faster than GCC

RObyDP · 2016-08-10 22:08:35

i7 2.6ghz
50KB json TMemoryStream

64bit
Snappy compress in 226.39ms, size=68140000, 287 MB/s
Snappy uncompress in 112.78ms, size=431020000, 3.5 GB/s
SynLZ compress in 605.05ms, size=63580000, 100.2 MB/s
SynLZ uncompress in 342.00ms, size=431020000, 1.1 GB/s

32bit
Snappy compress in 329.53ms, size=68140000, 197.1 MB/s
Snappy uncompress in 165.60ms, size=431020000, 2.4 GB/s
SynLZ compress in 458.01ms, size=63580000, 132.3 MB/s
SynLZ uncompress in 335.78ms, size=431020000, 1.1 GB/s

Last edited by RObyDP (2016-08-10 22:10:07)

ab · 2016-08-11 09:39:33

I've changed the test source, which was misleading.

The compression rate (in MB/s) was wrong, since it measured the output/compressed size bandwith, not the input. So numbers were misleading, since the SynLZ compression ratio was better.
And also to be fair, the snappy environment is now created and destroyed for each compression, since it is the same for SynLZ.
See http://pastebin.com/PEjvFA4j

Here are my tests with a 3KB JSON content:

WIN64
Snappy compress in 449.30ms, ratio=49%, 658.6 MB/s
Snappy uncompress in 207.18ms, 1.3 GB/s
SynLZ compress in 591.02ms, ratio=51%, 500.7 MB/s
SynLZ uncompress in 797.56ms, 371 MB/s

Win32
Snappy compress in 633.61ms, ratio=49%, 467 MB/s
Snappy uncompress in 260.42ms, 1.1 GB/s
SynLZ compress in 491.15ms, ratio=51%, 602.5 MB/s
SynLZ uncompress in 718.31ms, 411.9 MB/s

For a JSON of 754 chars, under Win32:

Snappy compress in 155.04ms, ratio=62%, 463.7 MB/s
Snappy uncompress in 87.73ms, 819.6 MB/s
SynLZ compress in 185.49ms, ratio=63%, 387.6 MB/s
SynLZ uncompress in 160.81ms, 447.1 MB/s

For a 120MB log text file, under Win32:

Snappy compress in 217.88ms, ratio=84%, 578.6 MB/s
Snappy uncompress in 126.58ms, 996 MB/s
SynLZ compress in 198.73ms, ratio=89%, 634.4 MB/s
SynLZ uncompress in 156.55ms, 805.4 MB/s

So for a small JSON content which is TYPICAL for real world REST requests, under Win32, SynLZ is still in the race.
And SynLZ has a slightly better compression ratio than Snappy.

RObyDP · 2016-08-11 11:40:20

Look, I'm passionate of software algorithms and I'm doing those tests only for fun (at the end also snappy is a lz77 class...)

Those my actual test with your updated source:
(btw I'm using your last version of mormot and delphi berlin in release build, fastmm4 default mm)
I use a 50kb json, a typical file order from ecommerce platform

64bit
Snappy compress in 238.27ms, ratio=85%, 1.6 GB/s
Snappy uncompress in 125.02ms, 3.2 GB/s
SynLZ compress in 698.31ms, ratio=86%, 588.6 MB/s
SynLZ uncompress in 375.44ms, 1 GB/s

32bit
Snappy compress in 347.30ms, ratio=85%, 1.1 GB/s
Snappy uncompress in 149.52ms, 2.6 GB/s
SynLZ compress in 464.66ms, ratio=86%, 884.6 MB/s
SynLZ uncompress in 316.27ms, 1.2 GB/s

ab · 2016-08-11 11:48:12

A 85% of compression rate seems a bit high for client-server JSON content.
It should consist of a lot of redundant information, like spaces.

My smaller 3KB/754B contents have 50-60% ratio.
partials.json = http://pastebin.com/puAjGqLE
transactions.json = http://pastebin.com/x4WF1ajH

Nice work on Snappy integration for mobile targets!

RObyDP · 2016-08-11 12:09:43

http://pastebin.com/wBwuGP9b
(it's a random order from a customer of mine, from a web ecommerce platform; they are all between 30kb-90kb)

btw. thanks for your time

ab · 2016-08-11 12:31:54

Here are my numbers:

64bit
Snappy compress in 311.80ms, ratio=85%, 1.2 GB/s
Snappy uncompress in 171.33ms, 2.3 GB/s
SynLZ compress in 523.07ms, ratio=86%, 785.8 MB/s
SynLZ uncompress in 383.86ms, 1 GB/s

32bit
Snappy compress in 408.14ms, ratio=85%, 1007.1 MB/s
Snappy uncompress in 189.94ms, 2.1 GB/s
SynLZ compress in 524.17ms, ratio=86%, 784.1 MB/s
SynLZ uncompress in 327.67ms, 1.2 GB/s

Close to yours, indeed.

As always, compression is hardly depending on the input data content and structure...

RObyDP · 2016-08-11 22:15:31

using latest cygwin and gcc 5.40

win 64bit (slightly slower than bcc64 llvm)
Snappy compress in 244.04ms, ratio=85%, 1.6 GB/s
Snappy uncompress in 124.18ms, 3.2 GB/s

32bit (a lot faster, bcc32 doesn't optimize through llvm?)
Snappy compress in 264.61ms, ratio=85%, 1.5 GB/s
Snappy uncompress in 108.62ms, 3.6 GB/s

the question is: should I redistribute C objects made with cygwin that's gnu without license infringement?
Can those libs be linked freely in closed source app?
Any help?

If the license permits, then I can try to build the android, osx, ios through gcc cygwin...

RObyDP · 2016-08-12 00:45:20

WIN32 speed solved

After testing a lot of environments for the toolchain (mingw, cygwin, ms visual clangc2, bcc, many config options), I have opted for clang 3.8.1 latest for the win32/win64

md5 tested ok

Http Json 50KB TMemoryStream file test
Intel core i7 2.6ghz, Windows 10 Pro

Compression ratio 6x

64bit WIN64
Snappy compress in 237.33ms, ratio=85%, 1.6 GB/s
Snappy uncompress in 92.43ms, 4.3 GB/s

32bit WIN32
Snappy compress in 269.96ms, ratio=85%, 1.4 GB/s
Snappy uncompress in 135.88ms, 2.9 GB/s

http://www.dellapasqua.com/snappy64/

tnx

ab · 2016-08-12 07:17:37

Here are some updated results:

Win32 Processing partials.json = 3103 B for 100000 times
 Snappy compress in 557.15ms, ratio=49%, 531.1 MB/s
 Snappy uncompress in 251.05ms, 1.1 GB/s
 SynLZ compress in 463.86ms, ratio=51%, 637.9 MB/s
 SynLZ uncompress in 719.84ms, 411 MB/s
Win64 Processing partials.json = 3103 B for 100000 times
 Snappy compress in 449.03ms, ratio=49%, 659 MB/s
 Snappy uncompress in 181.26ms, 1.5 GB/s
 SynLZ compress in 593.45ms, ratio=51%, 498.6 MB/s
 SynLZ uncompress in 798.86ms, 370.4 MB/s

Win32 Processing transactions.json = 754 B for 100000 times
 Snappy compress in 175.34ms, ratio=62%, 410 MB/s
 Snappy uncompress in 82.72ms, 869.2 MB/s
 SynLZ compress in 186.57ms, ratio=63%, 385.4 MB/s
 SynLZ uncompress in 166.34ms, 432.2 MB/s
Win64 Processing transactions.json = 754 B for 100000 times
 Snappy compress in 118.81ms, ratio=62%, 605.2 MB/s
 Snappy uncompress in 80.42ms, 894 MB/s
 SynLZ compress in 315.26ms, ratio=63%, 228 MB/s
 SynLZ uncompress in 186.26ms, 386 MB/s

Win32 Processing DRAGONFLY1-bhshdd001ServAppStarterApi.log = 126 MB for 1 times
 Snappy compress in 178.71ms, ratio=84%, 705.5 MB/s
 Snappy uncompress in 133.82ms, 942.1 MB/s
 SynLZ compress in 204.14ms, ratio=89%, 617.6 MB/s
 SynLZ uncompress in 156.17ms, 807.3 MB/s
Win64 Processing DRAGONFLY1-bhshdd001ServAppStarterApi.log = 126 MB for 1 times
 Snappy compress in 184.52ms, ratio=84%, 683.3 MB/s
 Snappy uncompress in 133.56ms, 943.9 MB/s
 SynLZ compress in 199.35ms, ratio=89%, 632.4 MB/s
 SynLZ uncompress in 178.86ms, 704.9 MB/s

Updated test source is at http://pastebin.com/zAKEksgm
The binaries created using a GNU tool like GCC are not part of the viral license.
So you are free to distribute the .o binaries with your own license.

But it could be a good idea to supply the .bat files with the appropriate command line switches for all the compilers you used.
May help a lot of people, including me!

ab · 2016-08-14 08:11:28

I've included SynZip for comparison.
See http://pastebin.com/aj77XehL for the updated source code sample.

Win64 Processing ecommerce.json = 42 KB for 10000 times
 Snappy compress in 313.60ms, ratio=85%, 1.2 GB/s
 Snappy uncompress in 139.01ms, 2.8 GB/s
 SynLZ compress in 510.52ms, ratio=86%, 805.1 MB/s
 SynLZ uncompress in 387.99ms, 1 GB/s
 SynZip1 compress in 3.93s, ratio=89%, 104.5 MB/s
 SynZip1 uncompress in 1.80s, 227.5 MB/s
 SynZip6 compress in 9.87s, ratio=92%, 41.6 MB/s
 SynZip6 uncompress in 1.56s, 262.8 MB/s
 
Win32 Processing ecommerce.json = 42 KB for 10000 times
 Snappy compress in 350.08ms, ratio=85%, 1.1 GB/s
 Snappy uncompress in 193.09ms, 2 GB/s
 SynLZ compress in 516.00ms, ratio=86%, 796.6 MB/s
 SynLZ uncompress in 348.42ms, 1.1 GB/s
 SynZip1 compress in 1.93s, ratio=89%, 212.2 MB/s
 SynZip1 uncompress in 757.43ms, 542.6 MB/s
 SynZip6 compress in 5.38s, ratio=92%, 76.3 MB/s
 SynZip6 uncompress in 647.36ms, 634.9 MB/s

You may find the Win32 number interresting.
In fact, our SynZip version has its own version of x86 zlib, whereas the Win64 version uses the "official" zlib as given with Delphi, with is much slower.

Here are some numbers for a log file:

Win64 Processing ServAppStarterApi.log = 126 MB for 1 time
 Snappy compress in 176.28ms, ratio=84%, 715.2 MB/s
 Snappy uncompress in 132.50ms, 951.5 MB/s
 SynLZ compress in 195.70ms, ratio=89%, 644.2 MB/s
 SynLZ uncompress in 179.69ms, 701.6 MB/s
 SynZip1 compress in 1.89s, ratio=89%, 66.6 MB/s
 SynZip1 uncompress in 600.32ms, 210 MB/s
 SynZip6 compress in 4.16s, ratio=92%, 30.3 MB/s
 SynZip6 uncompress in 542.18ms, 232.5 MB/s
 
Win32 Processing ServAppStarterApi.log = 126 MB for 1 time
 Snappy compress in 196.18ms, ratio=84%, 642.6 MB/s
 Snappy uncompress in 125.99ms, 1000.7 MB/s
 SynLZ compress in 210.89ms, ratio=89%, 597.8 MB/s
 SynLZ uncompress in 175.89ms, 716.8 MB/s
 SynZip1 compress in 825.56ms, ratio=89%, 152.7 MB/s
 SynZip1 uncompress in 298.19ms, 422.8 MB/s
 SynZip6 compress in 2.25s, ratio=92%, 55.8 MB/s
 SynZip6 uncompress in 255.32ms, 493.8 MB/s

We also use SynLZ for SQlite3 database file backup, with very high performance (much better than ZIP), and good enough compression ratio.

Then our LogView and SynDBExplorer tools are able to open such synlz-compressed file directly, with no speed penalty.

ab · 2016-08-14 08:18:14

About SynZip, we may investigate in direction of zlib forks by Intel or Cloudfare:
https://www.snellman.net/blog/archive/2 … tches.html

See https://github.com/cloudflare/zlib
and https://github.com/jtkukunas/zlib

Here are some preliminary numbers:

Processing DRAGONFLY1.log = 126 MB for 1 time

Win64 CloudFlare dll:
 SynZip1 compress in 446.49ms, ratio=88%, 282.3 MB/s
 SynZip1 uncompress in 250.76ms, 502.8 MB/s
 SynZip6 compress in 1.25s, ratio=92%, 100.1 MB/s
 SynZip6 uncompress in 206.34ms, 611 MB/s

Win64 Delphi's ZLib:
 SynZip1 compress in 1.88s, ratio=89%, 67 MB/s
 SynZip1 uncompress in 599.54ms, 210.3 MB/s
 SynZip6 compress in 4.16s, ratio=92%, 30.2 MB/s
 SynZip6 uncompress in 524.90ms, 240.2 MB/s

Win32 CloudFlare dll:
 SynZip1 compress in 497.60ms, ratio=89%, 253.3 MB/s
 SynZip1 uncompress in 276.20ms, 456.5 MB/s
 SynZip6 compress in 1.07s, ratio=92%, 117.1 MB/s
 SynZip6 uncompress in 246.43ms, 511.6 MB/s

Win32 SynZip:
 SynZip1 compress in 839.25ms, ratio=89%, 150.2 MB/s
 SynZip1 uncompress in 297.87ms, 423.2 MB/s
 SynZip6 compress in 2.30s, ratio=92%, 54.6 MB/s
 SynZip6 uncompress in 262.72ms, 479.9 MB/s

mORMot Open Source

#1 2016-08-04 23:05:16

SNAPPY delphi 64 port

#2 2016-08-05 06:57:56

Re: SNAPPY delphi 64 port

#3 2016-08-05 07:23:24

Re: SNAPPY delphi 64 port

#4 2016-08-05 09:56:08

Re: SNAPPY delphi 64 port

#5 2016-08-05 10:48:34

Re: SNAPPY delphi 64 port

#6 2016-08-05 12:28:17

Re: SNAPPY delphi 64 port

#7 2016-08-05 12:29:15

Re: SNAPPY delphi 64 port

#8 2016-08-05 12:45:36

Re: SNAPPY delphi 64 port

#9 2016-08-05 14:04:47

Re: SNAPPY delphi 64 port

#10 2016-08-05 14:24:54

Re: SNAPPY delphi 64 port

#11 2016-08-05 16:29:23

Re: SNAPPY delphi 64 port

#12 2016-08-05 16:40:24

Re: SNAPPY delphi 64 port

#13 2016-08-06 16:36:18

Re: SNAPPY delphi 64 port

#14 2016-08-06 16:55:37

Re: SNAPPY delphi 64 port

#15 2016-08-07 15:46:42

Re: SNAPPY delphi 64 port

#16 2016-08-07 21:02:55

Re: SNAPPY delphi 64 port

#17 2016-08-08 07:00:54

Re: SNAPPY delphi 64 port

#18 2016-08-10 15:06:22

Re: SNAPPY delphi 64 port

#19 2016-08-10 15:28:16

Re: SNAPPY delphi 64 port

#20 2016-08-10 22:08:35

Re: SNAPPY delphi 64 port

#21 2016-08-11 09:39:33

Re: SNAPPY delphi 64 port

#22 2016-08-11 11:40:20

Re: SNAPPY delphi 64 port

#23 2016-08-11 11:48:12

Re: SNAPPY delphi 64 port

#24 2016-08-11 12:09:43

Re: SNAPPY delphi 64 port

#25 2016-08-11 12:31:54

Re: SNAPPY delphi 64 port

#26 2016-08-11 22:15:31

Re: SNAPPY delphi 64 port

#27 2016-08-12 00:45:20

Re: SNAPPY delphi 64 port

#28 2016-08-12 07:17:37

Re: SNAPPY delphi 64 port

#29 2016-08-14 08:11:28

Re: SNAPPY delphi 64 port

#30 2016-08-14 08:18:14

Re: SNAPPY delphi 64 port

Board footer