#1 2023-09-12 10:36:28

okoba
Member
Registered: 2019-09-29
Posts: 106

ZSTD support

Currently mORMot supports many compression algorithm, and I wanted to ask if there is a plan for ZSTD support?

Offline

#2 2023-09-12 11:54:31

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,316
Website

Re: ZSTD support

Nor for mORMot 1, perhaps for mORMot 2.

In practice, libdeflate or liblizard are good enough for most use cases.

Offline

#3 2023-09-12 12:06:59

okoba
Member
Registered: 2019-09-29
Posts: 106

Re: ZSTD support

Yes I benchmarked all compression library available in mORMot and ZSTD was faster and had better compression. libdeflate is great, but ZSTD can do better.
Although I agree, libdeflate is the good way especially for server needs. But for custom client and server codes, ZSTD can be the better choice.
I could take care of adding ZSTD to mORMot 2 in the current TAlgoCompress style, but the problem is mostly managing to prepare the best static libraries (including custom emmory manger like you done for SQLite and others, or shared C Lib codes). I don't know C and building their libraries is just a pain.
If thats possible for you to prepare that, I can write the code for the current official dll.

Offline

#4 2023-09-12 12:12:51

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,316
Website

Re: ZSTD support

I am no C expert but I could do try to compile the code as we do for SQlite3.

My guess is that https://github.com/YWtheGod/ZSTD4Delphi … d4delphi.c may help as a startup main source.

Offline

#5 2023-09-12 12:40:23

okoba
Member
Registered: 2019-09-29
Posts: 106

Re: ZSTD support

Oh believe me, compare to me, you are a god of C smile
I will look into it.

Offline

#6 2023-09-13 06:05:58

TPrami
Member
Registered: 2010-07-06
Posts: 114

Re: ZSTD support

Also found this:

https://github.com/DenisAnisimov/ZSTD.pas

There is also LZ4 implementation.

Have no idea of the quality tough.

Both are interesting compression algorithms for sure.

Offline

#7 2023-09-13 07:22:37

igors233
Member
Registered: 2012-09-10
Posts: 235

Re: ZSTD support

Since 7z is already supported I think it would be better to check if 7z ZS is supported as well.
7z ZS supports ZTSD, brotli and other codecs and so it would give more options to users.
Brotli is also becoming more and more supported by webservers or API providers.

Offline

#8 2023-09-13 10:23:07

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,316
Website

Re: ZSTD support

7z is working about files, while we need something working on memory buffers.

Offline

#9 2023-09-13 10:53:45

igors233
Member
Registered: 2012-09-10
Posts: 235

Re: ZSTD support

7z by default works on buffers/streams (IInStream, IOutStream) which caller decides how to implement, as file stream or memory stream.

Last edited by igors233 (2023-09-13 10:53:59)

Offline

#10 2023-09-13 11:00:12

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,316
Website

Re: ZSTD support

7z works via streams, but over archive files, not raw memory buffers.

And the 7z library we support is a dll for Windows only.

Offline

#11 2023-09-13 11:20:36

igors233
Member
Registered: 2012-09-10
Posts: 235

Re: ZSTD support

ArchiveFile is also defined as working with IInStream (see IArchiveOpenCallback) so 7z archive can fully be in memory.
But I understand what you mean, overhead of having simple buffer treatead as archive instead of compressed buffer, I think that overhead isn't large and is worth having any compression supported.

> And the 7z library we support is a dll for Windows only.

7z does have debian/linux implementation, though I do not know how compatible it is or if 7z ZS has linux support.

BTW there's also LZMA2 compression (default 7z algo), InnoSetup has Pascal implementation of it and there are some other implementation but I don't know if they're cross platform.

Offline

#12 2023-09-13 15:12:11

tbo
Member
Registered: 2015-04-20
Posts: 337

Re: ZSTD support

Here are interesting benchmarks from Tommi on the subject too.

With best regards
Thomas

Offline

#13 2023-09-13 17:07:17

igors233
Member
Registered: 2012-09-10
Posts: 235

Re: ZSTD support

> Here are interesting benchmarks from Tommi on the subject too.

I've read somewhere that comparing speeds with different settings is wrong (apples/oranges) so all test should be done using same window sizes.

Following link has that test (brotli is faster for compression, zstd for decompression):
https://peazip.github.io/fast-compressi … ndard.html

Search for table marked: Fixed 128 MB window size for both algorithms

Last edited by igors233 (2023-09-13 17:15:57)

Offline

#14 2023-09-13 18:52:39

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,316
Website

Re: ZSTD support

You need to put libdeflate numbers into the comparisons.
It reaches very good ratio and performance, with plain zlib compatibility.

Offline

Board footer

Powered by FluxBB