mORMot Open Source

peterme

Member

Registered: 2013-02-13

Posts: 5

SynLZ ... possible to decompress only beginning of a file?

Hello!

Is it possible to SynLZdecompress only the first part of my file
opened eg. as TFileStream followed by a Read of the first hundreds of bytes?

The use case: i did  compress a lot amount of data using synlz (very fast, wow!)
and try to peek within them very quickly for to read some kind of metadata
that i stored at the beginning of those files.

Trying to achieve that with modifications of the size parameter of the
SynLZdecompress functions constantly did lead to exceptions.
Can somewhere be found a sample?

Many thanks!
Pet

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,224

Website

Re: SynLZ ... possible to decompress only beginning of a file?

Why not just compress the header by itself?

When you have huge amount of data, using some "windowing" of the content does make sense.
Average compression ratio won't be affected, and you are able to start uncompressing at any position, per chunk.

But I can do some decompression function with a maximum size.
Create a ticket for this, if not need it.

peterme

Member

Registered: 2013-02-13

Posts: 5

Re: SynLZ ... possible to decompress only beginning of a file?

Many thanks ab!

I thought about that (embedding a compressed item containing it's own size information) but was not successful  ...

Yes, i would appreciate very much a ticket for something like an optional resp. additional restriction / max parameter
(decompress until this value if given, else use size).
By that, it would not be needed to loadfromstream ... decompress 100 MB (uncompressed) only for to have a very
few meta data available ....  that might be used, for instance, for a list view for those files.
Eg. date information, comments, general contents information and so on.

Imagine you need to do that for 30 files, each 100 MB (uncompressed) in size ...
Very useful to have such a possibility!

Besides: i was absolutely surprised, impressed by the speed, at least regarding
compression (whereas decompression was somehow near to zlib).

Pet

peterme

Member

Registered: 2013-02-13

Posts: 5

Re: SynLZ ... possible to decompress only beginning of a file?

So i raised a ticked, UUID :     82ca067959f305af549cfee81b24a37ebdb6258d

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,224

Website

Re: SynLZ ... possible to decompress only beginning of a file?

I've added SynLZdecompress1partial() function for partial and secure (but slower) decompression - with corresponding regression tests.

This function is more secure, since it will avoid any buffer overflow error. But it will be slower than the other methods (and no asm version is available).

Tested with Win32 + Win64.

See http://synopse.info/fossil/info/69cb5f60ec and your ticket http://synopse.info/fossil/info/82ca067959

peterme

Member

Registered: 2013-02-13

Posts: 5

Re: SynLZ ... possible to decompress only beginning of a file?

Tried it and it works fine, just as expected :-)

(it doesn't play a role that it is slower or that there will be no asm version availale,
when decompressing only a small chunk of data such differences will not be noticeable).

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,224

Website

Re: SynLZ ... possible to decompress only beginning of a file?

Yes, a little speed degradation is not a problem.
I did not make any precise benchmark, but I expect only a 10-20% speed degradation, when compared to the asm optimized version, and somewhat around 5% when compared to pure pascal.

It is still very fast, in all cases.

peterme

Member

Registered: 2013-02-13

Posts: 5

Re: SynLZ ... possible to decompress only beginning of a file?

Many thanks for your extremely quick solution ab!