mORMot Open Source

M0V

Member

Registered: 2015-01-27

Posts: 10

Implementing progress event ?

Good day to you Ab!

I need something to display progress of AES256 Encryption/Decryption and for LZ/LZO Compression/Decompression
witch way is the best to do this ?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Implementing progress event ?

There is currently no such feature.

You may divide the content in chunks, then AES+compress the chunks, showing a progression bar.

IMHO the AES+SynLZ process is not worth a progression, when run from memory.
The speed is very high.
What is slow is the disk access, I guess. This is where you put your progression display.

M0V

Member

Registered: 2015-01-27

Posts: 10

Re: Implementing progress event ?

There is currently no such feature.

You may divide the content in chunks, then AES+compress the chunks, showing a progression bar.

IMHO the AES+SynLZ process is not worth a progression, when run from memory.
The speed is very high.
What is slow is the disk access, I guess. This is where you put your progression display.

You mean divide my content that need to be processed ? I think it will increase out data size more than if process it as single, right ?

Any way It will be nice to see this feature, because some times data is very large and we need to show that something is happening, mostly actual if compression+encryption process takes >= 10 sec.

For now I'm using LZ4 that has progress event (and it really fast) + slow AES256 that has progress too.
I would like add to my project synLZ, synLZO, synZIP compressions and fast synAES256, but can't do this for reason that they has no progress.

I looked at the source of syn'S trying implement progress and understood that I don't understand it )) I don't know asm ...

So I hope, you implement this ... may be in future ...

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Implementing progress event ?

Dividing into chunks won't change much the resulting data size.
If the chunk is about 1MB, it would definitively be bigger than the LZ window itself.
You may loose some bytes, perhaps 0.5% I guess.
If you let the content be in the CPU L3 cache (e.g. set the chunk size to 4MB, for a 8MB L3 cache size for Core i7), and run AES+SynLZ on each chunk, you would have very high speed.

Are you processing in-memory buffers, or streams?

M0V

Member

Registered: 2015-01-27

Posts: 10

Re: Implementing progress event ?

Are you processing in-memory buffers, or streams?

in memory, then result must be encoded with Base4O96 algorithm to be as unicode text, that's why I need size economy )
I think that chunking will result same size as default LZ4, and if yes then it make useless use of SynLZ (

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Implementing progress event ?

SynLZ compression speed is higher than LZ4 AFAIR.

M0V

Member

Registered: 2015-01-27

Posts: 10

Re: Implementing progress event ?

RE: Are you processing in-memory buffers, or streams?
Mistake, processing Streams.

RE: SynLZ compression speed is higher than LZ4 AFAIR
Yes, but if we need to compress 100Mb or more we must show to user some thing like progress

I think to try make progress event with non-blocking callback function like this one https://jonlennartaasenden.wordpress.co … i-updated/   but still not sure, I can't understand where to put execution of it  ))

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Implementing progress event ?

If you are processing streams, you need to use chunks for SynLZ, since the current implementation only rely on memory buffers.

Each chunk could be compressed + AES encrypted + CRC32C.

Then it is easy to show the progress state for each chunk.