mORMot Open Source

enrcas

Member

Registered: 2013-06-16

Posts: 20

Adding CRC32 to TSynHasher

Hi!

I am doing some hashing here, so I found the very nice function HashFile.

It seems TSynHasher can't do CRC32, as I need to use it, I want to ask as I've seen in sources, if I want to add CRC32 support to tSynHasher, all I need to do
is add the algo in THashAlgos, and update the functions in TSynHasher, Init, Update and Final ?

It seems easy, but cannot find as can I call the CRC32 init, update, etc... functions (in Syncommons)

I am currently using Indy10 hashing functions, but I found them slow, I want to do some benchmark here.
As I am coding a little tool to calc CRC32, MD5 and SHA1 hashes of files and adding them to an array, this is 99% of cpu, so I want to be as fast as possible.

In my tests, a basic 400MB set of files, with Ind10 I got around 21s total time, and just changing the SHA1 calculation to Mormot is reduced to 7s!
Possibly using CRC32 with Mormot I can get it faster (even when i know SHA1 is much more complex and so the savings)

Thanks!

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,669

Website

Re: Adding CRC32 to TSynHasher

Plain crc32 is in SynZip, not SynCrypto.

For performance, crc32c is the way to go on newer processors.
Or xxhash32.

None of those hashes are cryptography aware.
Even Md5 or Sha1 have collisions.
You may be surprised by the performance of our Sha256 or Sha512 or Sha3, especially on x86_64.

enrcas

Member

Registered: 2013-06-16

Posts: 20

Re: Adding CRC32 to TSynHasher

I am hashing files, not related to crypto. As I need to compare files with already calculated hashes in a text file (kind of .sfv files).
I just need CRC32, SHA1 (and possibly MD5).

I was reading syncommons because I found tons of crc32 use there, but will look in synzip and see if I can add it to Tsynhasher.

Thanks.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,669

Website

Re: Adding CRC32 to TSynHasher

Note that there are several crc32 variants, with several polynomials.

The most known "crc32" is the one used by zip/deflate, as implemented in SynZip.pas - but not available in SynCrypto.pas THasher.
It is not included in THasher wrapper since it would add a dependency to SynZip.pas, which we would like to avoid.

Internally in mORMot, we use the "crc32c", which has the same algorithm, but a diverse polynomial (the one from SCSI standard IIRC), therefore gives diverse hash values (with the same accuracy).
The benefit of this crc32c is that it has hardware accelleration on latest Intel and AMD CPUs, so resulting speed on x86_64 is more than 20GB/s. The fastest hash available, for sure.

enrcas

Member

Registered: 2013-06-16

Posts: 20

Re: Adding CRC32 to TSynHasher

Ok, thanks for the information, I need the "classic" crc32, so I think it will be easier to use my own crc32 then, Indy uses tfilestream. Maybe I could use the FileOpenSequentialRead as you used in hashfile, and your implementation of classic crc32.

Thanks again for such great library.

Last edited by enrcas (2018-03-31 20:04:18)