mORMot Open Source

TPrami

Member

Registered: 2010-07-06

Posts: 119

XXH3_64bits() really flies

https://xxhash.com/doc/v0.8.2/index.html

-Tee-

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,654

Website

Re: XXH3_64bits() really flies

Not sure it is a fair comparison.
For instance, they use non-parallel version of crc32c.

On my Core i5, here are some numbers of the diverse built-in in mORMot 2:

     2500 crc32c in 157us i.e. 15.1M/s or 33 GB/s
     2500 xxhash32 in 590us i.e. 4M/s or 8.8 GB/s
     2500 crc32 in 157us i.e. 15.1M/s or 33 GB/s
     2500 adler32 in 152us i.e. 15.6M/s or 34.1 GB/s
     2500 hash32 in 326us i.e. 7.3M/s or 15.9 GB/s
     2500 aesnihash in 93us i.e. 25.6M/s or 55.8 GB/s
     2500 md4 in 5.45ms i.e. 447.8K/s or 0.9 GB/s
     2500 md5 in 6ms i.e. 406.6K/s or 886.1 MB/s
     2500 sha1 in 2.27ms i.e. 1M/s or 2.2 GB/s
     2500 hmacsha1 in 2.53ms i.e. 0.9M/s or 2 GB/s
     2500 sha256 in 2.48ms i.e. 0.9M/s or 2 GB/s
     2500 hmacsha256 in 2.76ms i.e. 883.9K/s or 1.8 GB/s
     2500 sha384 in 7.31ms i.e. 333.8K/s or 727.4 MB/s
     2500 hmacsha384 in 8.62ms i.e. 283K/s or 616.8 MB/s
     2500 sha512 in 7.31ms i.e. 333.6K/s or 727 MB/s
     2500 sha512_256 in 7.31ms i.e. 333.7K/s or 727.2 MB/s
     2500 hmacsha512 in 8.62ms i.e. 283.1K/s or 617.1 MB/s
     2500 sha3_256 in 10.79ms i.e. 226.1K/s or 492.7 MB/s
     2500 sha3_512 in 19.68ms i.e. 124K/s or 270.3 MB/s

As you can see, our aesnihash reaches 55.8 GB/s on a Core i5 with a proven hash algorithm taken from the golang runtime.
And for hashing data in memory, it has the advantage (requirement?) of being secure against hash flooding http://ocert.org/advisories/ocert-2012-001.html

Note that sha1/sha256 numbers are very high on my PC because its CPU features SHA-NI.
And note that our xxhash32() version is a pretty simple implementation without any unrolling. But is gives a test of our Core i5 raw speed in respect to the Core i7 used for the tests on your links.

TPrami

Member

Registered: 2010-07-06

Posts: 119

Re: XXH3_64bits() really flies

Hard to tell are those parallel or non-parallel versions. What they publish.

SSE2 and New AVX (what ever it was) optimizations give quite nice boost.

     2500 crc32c in 157us i.e. 15.1M/s or 33 GB/s

Don't understand those values, 15M/s or 3GB/s, would need header for table/and table

-Tee-

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,654

Website

Re: XXH3_64bits() really flies

15M/s = 15 millions of hashes per seconds = per call speed
3GB/s = 33 GigaBytes per second = per data size speed
Our benchmark run on several data sizes, so the absolute performance is likely to be higher.

I don't see any reason to change our AesNiHash with those XXH3_64 which are pretty much experimental and likely to be slower.

TPrami

Member

Registered: 2010-07-06

Posts: 119

Re: XXH3_64bits() really flies

15M/s = 15 millions of hashes per seconds = per call speed
3GB/s = 33 GigaBytes per second = per data size speed

Ah, makes sense, thanks.

-Tee-

okoba

Member

Registered: 2019-09-29

Posts: 118

Re: XXH3_64bits() really flies

I don't see any reason to change our AesNiHash with those XXH3_64 which are pretty much experimental and likely to be slower.

Why experimental?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,654

Website

Re: XXH3_64bits() really flies

Where is it used on production, e.g. as hash function for maps, large scale?

It is not because it is reported as fast that it has proven to be good enough for production use, e.g. with a hash table.
Hashing is a very complex task. Our AesNiHash() is not experimental at all, because we reused the golang algorithm.

okoba

Member

Registered: 2019-09-29

Posts: 118

Re: XXH3_64bits() really flies

In https://xxhash.com/ footer there is a list of clients.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,654

Website

Re: XXH3_64bits() really flies

LIst of implementation libraries, but probably not using them as we need it in our framework.

The golang AesNiHash algorithm has been proven to be used as hash table source for years on ALL golang projects, which is much more than those few usage.
The golang AesNiHash algorithm has built-in protection against hash flooding, which xxHash64 does not.
The golang AesNiHash algorithm is faster than xxHash64 on our main productive target, i.e. Intel 32-bit or 64-bit (unless proven the contrary).

So why change?

okoba

Member

Registered: 2019-09-29

Posts: 118

Re: XXH3_64bits() really flies

No reason to change for mORMot I think. But interesting hash nevertheless.