OpenSSL 1.1 wrapper

mpv · 2020-02-22 20:39:26

I partially finish a OpenSSL1.1 wrapper and place it into feature/openSSL11 branch. I plane to add Cipher/Decipher and after this it will be ready to be merged into trunk (if @ab approve)

My goal for this work is to add a crypto module into SyNode, but as a side effect it allow to:
- use all crypto algorithms from OpenSS( 21 algorithm on my linux)
- use hardware tokens, hardware security modules
- support government certified engines like GOST (Russia), BEE2EVP (Belarusian), DSTU (Ukrainian), FIPS certified modules, etc.

For a while I finish THmac and THash classes. Speed test results is in sample 38 readme

For small data sizes (100 bytes) SynCrypto classes made by @ab is faster on both platforms. My congratulations to @ab - it's very cool to be faster when so known library as OpenSSL.

For data sizes > 100 OpenSSL is faster (at last compared to SynCrypto compiled using FPC). But IMHO here speed is not a main thing, but algorithms/engines supported by OpenSSL

Any feedback is welcome!

P.S.

//SHAKE256 XOF hash with custom result length
writeln(THash.hash('SHAKE256', data, 64)

Last edited by mpv (2020-02-22 20:45:36)

urhen · 2020-02-23 12:22:12

You missed the most important thing of OpenSSL 1.1.1 - TLS v1.3

TLSv1_3_method: function: PSSL_METHOD; cdecl;

and also

TLS_method: function: PSSL_METHOD; cdecl;

Otherwise very good work!

Unfortunately mORMot doesn't have TCP/UDP sockets + HTTP support with gzip because then I could get rid of Indy -.- mORMot is well maintained compared to some other libs...

Last edited by urhen (2020-02-23 12:25:07)

mpv · 2020-02-23 14:10:47

Yes, Chiper/Decipher/DiffmiHelmah (building blocks of TLS) is on the roadmap.

Unfortunately mORMot doesn't have TCP/UDP sockets + HTTP support with gzip

mORMot supports HTTP level compression out of the box - for server-side see THTTPServerGeneric.RegisterCompress. TLS is also supported for HTTP.SYS based server (Windows)

Client (requests to other resources) also supports HTTP/HTTPS/compression out of the box (both TWinHTTP & TCurlHTTP)

The only missing thing is TLS for socket based server. But in this case we highly recommend to use a reverse proxy with TLS termination. We use nginx with grate success (even in case our server hosted on windows we use nginx for security reasons - Windows itself is a big security hole unfortunately )

urhen · 2020-02-23 20:36:14

mpv wrote:

Yes, Chiper/Decipher/DiffmiHelmah (building blocks of TLS) is on the roadmap.
Unfortunately mORMot doesn't have TCP/UDP sockets + HTTP support with gzip
Client (requests to other resources) also supports HTTP/HTTPS/compression out of the box (both TWinHTTP & TCurlHTTP)

I was focussing on the client-side and meant something like Indy's TIdTCPClient which uses the platform APIs so that you don't need to include 3rd party libraries.
I know that mORMot's goal in something different (ORM!) but would be a nice to have to create TCP connection with TLS support - then I can completely replace Indy with mORMot

Last edited by urhen (2020-02-23 20:37:49)

Vitaly · 2020-02-24 04:32:36

urhen wrote:

I was focussing on the client-side and meant something like Indy's TIdTCPClient which uses the platform APIs so that you don't need to include 3rd party libraries.

Could you help me here to understand, please?
I'm using SynCrtSock.TWinHTTP as a universal TCP-client to get rid of using native Delphi and Indy clients (in handier for me way). Almost all my current projects, where I have to communicate with third-party services, are using it already. And these third-party services are often built on very different communication and security schemes.
So far everything works fine, but should I be aware of something for the future?

ab · 2020-02-24 07:16:20

TLS support via OpenSSL may be a plus to get rid of the libcurl dependency on POSIX - even if libcurl is very likely to be installed, and is a very proven library.
And it would allow TLS encryption e.g. for SynMongoDB connection - as with Windows uses the SChannel API.

Note that under Windows, there are 3 ways of using TLS/HTTPS in SynCrtSock.pas:
- TWinINet - deprecated
- TWinHttp - just fine
- TCrtSock with aTLS=true, using SChannel API - this last one was not quoted here, but is nice because it requires no 3rd party dependency since it is built-in in the Windows system.

@mpv
OpenSSL TLS implementation should follow the same wrapper than SChannel API.
See fSecure: TSChannelClient member in the TCrtSocket class.

mpv · 2020-02-24 10:07:26

For a while I do not plane to implement a TLS - only hash/hmac/encription/keys (pem/der/pfx).
The reason for this - for a hi load services (in my case ~ 1000 RPS average rate, sometimes more) TLS implementation should include ssl session cache, session tickets, OCSP Stapling and I think many more features I even don`t know about. My current knowledge level do not allow me to implement a TLS correctly

For a while my personal choice is libcurl for client and TLS termination on reverse proxy for server. Two years ago a switch to this stack and its works perfectly

ab · 2020-02-24 11:32:40

Yes, but I guess the only currently limitation for POSIX/FPC is that we don't support TLS when connecting to a MongoDB server.
I understand you don't use this NoSQL DB on your end.

urhen · 2020-02-24 14:08:25

ab wrote:

TLS support via OpenSSL may be a plus to get rid of the libcurl dependency on POSIX
Note that under Windows, there are 3 ways of using TLS/HTTPS in SynCrtSock.pas:
- TWinINet - deprecated
- TWinHttp - just fine
- TCrtSock with aTLS=true, using SChannel API - this last one was not quoted here, but is nice because it requires no 3rd party dependency since it is built-in in the Windows system.

So for unix there is only TCrtSocket with libcurl and without TLS?
Are there any examples on how to create a client TCP connection to a server?

ab · 2020-02-24 17:49:27

Plain TCP/UDP connections can indeed use TCrtSocket on any platform.

See e.g. how TDDDSynCrtSocket use it to implement an abstract IDDDSocket interface.
In unit dddInfraApps.

squirrel · 2020-07-17 11:22:35

Is it possible to use TLS v1.3 on tcp socket connections with mormot or this wrapper?

mpv · 2020-07-17 21:33:46

Its not implemented. For server side under Linux I consider to use reverse proxy for SSL termination, under Windows TLS is supported by HTTP.SYS (1.3 can be turned on by adding some regisrty key).
For client side under linux libcurl wrapper works perfectly, under Windows either libcurl or winhttp

ab · 2021-02-06 08:42:48

I have started OpenSSL 1.1.1 units for mORMot 2.
Check the latest commits in https://github.com/synopse/mORMot2/commits/master

Now we have an OpenSSL 1.1.1 raw access into mormot.lib.openssl11.pas.
With optional full API (huge) if needed - in an .inc file.

And basic PRNG and AES support in mormot.core.crypto.openssl.pas.
In fact, I was surprised that our stand-alone mormot.core.crypto.pas is slightly faster than OpenSSL for most algorithms, with the exception of TAesCtrOsl and TAesGcmOsl which are parallelized on OpenSSL so are faster.
You have some numbers on comments in https://github.com/synopse/mORMot2/blob … penssl.pas classes documentation.
I will eventually look into making some dedicated x86_64 asm for our TAesCtr to have parallelized process.

We had to make a huge refactoring of mormot.core.crypto.pas internals: still some work to do.
Also introduced a lot more of tests, including reference vectors and validation of our mormot.core.crypto library against OpenSSL - all green.

The next step is TLS support.
This is where an external and proven library is worth it, since TLS is very complex, and we can't easily implement it in stand-alone pascal, with full safety.
We will integrate it into mormot.net.core.pas so that we could have TCP/HTTP support of TLS - i.e. HTTPS support - with no dependency to libcurl. It will be mandatory e.g. for safe mormot.db.nosql.mongodb TLS connection.

On Windows, I don't advice using OpenSSL: we already have SChannel support, which is part of the OS. And the OpenSSL libraries are just awful to locate and execute.
On Linux/POSIX systems, OpenSSL is most of the time available.

mpv · 2021-02-06 09:13:43

@ab, is it possible to move a THmac and THash classes from my patch ( https://github.com/synopse/mORMot/compa … /openSSL11 ) to the new openssl class? IMHO it very simple to use ( inspired by nodejs) and, the most important - can calculate any hash algorithm.

The same architecture is good to have for encryption, becasuse currently we have one class per algo, what is bad. Many countries have his own plugable modules for ooenssl and his own algos.

About adding a TLS support, take into account what release of OpenSSL3 is expected in few month. May be we should wait for it, or at last use interfaces adeed by v1.1.1 for TLS, seams it will be the same in 3

Last edited by mpv (2021-02-06 09:14:44)

ab · 2021-02-06 13:56:09

The idea is to be able to create one class instance for the algo you expect.
See for instance High-Level TSynSigner/TSynHasher Multi-Algorithm Wrappers from mormot.core.secure.pas.
I am indeed thinking of something even more general...

OpenSSL 3 will be very close to OpenSSL 1.1.1, and the version 1.1.1 is the current LTS branch.
When OpenSSL 3 will be the next LTS branch, we will switch to it.
We try to use the most generic way of using OpenSSL - for instance, we get the ciphers or digests by name, not by function. It is the preferred way in latest revisions, IIRC.

mpv · 2021-02-06 15:46:36

Yes, the idea is a single class for all algorithm. TSynSigner/TSynHasher is very close to it BUT they limited to certain list of algorithms (THashAlgo enum).
While in the OpenSSL new algorithms can be added by plug-able modules.
This is why in my pull request THash class accept algorithm as a string and support 21 algo ( available algos can be obtained using `openssl list -digest-algorithms` command)

Even if we do not spoke about country-specific algorithms, like GOST (Russia, Kazakhstan), BEE2EVP (Belarusian), DSTU (Ukrainian) etc. in enterprise/government apps even for AES/DES/etc in some countries we need to use a FIPS certified plugable modules, so limiting to THashAlgo enum is a big limitation . The same is for ciphers.

Last edited by mpv (2021-02-06 15:48:27)

ab · 2021-02-06 23:19:34

Yes, you are right, of course.

We used classes because we implemented the process. So we needed the classes there: one class per implementation.
Some libraries use one class for cipher, then apply the chaining in a parent code. In practice, this doesn't allow to make very tuned implementation as we did.

I will add a more general - and OpenSSL compatible - API for mormot.core.crypto.pas for ciphers, hashers and signature.

ab · 2021-02-20 18:35:06

I just committed OpenSSL support for TLS/HTTPS client.

It is located in mormot.lib.openssl.pas so it is very lightweight and add no unit dependency if all that you need is a small TLS/TCP connection.
From my tests, it is running just fine.
It has some advanced options, e.g. for private/public certificates, and also detailed peer and cipher info if needed in TNetTLSContext - as TCrtSocket.TLS new field:

  with THttpClientSocket.Create do
  try
    TLS.WithPeerInfo := true;
    // TLS.IgnoreCertificateErrors := true;
    // TLS.CipherList := 'ECDHE-RSA-CHACHA20-POLY1305';
    OpenBind('synopse.info', '443', {bind=}false, {tls=}true);
    writeln(TLS.PeerInfo);
    writeln(TLS.CipherName);
    writeln(Get('/forum/', 1000), ' len=', ContentLength);
    writeln(Get('/fossil/wiki/Synopse+OpenSource', 1000), ' len=', ContentLength);
  finally
    Free;
  end;

I also tested again the existing SChannel layer, and I got unexpected EEC_E_INVALID_TOKEN error from time to time during SChannel handshake on the latest Windows 10 build. For both mORMot 1.18 and mORMot 2 (so it is not a mORMot 2 regression).
After some time searching the Internet, I didn't find anything wrong with our code, which follows the samples I found.
There was no answer to similar problems with libcurl https://github.com/jeroen/curl/issues/127
So I guess OpenSSL should be used also on Windows, for safety, if you need a trusted TLS layer!

ab · 2021-02-21 11:13:42

Update:

I have finally fixed the SChannel implementation on both mORMot 1.18 and mORMot 2.
- no random SEC_E_INVALID_TOKEN any more !
- and mORMot 2 TLS.IgnoreCertificateErrors is supported now.

okoba · 2021-02-22 21:31:23

Well written post. Just one question I like to know more. Should I use OpenSSL (Linux and Windows) for ultimate security or let mORMot choose the best as default?
I understood the speed part, mORMot is smart in that, my question is about the security part in both client and sever usage in both OS.

Last edited by okoba (2021-02-22 21:33:26)

ab · 2021-02-23 14:55:21

OpenSSL had big security flaws in the past, but is sometimes required because it has been audited and you can instance a certified OpenSSL version if needed.
mORMot has been audited internally by some companies, with no big flaw found - and it is less used, so unknown by most attackers.

On Linux, I don't see any reasons not to include OpenSSL on server side, if it is part of the system.
On Windows, I tend not to use OpenSSL because of the dll Hell: you may have unexpected/difficult problems in the field, depending on the computer used.

Anyway, if the OpenSSL version is not at least 1.1.1, it won't load and fallback to mORMot's code, so it won't hurt.

ab · 2023-04-20 06:39:11

Note that since this post, we also implemented OpenSSL 3 support.

The unit is able to switch from OpenSSL 1.1.1 to OpenSSL 3.x at runtime, depending on the available dll/so.

We noted that OpenSSL 3 is really slower than OpenSSL 1.1.1.
For instance, the certificates reading is known to be much slower, and our mORMot2tests benchmark make OpenSSL 3 more than twice slower than OpenSSL 1.1.1 on encryption.
AFAICT, this is not due to the encryption engine itself, but it is due to all internal modifiations of the API and its structures.
See e.g. https://github.com/openssl/openssl/issues/16871

This is sad that some Linux systems switches to OpenSSL 3 by default, as it is the long-term-supported version.
For instance, on Debian testing, the mORMot CTR and GCM are much faster then OpenSSL:

- Benchmark: 112,500 assertions passed  182.55ms
     2500 mormot aes-128-cfb in 2.53ms i.e. 0.9M/s or 2 GB/s
     2500 mormot aes-128-ofb in 2.52ms i.e. 0.9M/s or 2 GB/s
     2500 mormot aes-128-c64 in 3.79ms i.e. 644.1K/s or 1.3 GB/s
     2500 mormot aes-128-ctr in 571us i.e. 4.1M/s or 9 GB/s
     2500 mormot aes-128-cfc in 2.68ms i.e. 908.2K/s or 1.9 GB/s
     2500 mormot aes-128-ofc in 3.02ms i.e. 807.6K/s or 1.7 GB/s
     2500 mormot aes-128-ctc in 913us i.e. 2.6M/s or 5.6 GB/s
     2500 mormot aes-128-gcm in 992us i.e. 2.4M/s or 5.2 GB/s
     2500 mormot aes-256-cfb in 3.39ms i.e. 720.1K/s or 1.5 GB/s
     2500 mormot aes-256-ofb in 3.39ms i.e. 720.1K/s or 1.5 GB/s
     2500 mormot aes-256-c64 in 4.66ms i.e. 523.2K/s or 1.1 GB/s
     2500 mormot aes-256-ctr in 707us i.e. 3.3M/s or 7.3 GB/s
     2500 mormot aes-256-cfc in 3.58ms i.e. 681.5K/s or 1.4 GB/s
     2500 mormot aes-256-ofc in 3.91ms i.e. 623.4K/s or 1.3 GB/s
     2500 mormot aes-256-ctc in 1.06ms i.e. 2.2M/s or 4.9 GB/s
     2500 mormot aes-256-gcm in 1.13ms i.e. 2.1M/s or 4.5 GB/s
     2500 openssl aes-128-cfb in 4.95ms i.e. 493K/s or 1 GB/s
     2500 openssl aes-128-ofb in 3.51ms i.e. 694.5K/s or 1.4 GB/s
     2500 openssl aes-128-ctr in 1.20ms i.e. 1.9M/s or 4.3 GB/s
     2500 openssl aes-128-gcm in 2.49ms i.e. 0.9M/s or 2 GB/s
     2500 openssl aes-256-cfb in 5.82ms i.e. 418.9K/s or 913 MB/s
     2500 openssl aes-256-ofb in 4.37ms i.e. 558K/s or 1.1 GB/s
     2500 openssl aes-256-ctr in 1.33ms i.e. 1.7M/s or 3.8 GB/s
     2500 openssl aes-256-gcm in 2.56ms i.e. 0.9M/s or 2 GB/s

It makes a significant performance drop in respect to OpenSSL 1.1.1.

I hope OpenSSL 3.1 is better - they stated the performances were better.
https://www.openssl.org/blog/blog/2023/ … .1Release/

On the other hand, it is a concern that Indy is still officially stuck in OpenSSL 1.0 branch... after years working on 1.1 support.

mORMot Open Source

#1 2020-02-22 20:39:26

OpenSSL 1.1 wrapper

#2 2020-02-23 12:22:12

Re: OpenSSL 1.1 wrapper

#3 2020-02-23 14:10:47

Re: OpenSSL 1.1 wrapper

#4 2020-02-23 20:36:14

Re: OpenSSL 1.1 wrapper

#5 2020-02-24 04:32:36

Re: OpenSSL 1.1 wrapper

#6 2020-02-24 07:16:20

Re: OpenSSL 1.1 wrapper

#7 2020-02-24 10:07:26

Re: OpenSSL 1.1 wrapper

#8 2020-02-24 11:32:40

Re: OpenSSL 1.1 wrapper

#9 2020-02-24 14:08:25

Re: OpenSSL 1.1 wrapper

#10 2020-02-24 17:49:27

Re: OpenSSL 1.1 wrapper

#11 2020-07-17 11:22:35

Re: OpenSSL 1.1 wrapper

#12 2020-07-17 21:33:46

Re: OpenSSL 1.1 wrapper

#13 2021-02-06 08:42:48

Re: OpenSSL 1.1 wrapper

#14 2021-02-06 09:13:43

Re: OpenSSL 1.1 wrapper

#15 2021-02-06 13:56:09

Re: OpenSSL 1.1 wrapper

#16 2021-02-06 15:46:36

Re: OpenSSL 1.1 wrapper

#17 2021-02-06 23:19:34

Re: OpenSSL 1.1 wrapper

#18 2021-02-20 18:35:06

Re: OpenSSL 1.1 wrapper

#19 2021-02-21 11:13:42

Re: OpenSSL 1.1 wrapper

#20 2021-02-22 21:31:23

Re: OpenSSL 1.1 wrapper

#21 2021-02-23 14:55:21

Re: OpenSSL 1.1 wrapper

#22 2023-04-20 06:39:11

Re: OpenSSL 1.1 wrapper

Board footer