mORMot Open Source

sjerinic

Member

Registered: 2013-02-11

Posts: 51

Encryption/Decription

Hi!

I need to encrypt RESTFUL communication.

My idea was on client side encrypt data like in sample:
      fTableJSON := FServiceStandard.GetAllConfigData(CryptDataUTF8(JSONEncode(['idAgency', AgencyID]), CrKeyIn));
and on server side decrypt data in GetAllConfigData function:
      fParamsJSON := DecryptDataUTF8(inputData, CrKeyIn);
      JSONDecode(fParamsJSON, ['idAgency'], Values, true);
...

Problem is when Server side receive RawUTF8 string value which contains byte 0 (zero). In that case received byte is wrong cause it has just bytes before zero byte.
For encrypt/decrypt data I use AESSHA256 function.
Do you have some suggestion how I can fix that problem or use encrypted server-client communication without SSL?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Encryption/Decription

I've just added AES encryption over HTTP.
See http://blog.synopse.info/post/2014/01/0 … -over-HTTP

In addition to regular HTTPS flow encryption, which is not easy to setup due to the needed certificates, mORMot proposes a proprietary encryption scheme. It is based on SHA-256 and AES-256/CTR algorithms, so is known to be secure. You do not need to setup anything on the server or the client configuration, just run the TSQLHttpClient and TSQLHttpServer classes with the corresponding parameters.

Note that this encryption uses a global key for the whole process, which should match on both Server and Client sides. You should better hard-code this public key in your Client and Server Delphi applications, with some variants depending on each end-user service. You can use CompressShaAesSetKey() as defined in SynCrypto.pas to set globally this Encryption Key, and an optional Initialization Vector. You can even customize the AES chaining mode, if the default TAESCTR mode is not what you expect.

When the aHttpServerSecurity parameter is set to secSynShaAes for the TSQLHttpServer.Create() constructor, this proprietary encryption will be enabled on the server side. For instance:

MyServer := TSQLHttpServer.Create('888',[DataBase],'+',useHttpApiRegisteringURI,32,secSynShaAes);

On the client side, you can just set the TSQLHttpClientGeneric.Compression property as expected:

MyClient.Compression := [hcSynShaAes];

Once those parameters have been set, a new proprietary encoding will be defined in the HTTP headers:

ACCEPT-ENCODING: synshaaes

Then all HTTP body content will be compressed via our SynLZ algorithm, and encoded using the very secure AES-CTR/256 encryption.

Since it is a proprietary algorithm, it will work only for Delphi clients. When accessing for a plain AJAX client, or a Delphi application with TSQLHttpClientGeneric.Compression = [], there won't be any encryption at all, due to way HTTP accepts its encoding. For safety, you should therefore use it in conjunction with our per-URI Authentication.

The corresponding commit is http://synopse.info/fossil/info/f9b04a46c8

sjerinic

Member

Registered: 2013-02-11

Posts: 51

Re: Encryption/Decription

Thanks AB!

Everything is working perfect, also implementation is simple. Great job!

I have one question.
In line: "MyServer := TSQLHttpServer.Create('888',[DataBase],'+',useHttpApiRegisteringURI,32,secSynShaAes);" I sow value 32 which is ServerThreadPoolCount. In documentation explanation for that field is:
    // - the ServerThreadPoolCount parameter will set the number of threads
    // to be initialized to handle incoming connections (default is 32, which
    // may be sufficient for most cases, maximum is 256)
Is that mean 32 maximal connection if I use sicClientDriven in ServiceRegister?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Encryption/Decription

Is that mean 32 maximal connection if I use sicClientDriven in ServiceRegister?

No, it means 32 threads in the thread pool for the process.
Should be enough to use 32 CPU cores at once!

You can have thousands of connections without any problem, even with less threads in the pool.
See http://blog.synopse.info/post/2013/09/1 … -of-mORMot which shows how 50000 client concurrent connections were made on the same mORMot server, without any problem.

sjerinic

Member

Registered: 2013-02-11

Posts: 51

Re: Encryption/Decription

Thanks again!

sjerinic

Member

Registered: 2013-02-11

Posts: 51

Re: Encryption/Decription

Hi AB!

After test with Wireshark I found that communication Client-Server is not encrypted or have a problem.
For Example:

POST /root/RemoteVirtualShopServerStandard.MakeBill/9?session_signature=124B2EC4001A14F67AA2048C HTTP/1.1
Cache-Control: no-cache
Connection: Keep-Alive
Pragma: no-cache
Content-Type: application/json; charset=UTF-8
Accept: */*
Accept-Encoding: synshaaes
User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows; Synopse mORMot 1.18 TWinHTTP)
Content-Length: 61
Host: demo.elbet.rs:213
[2,-1,47.62,1,1,1,"|01|02|03|04|05|06|",0,0,0,"",0,"",0,0,""]

The last row is what we send from client to server.
Server's answer looking OK:

Content-Type: application/json; charset=UTF-8
Content-Encoding: synshaaes
Server: Microsoft-HTTPAPI/2.0
X-Powered-By: SynCrtSock http://synopse.info
Server-InternalState: 0
Date: Mon, 06 Jan 2014 22:09:16 GMT
Content-Length: 640
....[.|4!..c....U..}'..[.1....b...2..c......#....g...LZO...fv.B...3.>@*...>.eR....8F=Sp.*.)..C..0...Q..c..S....w.q.).p.E.G.u.L.;`.(....zv.~...\.e.*..7.TL..e..a-H[..!...z?~....`..(.:Io....Z.P...n.L.D........6_.}..*w..{.A..n.........#.GxF.?.S.:....0.....e..........Z.oE.]/...Ed..Uv;M_|>Rz.[.'.Z-....bh....p...>.k..b.#d....._....r....ngK0Z).T..#R...1j.....&nC:.......%"0./..Zw>s6.....h.T.f&.fDl..3...s..~U.G..!H.m...pN..b.'.+A...bI...Y.4..'..........hC..p(R'y...!..Ip..o{.z.M*....J.cUF.......I.).....1...N... ..J$...t..\....+Jpf.....r..mh;......3..8.y.L.I..`..{.@.....^..*..Q.K.6x.....D..~O.?..L.I.........K.,Py..~..!......R...........@....4.Y

Is it possible to hide/encrypt the client's request?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Encryption/Decription

Ticket http://synopse.info/fossil/info/73da2c17b1 was in fact incorrect. I committed it without checking the resulting process.

It disabled client-side compression.

I have re-enabled it, so it should work as expected.

See http://synopse.info/fossil/info/39036dba97

sjerinic

Member

Registered: 2013-02-11

Posts: 51

Re: Encryption/Decription

Thanks AB!

Everything is working properly.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Encryption/Decription

I've just added AES encryption using official Microsoft AES Cryptographic Provider (CryptoAPI) in SynCrypto.
See http://synopse.info/fossil/info/6182b5149a
It may be used instead of our own classes.

But it is slower, and does not implement all chaining modes.
So, by default, we will stick to our good optimized asm versions.

Delphi rocks!

sjerinic

Member

Registered: 2013-02-11

Posts: 51

Re: Encryption/Decription

Thanks, but our good optimized asm version sounds better solution in my case.

nirnir

Member

Registered: 2013-11-11

Posts: 66

Re: Encryption/Decription

Can I encrypt only some of the requests and not all of them ? (The server will decide when)
I'm afraid it will decrease performance.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Encryption/Decription

The server will encrypt using AES only both the client and server do state that they support the encryption.
So you can let the client decide, as the server, globally but not by request.

This is a feature implemented at transmission layer level (i.e. in TSQLHTTPServer), not at application layer (i.e. in TSQLRestServer)...

zemorango

Member

Registered: 2017-09-28

Posts: 5

Re: Encryption/Decription

First off all I want to apologize to AB, because yesterday I created a Ticket for this and it was a huge mistake because that was not the place to do it, sorry AB for wasting you time, trully I'm.

A I said I'm a newbie to Mormot, But I already managed to recreate the sample nº 16 to work in Unigui just like the VCL one , This is really a great Framework.

But I would like to Encrypt the data between the client and server. I did as AB tells on this post https://synopse.info/forum/viewtopic.php?id=1553

But I can't see where to use the CompressShaAesSetKey() or how to use the global key.

I have this lines of code in server or the Sample nº 16 :

     // register our IRemoteSQL service on the server side
      aServer.ServiceRegister(TServiceRemoteSQL,[TypeInfo(IRemoteSQL)],sicClientDriven).
      // fProps should better be executed/released in the one main thread
       SetOptions([],[optExecInMainThread,optFreeInMainThread]);
      // launch the HTTP server
      aHTTPServer := TSQLHttpServer.Create(PORT_NAME,[aServer],'+',useHttpApiRegisteringURI,32,secSynShaAes);

And this one in the Client :

 fModel := TSQLModel.Create([],ROOT_NAME);
  fClient := TSQLHttpClient.Create('localhost',PORT_NAME,fModel);
  TSQLHttpClientWinHTTP(FClient).Compression := [hcSynShaAes];
 

So what its missing ? how to use the  CompressShaAesSetKey() or where to use the key for the encryption ?

Regards

Zemorango

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Encryption/Decription

For encryption, you could rather use https (for JavaScript/C# clients) or websockets (for Delphi clients).

zemorango

Member

Registered: 2017-09-28

Posts: 5

Re: Encryption/Decription

Hi AB, thanks for the quick answer,

I try to change the sample 16 to work with websocks , the server start , but when i Start the Client I've got an error.

my code onde the server is this one :

aServer := TSQLRestServerFullMemory.Create(aModel,'users.json',false,true);
    try
      // register our IRemoteSQL service on the server side
      aServer.ServiceRegister(TServiceRemoteSQL,[TypeInfo(IRemoteSQL)],sicClientDriven).
        // fProps should better be executed/released in the one main thread
       SetOptions([],[optExecInMainThread,optFreeInMainThread]);
      // launch the HTTP server

     // aHTTPServer := TSQLHttpServer.Create(PORT_NAME,[aServer],'+',useHttpApiRegisteringURI,32,secNone);

       aHTTPServer := TSQLHttpServer.Create(AnsiString(PORT_NAME), [aServer], '+', useBidirSocket);
            TWebSocketServerRest(aHTTPServer.HttpServer).ServerKeepAliveTimeOut := CONNECTION_TIMEOUT;
            { WebSocketServerRest := } aHTTPServer.WebSocketsEnable(aServer, '2141D32ADAD54D9A9DB56000CC9A4A70', false);

my client code is running using unigui, so its a web aplication,  the code  for the initialization of the request to the server is .

fModel := TSQLModel.Create([],ROOT_NAME);
 // fClient := TSQLHttpClient.Create('localhost',PORT_NAME,fModel);

   fClient := TSQLHttpClientWebsockets.Create(AnsiString('localhost'), AnsiString(PORT_NAME), fModel,
      SendTimeout, ReceiveTimeout,
       ConnectTimeout);

   TSQLHttpClientWebsockets(fClient).KeepAliveMS := CONNECTION_TIMEOUT;
   (fClient as TSQLHttpClientWebsockets).WebSocketsUpgrade('', False);
   (fClient as TSQLHttpClientWebsockets).WebSocketsUpgrade('2141D32ADAD54D9A9DB56000CC9A4A70', False);

This is the result when I start the client

20170930 15095609  !  +    mORMotHttpServer.TSQLHttpServer(0230A510).Create useBidirSocket (secNone) on port 888
20170930 15095609  " info  SetThreadName 00001BEC=TSQLHttpServer 888/root TWebSocketServerRest
20170930 15095610  " trace mORMot.TSQLRestServerFullMemory(0237B680) BeginCurrentThread(TWebSocketServerRest) root=root ThreadID=00001BEC ThreadCount=1
20170930 15095610  ! http       mORMotHttpServer.TSQLHttpServer(0230A510) {"TWebSocketServerRest(02337A70)":{"ServerConnectionCount":0,"ServerKeepAliveTimeOut":3000,"TCPPrefix":"","ThreadPool":{"TSynThreadPoolTHttpServer(023A6840)":{"HeaderErrors":0,"HeaderProcessed":0,"BodyProcessed":0,"BodyOwnThreads":0,"RunningThreads":2}},"ThreadPoolContentionCount":0,"ThreadPoolContentionAbortCount":0,"APIVersion":"WinSock 2.0.514","ServerName":"mORMot (Windows)","ProcessName":"root "}} initialized for root
20170930 15095610  !  -    00.004.449

Background server is running.

Press [Enter] to close the server.

20170930 15103847  # info  SetThreadName 00002F14=TSQLHttpServer 888/root TWebSocketServerResp
20170930 15103847  # trace mORMot.TSQLRestServerFullMemory(0237B680) BeginCurrentThread(TWebSocketServerResp) root=root ThreadID=00002F14 ThreadCount=2
20170930 15103847  # EXC   ESynCrypto {"Message":"TAESCFB.DecryptPKCS7: Invalid InputLen=40"} at 0054AA76  stack trace API 0055BB3B 0055BB64 0040A880 775FD544 775EAD8F 0054AA76 0054AD9D 0054AE74 0054DD89 00662FA0 006647D1 006654DF 00665EA2 0066629E 006488DE 004B5E04 0040B36E 74F43744 775D9E54 775D9E1F
20170930 15103847  # EXC   EWebSockets {"Message":"TWebSocketProtocolBinary.AfterGetFrame: encryption error sprInvalidMAC"} at 00662FDE  stack trace API 0055BB3B 0055BB64 0040A880 775FD544 775EAD8F 00662FDE 006647D1 006654DF 00665EA2 0066629E 006488DE 004B5E04 0040B36E 74F43744 775D9E54 775D9E1F
20170930 15103847  # trace mORMot.TSQLRestServerFullMemory(0237B680) EndCurrentThread(TWebSocketServerResp) ThreadID=00002F14 ThreadCount=1

If I remove this line ( regarding encryption), then it work just fine, but using Fiddler I can se all the comunication in plain text, no security

 // AES encrypt
(fClient as TSQLHttpClientWebsockets).WebSocketsUpgrade('2141D32ADAD54D9A9DB56000CC9A4A70', False);

IF Using https, fiddler make a MIM attack and decode all the SSL encoding, so how can I protect my httprequest?
I'm building a big Unigui application, and I really just need a way to protect the data when making my request to the server.

Thank's in advance

zemorango

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Encryption/Decription

No need to upgrade the socket, or set ServerKeepAliveTimeOut.

Just use a TSQLHttpClientWebsockets and call WebSocketsUpgrade() once.

Take a look at sample 31, especially Project31LongWorkClient.dpr.

zemorango

Member

Registered: 2017-09-28

Posts: 5

Re: Encryption/Decription

Hi Ab, thanks for the help,
Now it works without errors, but no encryption is made, in fiddler in the header request there is no indication of encryption only gzip compression, so I can see everything in plain text,

Don't know what else to do to give some protection to the data, do you have any support plan that I can buy and get help?

Tanks

zemorango

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Encryption/Decription

Encryption is working only if both sides do enable the protocol.
See above https://synopse.info/forum/viewtopic.php?pid=9435#p9435

Anyway, hcSynShaAes is not a full security feature, since it is optional.

HTTPS is the way to go, for something standard, with non Delphi clients.
Or WebSockets between mORMot clients.

zemorango

Member

Registered: 2017-09-28

Posts: 5

Re: Encryption/Decription

Hi, Ab thank for you help, I know that I'm being persistent and boring but please,

I have enabled the protocol on both sides

On server

  { WebSocketServerRest := } aHTTPServer.WebSocketsEnable(aServer, '2141D32ADAD54D9A9DB56000CC9A4A70', false);

On client  ( Delphi unigui aplication running on browser)

(fClient as TSQLHttpClientWebsockets).WebSocketsUpgrade('2141D32ADAD54D9A9DB56000CC9A4A70', False);

And still in fiddler or using the browser network console I can see all the data in plaintext, In my headers I just can't make  the ACCEPT-ENCODING: synshaaes
apears,

Do I missing something? If I'm using websocks do I need to still using  this code on server?

TSQLHttpServer.Create('888',[DataBase],'+',useHttpApiRegisteringURI,32,secSynShaAes);

and this one on client?

MyClient.Compression := [hcSynShaAes];

ps : I know that you are a busy man, but don't you have a support plan that I can pay for it and receive some help?

Regards

zemorango

Last edited by zemorango (2017-10-02 10:03:15)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Encryption/Decription

WebSocketsEnable don't use synshaaes at all.
It has its own encryption ONCE upgraded to WebSockets, in the binary frames.
A WebSocketsEnable GET request from a browser will show a plain HTTP connection, by definition.