Response caching and 304 Not Modified

chapa · 2013-04-12 06:38:20

I would like to share my use case which can be implemented as part of the framework, probably property at REST server level.

I use to return JSON response to the clients (in my case browsers) using method based services generating custom and dynamic json response.
Browsers query the server at high rate (every second) to catch and display any data change which may occur.
Simple function called on every content response:

procedure ReturnContent(var aCtxt: TSQLRestServerCallBackParams; const aContent: RawUTF8);
var
  clientETag: RawUTF8;
  serverETag: RawUTF8;
begin
  clientETag := FindIniNameValue(pointer(aCtxt.Call.InHead),'IF-NONE-MATCH: ');
  serverETag := '"' + CardinalToHex(Hash32(@aContent[1], Length(aContent))) + '"';

  if clientETag = serverETag  then
    aCtxt.Returns('', 304)
  else
    aCtxt.Returns(aContent, 200, JSON_CONTENT_TYPE_HEADER + #13#10 + 'ETag: ' + serverETag);
end;

Any decent browser will handle 304 Not Modified result and resent ETag according HTML specifications.
But actual transfer of data will not occur, which is big benefit if handling a lot of clients and they need to query the server at high rate.
Unfortunately, still the response will take server resources to generate the response content in order to build its ETag.

ab · 2013-04-12 09:38:20

I've included this feature to the official TSQLRestServerCallBackParams object.
See http://synopse.info/fossil/info/96fbe48fa8

Thanks a lot for the idea!
Feedback is welcome, as usual.

The official (just updated) documentation wrote:

Browser speed-up for unmodified requests
When used over a slow network (e.g. over the Internet), you can set the optional Handle304NotModified parameter of both Ctxt.Returns() and Ctxt.Results() methods to return the response body only if it has changed since last time.
In practice, result content will be hashed (using crc32 algorithm) and in case of no modification will return "304 Not Modified" status to the browser, without the actual result content. Therefore, the response will be transmitted and received much faster, and will save a lot of bandwidth, especially in case of periodic server pooling (e.g. for client screen refresh).
Note that in case of hash collision of the crc32 algorithm (we never did see it happen, but such a mathematical possibility exists), a false positive "not modified" status may be returned; this option is therefore unset by default, and should be enabled only if your client does not handle any sensitive accounting process, for instance.

mpv · 2013-04-12 21:09:06

To implement http level cache (not only etag but "not modified since" and so on) we must first change auth scheme used in mormot.
Now we put different session_signature in every url, so browser decide
root\blabla?session_signarute=1 and
root\blabla?session_signarute=2
is different url and cashe is therefore impossible.
So my suggestion is to change auth scheme. As minimum - move session signature to request header section, as optimal - use standart HTTP algoritm for auth(we can add out algorotm to 'WWW-Authenticate: Synopse, NTLM, digest` - in this case delphi client can use synopse algoritm, browser NTLM or digest).
I know all PRO of current imlpementation, but the contra is -
1) hard to work from browser,
2) HTTP level cache is impossible
3) impossible to work with 3party software - for example I have idea to implement WebDAV access to my server, but webDAV client know nothing about current auth algoritm.
What do you think?

ab · 2013-04-12 21:17:12

About the CONs:
1) There are several JavaScripts implementations in this forum, and not difficult to work with.
2) You can add this at higher level
3) You can bypass the authentication for every method-based service you need.

My main concern is that headers are HTTP specific, not RESTful...

mpv · 2013-04-13 08:34:26

Yes, I understand that headers are HTTP specific... But in current auth scheme does not make sense to send 304 response.

This is because in case of BROWSER, client cache is implemented on BROWSER level (I'm not control it) for each URI. So scenario:

-> GET root\entitiy\1?session_signature=11
<- 200 OK
ETag:1etag
resp body

10ms after
-> GET root\entitiy\1?session_signature=11 (signature not changed because of 250 ms period)
HEADER IF-NONE-MATCH: 1etag (here is ETag I got for this URI on prev responce. This header is added by BROWSER)
<- 304 Not modified

1 min after:
-> GET root\entitiy\1?session_signature=12 (from logical POV i try to GET the same entity, but from browser POV is not because URI cahnged)
HEADER (BROWSER not include here IF-NONE-MATCH: header, because no entry in browser cache for URI root\entitiy\1?session_signature=12
<- 200
ETag:1
resp body

So, browser client use cache only 250 ms (the time session_signature not changed) after this period GET request going with another URI
And the worst - I overflow browser cache, because I never send GET root\entitiy\1?session_signature=11 in future

Also responce may cached by proxy between server and client. And I don't control proxy at all.

Yes - it is possible to implement cache at higher level I already done ( I get server responce, store in in browser indexedDB with key = crc32(request body) store ETag=CRC32(respondeBody) in indexedDB, next time I send POST with body {bla-bla, ETag: xxx}, server analyse post body and return 200 with body {notModified: true} in case nothing changed. But such implementation requier HUGE amount of javaScript client code, do not use caching responces on proxy between client and server, need drums in non indexedDB browser and so on.

In fact we already widely use HEADER in mORMot (JSON_CONTENT_TYPE_HEADER in every responce) so I don't see contra's to use it in AUTH request. But in result we have STANDART auth scheme, supported by ANY HTTP compartible client ( we can use digest-md5 and NTLM and continue to add session_signature for autentification but in header section).

One example is WebDAV - webDAV client use NTLM or digest auth and don't know anything about session_signature. I can't bypass auth on server because depending of user I must return different content.
Second - I need to connect to mORMot from Excel (get some data and build report using pivotTable). If use standatd auth scheme I do nothing, just cheate XMLHTTPRequest COM object and send request - server authorise me using Kerberos (NTLM). In current scheme I must write lot of code.
Third - I have iPad client for my server written on ObjectiveC. I spent couple of time to explain for Objective C developer mORMot auth schema. Developer spand lot of time to implement it.
And this happens every time I need to connect new type of client to my server

ab · 2013-04-13 11:02:26

You are right: the "Response 304" does not make any sense when used with URL-level authentication.
This is why it is implemented only for method-based services, and you should by-pass the authentication for those methods, via a TSQLRestServer.ServiceMethodByPassAuthentication() call.

"HTTP digest authentication" is less reliable than our RESTful scheme, since it is more vulnerable to a man-in-the-middle (MitM) attack, IMHO.
It is about authentication, not message signature.

I've added the following remark to the documentation:

SAD wrote:

Be aware that you should disable authentication for the methods using this Handle304NotModified parameter, via a TSQLRestServer.ServiceMethodByPassAuthentication() call. In fact, our RESTful authentication - see 18 - use a per-URI signature, which change very often (to avoid men-in-the-middle attacks). Therefore, any browser-side caching benefit will be voided if authentication is used: browser internal cache will tend to grow for nothing since the previous URIs are deprecated, and it will be a cache-miss most of the time.

For specific requests (like WebDAV, or Excel connection), you may have to use method-based services without authentication, not interface-based services.

It does make sense to provide other authentication schemes, but it has to feature all that we need, i.e. user/group authentication.
For instance, Windows authentication has been added, with only small code change.

And there is already some simple versioning/numbering system, avoid most requests, with the "InternalState" cardinal value as returned with every request, which changes on any DB write.
It is pretty naive, but works well for most simple projects, which mostly read data, and not often change it - even if it is of no benefit when the data change often.
A more sophisticated scheme is possible, of course.

Non standard HTTP headers sometimes are cleared by proxies or antiviruses/firewalls, from my experiment...
We may have to find a working pattern, when including the signature within the headers.
And it would not help with the fact that it will require a dedicated dev (as for your ObjectiveC problem). But I guess our mORMot auth scheme is pretty easy to follow and implement.

As always, and especially even more about security, any input and proposal is welcome!

mpv · 2013-04-14 16:36:49

I propose add support for multiple AUTH schema in mORMot.
I understand what it require some breaking changes, but as a result we got auth, what compatible with standards and give for mORMot some additional benefits, (we can implement OAuth for example, very easy to use mORMot from browser, we can implement cache according to HTTP standards).
So, the idea is:
1) implement something like TSQLRestServer.RegisterAUTHScheme(const schemeName RawUTF8; onAuthentificate: function(Ctxt), onGetAuthentificationParams: function(Ctxt), onAuthorize: function(Ctxt)).

2) in TSQLRestServer.URI instead of

  URI.Error('',HTML_FORBIDDEN); // 403 in case of authentication failure

return

  
URI.Error('', 401);
  for each AuthScheme in registeredAuthScheme
     URI.Call.OutHead.Add('WWW-Authenticate:' + AuthScheme.SchemeName + AuthScheme.onGetAuthentificationParams(Ctxt));
  end;

HTTP protocol allow to add multiple WWW-Authenticate headers. The task of the client to choose the strongest of the supported schemes.

In case we register schemas [mORMot, Negotiate, NTLM, Digest, Basic] we got something like:

-> GET /root/something
<- 401 
  WWW-Authenticate: mORMot nonce="nonce_value"  
  WWW-Authenticate: Negotiate ...
  WWW-Authenticate: NTLM ...
  WWW-Authenticate: Digest realm="mORMot digest", qop="auth,auth-int", nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093", opaque="5ccc069c403ebaf9f0171e9517f40e41"
  WWW-Authenticate: Basic realm= "mORMot basic"'

in case of Delphi client or other client what support mORMot auth scheme next step is:

-> GET /root/something
  Authorization: mORMot UserName=..., PassWord=..., ClientNonce=... 
<- 401 
  WWW-Authenticate: mORMot session number and a private key

a) second step
-> GET /root/something
  Authorization: mORMot session_signature="zzzzzzzzzzzzzzzzzzzzzzzzzzzz"
<- 200 Ok
  content here

all other client request simple add header

   Authorization: mORMot session_signature="...."

Browser or client are not familiar with mORMot auth scheme just choose another. For example if I do request to intranet zone IE ignore WWW-Authenticate: mORMot and choose Negitiate, if request go to "not trusted" site - Digest and so on.

3) In TSQLRestServer.URI if HandeAuthentification=true check "Authorization: schemaName" header, find Auth scheme with schemaName from registered and call AuthScheme.onAuthirize(..)
for mORMot schema, for example, onAuthirize do the same as in TSQLRestServer.URI in current implemetation.

4) for Neginiare and NTLM schema we can use chaa imlementation given here: http://synopse.info/forum/viewtopic.php?pid=5809#p5809
We just must add ConnectionId to the Call structure instead of Remote-IP from chaa's code. ConnectionId is retrived as Eric wtite in this post:
http://synopse.info/forum/viewtopic.php?pid=5816#p5816
"To the Process method I just added a connectionId parameter, which takes the Req^.ConnectionId for the http.sys server, and a Int64(Socket) for the thread/socket-based one"

Negotiate is a connection authentication scheme - not http authentication scheme, so client must be keep-alive, ConnectionId must be used instead of sessionCardinal. onAuthirize(..) handler for Negotiate just look session with ID = ConnectionId exist in sessionList.

5) Also I propose to add custom handler for user password checking for each schema (it allow not use TAuthUser/TAuthGroup but, for example, LDAP) and the same handler to calculate Group's for user ( In my case, for examlpe, I need to check LDAP groups for user and assotiate it with TAuthGroup

sorry for my english

ab · 2013-04-15 08:09:43

Somewhat huge modification, in order to add generic authentication...
But certainly worth it.
I'm just afraid it may break our KISS design.

The headers are available in all communication protocols, so can be used as such.
I'm still not convinced that DIGEST and BASIC may be used on browser side nice & easily: you need to enter a password in an awfull login window, and there is no mean to logout.
See what I wrote in the SAD pdf, and http://stackoverflow.com/questions/3195 … 64#7158864
and http://broadcast.oreilly.com/2009/12/pr … ation.html

Some points:
Basic at http://technet.microsoft.com/en-us/library/cc784037
Digest at http://technet.microsoft.com/en-us/library/cc738318
Negotiate at http://msdn.microsoft.com/en-us/library/ms995329

Perhaps some tickets to be created, not to put everything in the same ticket, but create some INVEST use cases.
See http://en.wikipedia.org/wiki/INVEST_(mnemonic)

Some potential stories:
- Add a "mORMot" authentication mode at header level, in addition to the user mode;
- Add a generic way to add authentication;
- Add other authentication modes (Google, Digest, Basic....).

Some questions:
Which authentication scheme are worth implementing?
Will we allow authentication with lower security than our own mechanism?
Will it break the KISS design?

Chaa · 2013-04-15 09:17:51

I would suggest to do the following:

1. Move session_signature from URI to header.

Authorization: mORMot session_signature="...."

This is a major breaking change. But this would solve the main problem - browser caching and "304 Not Modified" responses.

2. Then add ConnectionId, as I wrote in http://synopse.info/forum/viewtopic.php?pid=5858#p5858 or any other way.
It is necessary to identify client connection in a two-step authentication.

3. Instead of the RegisterAUTHScheme (as mpv wrote) may be add a virtual method for authentication and for retrieving users and groups?

ab · 2013-04-15 09:44:03

I suspect point 1 should be optional.

Point 2 sounds just right.

My concern about several authentication modes is that it should be on both client and server sides...
I suspect we may just add additional modes for non-Delphi clients.
A RegisterAUTHScheme () method is perhaps easier to work. With dedicated classes to implement mode, instead of method callbacks.

mpv · 2013-04-15 13:58:56

I'm still not convinced that DIGEST and BASIC may be used on browser side nice & easily: you need to enter a password in an awfull login window, and there is no mean to logout

It's easy to implement DIGEST and BASIC auth in browser without ugly login window in AJAX application - we just catch 401 responce in global level and show our login window. Examlpe of pure javaScript digest auth is here http://marcin-michalski.pl/2012/11/01/j … ript-ajax/ (see JavaScript client section).

About security - YES I agree with you - BASIC is not secure at all. Digest is secure in case we use HTTPS + qop="auth-int" + timestamp in server nonce.
But sometimes we need basic auth. The first example - using ApatchBench for benchmark. In current implementation in mORMot I set HandleUserAuthentication = false before benchmark, but this is not real-life bench. If we implement multiple auth, I can set BASIC and test real business logic.

Which authentication scheme are worth implementing?

I think mORMot / Negotiate / Basic.

Will we allow authentication with lower security than our own mechanism?

It's on developer choice. If I register BASIC AUTH schema for my server - I have low security, if register only mORMot - Hi security.

Will it break the KISS design?

IMHO - not. In case I develop non-Delphi ( JavaScript, VisualBasic, ObjectiveC, Java client for Android) app it's much SIMPLE to use basic scheme when I create prototype and then switch to more secure schema it my prototype will need someone.

A RegisterAUTHScheme () method is perhaps easier to work. With dedicated classes to implement mode, instead of method callbacks.

You are 100% right, my mistake.

ab · 2013-04-15 14:29:46

So do you agree of adding those feature requests as one or several tickets:

1. Handle RegisterAuthenticationScheme() with one dedicated class, for mORMot URL-level signature (i.e. current scheme) - this will stay the default;
2. Implement a new abstract authentication, parent for all HTTP header-level new mode;
3. Implement new authentication class for mORMot header-level signature;
4. Implement Basic authentication;
5. Implement Negotiate authentication.

In fact, Digest may be difficult to implement, since we stored the SHA-256 hash in the DB, and the Digest mode expects some MD5 hash.

I suspect the RegisterAuthenticationScheme() should be for the whole server - unless TSQLRestServer.ServiceMethodByPassAuthentication() is called for some particular methods.
We may be able to do it at the interface/method level (as some new TServiceMethodExecutionOption types?), but it may need some big code refactoring. Is it worth it?
Or one authentication scheme for all methods is a good option?
Do several authentication schemes make sense? In fact, for instance if we allow both BASIC and mORMot schemes at the same, it does not make sense to use mORMot style, unless you have a per-interface/per-method scheme.

mpv · 2013-04-15 16:16:30

I'm not sure how you want to combine 1. and 2. in the same time
3.4.5. is OK.
For Digest - yes - digest store md5(Username + ':' + realm + ':' + Password) as a password hash. So to use Digest combined with other methods the only way is to add 2 password column in user table - for mORMot and for Digest.

I suspect the RegisterAuthenticationScheme() should be for the whole server

I also

Do several authentication schemes make sense?

I'm not sure that I understood correctly. If you mean:

 
TSQLRestServer.RegisterAuthenticationScheme('mORMot', TMoromAuth);  // here I mean mORMot algorithm using HEADER
TSQLRestServer.RegisterAuthenticationScheme('Negotiate', TNegotiateAuth);  
TSQLRestServer.RegisterAuthenticationScheme('Basic', TBasicAuth);

then yes - it make sense. In this case Delphi client choose mORMot, browser for domain user choose Negotiate (as more strong auth method) and for non-domain user - Basic

if you mean use mORMot-style (session_signature in URI) and header style auth in the same time - no, it not make sense as for me.

And from my POV here we need:
6. add connectionID attribute to TSQLRestServerURIParams ( for the http.sys server = Req^.ConnectionId, for socket-based = Int64(Socket), for namedPaper = Int64(GetNamedPipeClientProcessId/GetNamedPipeClientSessionId), for messages = Int64(window handle)

7. add event onUserAuth in witch I can check user credential and fill user GroupRights (I actually get a user groups from LDAP and associate it with group binary mask). See also http://synopse.info/forum/viewtopic.php?id=1218

Last edited by mpv (2013-04-15 16:19:18)

mpv · 2013-04-15 17:32:27

And about logout for basic/DIGEST type of auth - the easyest way is to implement logout method and drop session as we did in current implementation(in case of basic or digest where is only one session per user name possible of course). The same algoritm we can use for other auth type, for negotiate found session by connectionID, for mormot by sessionid.

ab · 2013-04-17 12:10:29

I've created 3 feature requests:

- ConnectionID parameter - http://synopse.info/fossil/info/0636eeec54
- Authentication classes - http://synopse.info/fossil/info/8c8a2a880c
- Authorization classes - http://synopse.info/fossil/info/7c079c49d6

In short, two new sets of class hierarchy will be available to set Authentication and Authorization, on purpose.
See http://technet.microsoft.com/en-us/library/cc759647

My proposal is to use some interfaces for it, so that we may be able to use remote services for both purposes.

mpv · 2013-04-17 16:28:01

Very good! I make some remarks in tickets:
- in Authentication classes - http://synopse.info/fossil/info/8c8a2a880c seems "this presentation" link is broken...
- in Authorization classes - http://synopse.info/fossil/info/7c079c49d6 - As for me it will be good to combine onUserAuth + authorization class hierarchy. For example in my case in some services I do not use TSQLAuthUser/Group ( some of my services not use any DB at all - only LDAP for auth - for example cache service in case of ACTIVE-ACTIVE load balancing on main server)

ab · 2013-04-18 08:54:37

Yes, the "presentation" link was your own reference link - http://marcin-michalski.pl/2012/11/01/j … ript-ajax/
But having to implement everything in JavaScript is just not a better option than implementing our mORMot scheme, which is more secure.

Yes, I would like authentication/authorization classes/interfaces to be as generic as possible, i.e. as needed by TSQLRestServer.URI process, so allow resource access without no-predefined scheme, but still compliant with the current KISS implementation.

I'm still not convinced about the ConnectionID parameter.

I have implemented it - http://synopse.info/fossil/info/0636eeec54 - but am still a bit doubtful of its utility.

It does not make any problem to me to have several physical connections used for very same session.
For instance, it is how event notification may be implemented in the future (via websockets or long-pooling.

Chaa · 2013-04-18 10:01:19

I'm still not convinced about the ConnectionID parameter.

In current NTLM implementation we send SecCtxId as parameter to Auth function.

GET ModelRoot/auth?UserName=&id=<SecCtxId>&data=...

But if we use NTLM as it is used in browser-based authentication, there will be no place for SecCtxId.
So we need a way to distinguish clients before to authentication.
The same problem will be in any two-step authentication, like Digest.
To solve this problem, I propose to use ConnectionID.

In sample code http://synopse.info/forum/viewtopic.php?pid=5809#p5809 I use RemoteIP header to distinguish clients, but it is an unreliable method.

ab · 2013-04-18 10:15:42

... but you can use a generic nonce (generated on the server side), instead of a connection-based identifier, AFAIK...

This is what we do with the mORMot authentication scheme (with some additional steps/security points).

In all cases, there is a ConnectionID available now.

Chaa · 2013-04-18 10:35:14

Generic nonce or other identifier we need to send from a server to a client at the first step of authentication, and from the client to the server in the second step.
But the browser will not to do that.

ab · 2013-04-18 11:56:42

Chaa wrote:

But the browser will not to do that.

... unless you do it in JavaScript, like with our mORMot scheme.

Chaa · 2013-04-18 14:09:09

unless you do it in JavaScript

I wrote about integrated authentication (NTLM), there is no way to use javascript.

ab · 2013-04-18 14:20:07

Chaa wrote:

I wrote about integrated authentication (NTLM), there is no way to use javascript.

You are right.
I meant the "Digest" scheme only.
With NTLM it does make sense.

But it all cases, even NTLM can use a server-side nonce, since we implement it on our server side, right?

mpv · 2013-04-18 18:10:13

There is no need to put connectionID into header - we just need it on the server. NTLM scheme authorize connection - it mean if client close socket and open another it MUST repeat NTLM handshake. This is why we must use only keep-alive HTTP connection and can use http.sys level Req.connectionID parameter to identify 2nd part of hanshake (or socket handle in case of plain socket server) - and all other client requests. After success handshake we MUST use http.sys req.connectionID(socket handle) for each client request to authorize client request - this is client sessionID. Current implementation of NTLM do not do it (as I understand) - currently it is a mix of NTLM authentification and mORMot authorization. I`ll make code sample tomorrow - it seems my english very poor to explain what I meen.

Last edited by mpv (2013-04-18 18:16:55)

Chaa · 2013-04-19 02:38:11

Yes, the current version authenticates the client only once in the call to Auth.
After that, the customer is given the SessionID, and subsequent requests are processed as usual.

In my opinion, there is no need to authorize each request, sufficient that the function Auth gave out the session identifier.
In this case, only one function (Auth) depends on the method of authentication, all other calls processed regardless of the manner in which the client is logged in.

The only problem - when the server restarted, and clients must re-authenticate.
For clients in Delphi this does TSQLRestClientURI.URI, but we need that re-authentication is performed for the AJAX client, too.

ab · 2015-03-25 15:48:24

Today, we get rid of "ConnectionID:" in the HTTP headers, since we now have the TSQLRestURIParams.LowLevelConnectionID corresponding to the HTTP server context, e.g. for SSPI authentication.

Perhaps we would have to check and validate the code after this change.

mORMot Open Source

#1 2013-04-12 06:38:20

Response caching and 304 Not Modified

#2 2013-04-12 09:38:20

Re: Response caching and 304 Not Modified

#3 2013-04-12 21:09:06

Re: Response caching and 304 Not Modified

#4 2013-04-12 21:17:12

Re: Response caching and 304 Not Modified

#5 2013-04-13 08:34:26

Re: Response caching and 304 Not Modified

#6 2013-04-13 11:02:26

Re: Response caching and 304 Not Modified

#7 2013-04-14 16:36:49

Re: Response caching and 304 Not Modified

#8 2013-04-15 08:09:43

Re: Response caching and 304 Not Modified

#9 2013-04-15 09:17:51

Re: Response caching and 304 Not Modified

#10 2013-04-15 09:44:03

Re: Response caching and 304 Not Modified

#11 2013-04-15 13:58:56

Re: Response caching and 304 Not Modified

#12 2013-04-15 14:29:46

Re: Response caching and 304 Not Modified

#13 2013-04-15 16:16:30

Re: Response caching and 304 Not Modified

#14 2013-04-15 17:32:27

Re: Response caching and 304 Not Modified

#15 2013-04-17 12:10:29

Re: Response caching and 304 Not Modified

#16 2013-04-17 16:28:01

Re: Response caching and 304 Not Modified

#17 2013-04-18 08:54:37

Re: Response caching and 304 Not Modified

#18 2013-04-18 10:01:19

Re: Response caching and 304 Not Modified

#19 2013-04-18 10:15:42

Re: Response caching and 304 Not Modified

#20 2013-04-18 10:35:14

Re: Response caching and 304 Not Modified

#21 2013-04-18 11:56:42

Re: Response caching and 304 Not Modified

#22 2013-04-18 14:09:09

Re: Response caching and 304 Not Modified

#23 2013-04-18 14:20:07

Re: Response caching and 304 Not Modified

#24 2013-04-18 18:10:13

Re: Response caching and 304 Not Modified

#25 2013-04-19 02:38:11

Re: Response caching and 304 Not Modified

#26 2015-03-25 15:48:24

Re: Response caching and 304 Not Modified

Board footer