HTTP Server / DWScript

Eric · 2012-11-26 16:43:15

Just asking, is there already some code on integrating DWScript with the mORMot HTTP server?

Like some basic exposure for HTTP requests & response fields, in the case of a simple file-server with some dws processed pages alongside raw files.

Initially that would be for making a simple HTTP server demo for DWScript multi-threading & caching features.

ab · 2012-11-27 07:44:14

There is no code available yet interfacing DWS and mORMot.

I suspect it could make sense, at several levels:

1. Use http.sys kernel mode high performance server to serve static and DWS-generated content.
See sample "09 - HttpApi web server" for a basic HTTP server.
It should be easy to add DWS process to it.

2. Use DWS/SMS javascript compiler to create Client-side code.
This is on the official mORMot roadmap.
We would like to use SMS as client back end, for remote ORM, interface-based services access.

3. Expose some mORMot methods to the DWS engine, like ORM, interface-based services, and so on.
This may be at the same level as the Client-side code.

4. Includes DWS as server-side scripting engine.
Some mORMot users use JavaScript for server-side business logic coding.
IMHO it could be great to use DWS as server-side scripting engine. It has great performance and low overhead (much lower than JavaScript).

May be we may start some joined Open Source link between our projects?

Eric · 2012-11-27 08:45:49

ab wrote:

1. Use http.sys kernel mode high performance server to serve static and DWS-generated content.
[...]
4. Includes DWS as server-side scripting engine.

Yes those are the two early aspects I was after.

We have an http server tech DWScript runs on here (at work), but it's proprietary -- hence the lack of webserver demos in DWS repository, which is a bit ironical for a script engine with "Web" in its name, web-server roots, etc. ;-)

I started sometime ago a demo on Indy (to provide the open-source http server), but I don't use or know Indy much, and the performance wasn't so good, so I never really bothered finishing the demo.

ab wrote:

3. Expose some mORMot methods to the DWS engine, like ORM, interface-based services, and so on.

Yep, that could be a second step.

Using DWScript connectors and language extension mechanisms, it should be possible to expose many things in various ways.

One simple language extension I had in mind was being able to have inline SQL. Assuming the DB schema is known at script compile time, something like that could be supported, with full compile-time syntax-checks & strong typing, both for the Pascal & the SQL

filter := 123;
cursor := select whatever from table where field = @filter;
while not cursor.Eof do begin
...
cursor.Next;
end;

Eric

ab · 2012-11-27 09:33:00

Eric wrote:

One simple language extension I had in mind was being able to have inline SQL. Assuming the DB schema is known at script compile time, something like that could be supported, with full compile-time syntax-checks & strong typing, both for the Pascal & the SQL
filter := 123;
cursor := select whatever from table where field = @filter;
while not cursor.Eof do begin
...
cursor.Next;
end;

This could be just great...
Even if ORM tries to avoid writing SQL, and compute it for you...

TPrami · 2012-11-27 11:05:30

ab wrote:

2. Use DWS/SMS javascript compiler to create Client-side code.
This is on the official mORMot roadmap.
We would like to use SMS as client back end, for remote ORM, interface-based services access.

If I recall, the JavaScript compiler is now part of Smart Mobile Studio, and no longer OpenSource.

So not for everyone, which is slight bummer...

But anyways the Fast HTTP-server and any scripting for that would be useful in many projects...

-Tee-

Eric · 2012-11-27 11:26:22

ab wrote:

Even if ORM tries to avoid writing SQL, and compute it for you...

Yep, but sometimes good old fashionned SQL beats OO in terms of expressiveness. I find OO is practical for browsing & updating data, but SQL is better for joining/filtering "en masse" or spitting out reports/statistics.

I don't find LINQ f.i. to be much neater than the corresponding SQL would be IMHO. I've seen several articles caliming the opposite, but the examples were typically twisted and the SQL looked like auto-generated SQL (like f.i. http://www.linqpad.net/WhyLINQBeatsSQL.aspx)

There is another thing I would like to give a try to: exposing an array of objects or records (in memory) as transient SQLite virtual tables.
This would allow things like (using TPoint just for illustration, the query is meaningless)

a1 : array of TPoint;
a2 : array of TPoint;
...
result := (select a1.x, a2.y
from a1 left join a2 on (a1.x=a2.x)
where a1.y<a2.y).ToArray(TPoint);

performance might not always be extraordinary, but you could express wicked things, and the work required to make the above would be reasonnable given that SQLite would be doing all the hard stuff.

ab · 2012-11-27 12:31:12

@Eric
I fully agree about LINQ.
"Pure" SQL syntax does make better sense that this C# / SQL mixture.
SQL is truly a programming language.

As Dr Hipp states: "SQL Is A High-Level Scripting Language" - see http://www.fossil-scm.org/xfer/doc/trun … eory1.wiki

Including SQL statements within DWS language, including strong typing and such, would need a full SQL parser embedded within DWS.
Not an easy task. But I'm quite sure you could do it!
There is already a simple but very efficient lexer created for SQLite - see http://www.hwaci.com/sw/lemon - which includes the updated SQL grammar. Could make sense to re-use the existing grammar when including into DWS source tree. I suspect it could be possible to create a pascal source code instead of C, or just re-use the statically linked C parser, already included within SQLite.

Your sample code about TPoint could make sense with SQLite, and it may be VERY fast, for several reasons:
- There is a RTREE extension in SQLite, making such queries blazzling fast;
- Virtual tables in SQLite is a very powerful feature we use a lot in mORMot (e.g. to map external DB in MS SQL or Oracle as a "native" SQLite table) - you may easily use such virtual tables from an in-memory array of any structure - that is, no need to create a temporary DB, but direct access to the data;
- So you can have both LINQ to SQL and LINQ to data features with the same SQLite3 kernel, also accessing external tables with mORMot database mapping to external data.

I believe that using the proven SQLite3 engine is better than reinventing the wheel, as with LINQ.
SQLite3 can be a powerful kernel for accessing data.
mORMot can offers every low-level SQLite3 process needed by DWS.
See http://blog.synopse.info/post/2011/08/0 … e3-limited about virtual tables and how we use SQLite3 to access internal or external data in a RESTful way.

Eric · 2012-11-27 12:53:01

Including SQL statements within DWS language, including strong typing and such, would need a full SQL parser embedded within DWS.

Actually I hope to get away with a "stupid" parser that would leverage the full SQL parser of SQLite, something like:

start "parsing" sql when encountering "select"
grab everything up to the end of the statement, noting variable references and expression in passing (maybe prefixed with a '@'), and replace them with parameters
feed the SQL to SQLite, have it prepare the query, report errors
assume sqlite3_column_decltype is meaningful enough to be used for strong typing

Of course that means the compiler must have access to the db at compile time (which is at runtime anyway, for a script), and that may not work out when connecting to arbitrary databases. But for dedicated web servers, that limitation should be okay in many cases.
And it would still be possible to compile all script at the web server startup, and thus "statically" validate everything.

And yes, I'm already using virtual tables like you to make queries across different databases. When worse comes to worse, the virtual tables can still be used as a way to conveniently filter and import a temporary table in SQLite, index it and get back the full speed.

ab · 2012-11-27 15:19:00

It could be a good idea to use all SQL commands, including INSERT/UPDATE/DELETE...

If the tables are virtual, you can use an in-memory database at compile time, with no speed penalty.
Even if it is not immediate to create an in-memory db or a virtual table.

You just need to know the context of the SQL statement from the pascal script point of view (like @arrayvariables and such) and feed SQLite3 as expected.

Eric · 2012-11-27 15:29:45

Had look at HttpApiServer demo, now I've a couple questions to ask:

1) If I understood correctly, the Request method is invoked from multiple threads at the same time, and those threads are from ServerThreadPoolCount ones? What does happen when the thread pool is exhausted? are new threads created on the fly or is there a backlog?

2) the Request methods takes raw http headers, however at the THttpSocket level, headers seem to have been already parsed. If there a standard way to get those pre-parsed headers instead? (to avoid re-parsing them)

3) Are there some url-encoded encoding/decoding routines? (not too complex, but might as well not replicate the effort) oops, saw them

4) Is there already multi-part/content-encoding parser for the request? (thinking of posted forms, might as well not replicate the effort for that)

Also, but that's beyond the scope of the demo, but is there already some form of URL rewriting and/or request dispatch API?
(thinking of being able to have a single virtual, url-rewritten web server serve DWScript, static files, and other mORMot services on the same port)

Last edited by Eric (2012-11-27 15:47:07)

Eric · 2012-11-27 15:35:52

ab wrote:

It could be a good idea to use all SQL commands, including INSERT/UPDATE/DELETE...

Yes, though a limiting factor would be how to distinguish an SQL query from regular Pascal code.
F.i. "CREATE" would awfully look like a class constructor to the parser.

A simple language extension can react on a keyword, if you've got to parse several terms to figure out what's happening, things become quite a lot more complicated.

So to keep things simple, some more complex start marker/keyword would be needed.

ab · 2012-11-27 15:48:31

1) Implements uses IOCP: threads receive pending requests from http.sys, handle them, then loop to the next.
No new threads are created on the fly. A call to THttpApiServer.Clone() initialize the threads (up to 256).
It is not necessary/wanted to increase the ppol, since more threads would not make it faster, from my experiment.
The backlog is handled by http.sys itself, at the TCP/IP level.
It is up to http.sys to handle low-level DOS attacks and such slow down.
In practice, http.sys is doing great work. Only some https issues occurs in XP SP2, but not since Vista/Seven/2008.

2) The SynCrtSock unit uses a THttpServerGeneric abstract class, which can be implemented with http.sys or plain socket.
So the headers are sent as one CRLF separated text buffer in both cases.
I do not think parsing will be an issue here. SynCommons has some very efficient functions to handle it with no memory allocation, like function IdemPCharAndGetNextLine().

3) SynCommons has those functions UrlEncode/UrlDecode, also optimized for speed.

4) multi-part is not handled by our units. I'm not sure, but I guess it is handled at http.sys level.
What is not handled by use is chunking for response - only for input.

5) Routing is handled by http.sys itself.
THttpApiServer.AddUrl() method is to be used to register a domain name. See method definition/documentation.
Then, at mORMot level, routing is dispatch in SQLite3Commons.pas, in TSQLRestServer.URI() method. It will dispatch content according from the supplied URI to either RESTful ORM remote access, method-based service, or interface-based services.
The easiest and fastest in your case is to use a method-based service, which will return the content as expected - this was what tested in the Roberto Scheinders blog article.
You can send a specific output type (HTTP_RESP_STATICFILE + OutContent as the UTF-8 encoded file name) to let http.sys serve a file content in the background - ideal for static sending; it also features range sending.
TSQLRestServer is generic, so it can then expose its content via several protocols, even at once: HTTP, GDI messages, named pipes... whatever you need.

ab · 2012-11-27 15:51:45

Eric wrote:

Yes, though a limiting factor would be how to distinguish an SQL query from regular Pascal code.
F.i. "CREATE" would awfully look like a class constructor to the parser.
.

I thought you were expecting a new internal type, like sql or even a variant, just like automation.
That is, will let

var data: array of integer; 
  request: sql;
begin
  request := insert into @data values (10,20);
  assert(request.success);
  assert(request.insertedrowcount=2);
  assert(length(data)=2);
  assert(data[0]=10);

be recognized as a SQL statement.

Eric · 2012-11-27 16:09:59

ab wrote:

I thought you were expecting a new internal type, like sql or even a variant, just like automation.

Yes, but that's not enough for instance you could have:

var insert : TWhatever;
...
request := insert.GetTheSQL();

or

insert : sql;
...
request := insert;

If the language extension registers "insert" as a new keyword, the above would fail.

And while select isn't too common in Pascal (IME), insert, create, update & delete are quite common.

ab · 2012-11-27 16:26:45

What about just embracing SQL statement with [ ] since it should not appear after an assignment token in normal pascal?

var data: array of integer; 
  request: variant;
begin
  request := [insert into @data values (10,20)];
  assert(request.success);
  assert(request.insertedrowcount=2);
  assert(length(data)=2);
  assert(data[0]=10);

It may help the coder switch from one syntax to another: whatever is between [ ] is SQL, not object pascal, and will return a variant with methods about statement execution.

Eric · 2012-11-28 09:10:35

ab wrote:

What about just embracing SQL statement with [ ] since it should not appear after an assignment token in normal pascal?

But it's allowed in DWScript to have inline static arrays.

In Smart it's already used in WebGL to declare vectors and matrix inline.

vector := [1, somevar, 3];
matrix := [1, 0, 0,
              0, 2, 0,
              0, 0, 1];

That said, another option could be to have an 'sql' keyword, for all sql statement that don't start with select, or requiring a db connection/object in front (which will probably be required anyway to support different db connection)

Last edited by Eric (2012-11-28 09:10:48)

Eric · 2012-11-28 15:46:06

Added SynopseWebServer demo to DWScript SVN.
Still pretty basic and missing tons of stuff, but works quite well with good perf already
I'll try to jazz it up in future versions to something usable as a "reference" for small production servers (ie. add some security, file I/O from DWS, stuff like that), so if you find mistakes & misuses, let me know.

Btw, is there any boiler plate comment header you want me to add in the headers of the units mentioning Synopse? a link to http://synopse.info or somewhere else? Or should it refer to mORMot instead?

ab · 2012-11-28 16:02:06

You can put a reference to http://synopse.info and the original sample source, like this.

{
    Will serve static content and DWS dynamic content via http.sys
    kernel mode high-performance HTTP server (available since XP SP2).
    See http://blog.synopse.info/post/2011/03/11/HTTP-server-using-fast-http.sys-kernel-mode-server
    WARNING: you need to first register the server URI and port to the http.sys stack.
    That is, run the application at least once as administrator.

    Sample based on official mORMot's sample 
    "SQLite3\Samples\09 - HttpApi web server\HttpApiServer.dpr"

    Synopse mORMot framework. Copyright (C) 2012 Arnaud Bouchez
      Synopse Informatique - http://synopse.info

    Original tri-license: MPL 1.1/GPL 2.0/LGPL 2.1

    You would need at least the following files from mORMot framework
    to be available in your project path:
    - SynCommons.pas
    - Synopse.inc
    - SynLZ.pas
    - SynZip.pas
    - SynCrtSock.pas
    - SynWinWock.pas
    http://synopse.info/fossil/wiki?name=Downloads

}

I think I will add something similar to the official mORMot code.
What is to be used to refer to DWS project itself?

I suspect we may have much better performance, by maintaining an in-memory cache of the .dws scripts AST.
Is it difficult to implement?

Eric · 2012-11-28 16:25:23

ab wrote:

You can put a reference to http://synopse.info and the original sample source, like this.

Done!

ab wrote:

I think I will add something similar to the official mORMot code.
What is to be used to refer to DWS project itself?

This should be enough

DelphiWebScript project is released under the Mozilla Public License v1.1

Source repository and downloads
http://code.google.com/p/dwscript/

Project website and blog
http://delphitools.info/dwscript

I suspect we may have much better performance, by maintaining an in-memory cache of the .dws scripts AST.
Is it difficult to implement?

It is already implemented in the demo

A limitation is that the cache currently doesn't expire, doesn't have memory use limitations (once a file is cached, it stays in forever), and won't be purged if the underlying dws files are changed.
Another limitation is that there is a single compiler instance, but that'll only matter during warmup of websites that have legions of large dws files accessed in random order.

Eric · 2012-11-29 09:15:10

Eric wrote:

(once a file is cached, it stays in forever), and won't be purged if the underlying dws files are changed.

That limitation is now gone from the demo (added basic support for FindFirstChangeNotification to automatically flush the cache)

ab · 2012-11-29 09:30:41

Nice.

I've seen that you added compilation optimization flags.

What about the exposed APIs to the .dws scripts?
What I like with php is that you have a lot of APIs/functions at hand.
The same is necessary for .dws scripts to work on real projects.

Eric · 2012-11-29 10:03:57

Currently the demo exposes only the internal functions, so this is fairly limited indeed.

Adding the JSON connector and a new restricted file I/O library should be fairly easy. There is also the COM connector, though this one would have to be optional since it breaks out of the sand-box in a rather unlimited fashion.

DB library is also a must, though that has to be developed.

And then there is everything else... so it's not a php competitor in terms of libraries just yet

One could also expand the RTTI environment to provide light-weight exposure of Delphi stuff, however that's problematic in that Delphi classes aren't garbage-collected and usually quite fragile, so it's easy to leak, create AV, etc. which isn't very desirable in a web server. This is also a reason why I haven't devoted too much time in DWS to RTTI, the Delphi RTL & VCL just isn't solid enough for abuse.

ab · 2012-11-29 13:19:41

So perhaps this is where mORMot classes may be useful...
You have a lot of stuff here, like ORM, DB access, logging, caching, interface-based services...
We use JSON for datamarshaling of value objects, so could be very easy to handle such content with dws memory model.
That is, do not use Delphi classes, but interfaces for defining services, sessions to handle instance lifetime, and JSON/records/arrays for COW value objects.
Perhaps some kind of TSQLRecord for DWS may help work with shared ORM code.

I would like to make it as generic as possible, in order to be able to use almost the same classes on both sides, server (via in-memory dws execution) and client (either in Delphi process or after JavaScript compilation), when it comes to shared business logic and communication.
I would like to maximize code re-use, and minimize APIs divergences.

Eric · 2012-11-29 14:08:14

ab wrote:

So perhaps this is where mORMot classes may be useful...
You have a lot of stuff here, like ORM, DB access, logging, caching, interface-based services...

Indeed!

IME (wrapping classes for use at work), a lot of the time spent in exposing classes and function to scripts actually goes to the securing more than the wrapping, to ensure there are no leaks, that erroneous parameters don't result in AVs, etc.

Also basic things WebResponse/WebRequest in the demo currently use RTTI, but this is essentially a shortcut for the demo, as RTTI can quickly end up exposing things that are not safe at all.

ab wrote:

That is, do not use Delphi classes, but interfaces for defining services, sessions to handle instance lifetime, and JSON/records/arrays for COW value objects.

I haven't really investigated what is available in mORMot so far, but I assume a lot of the work is already done!

Extending the demo to integrate as a service instead, or to have managed sessions, Windows-based authentications, etc. would be nice.

ab · 2012-11-29 14:30:24

Eric wrote:

Extending the demo to integrate as a service instead, or to have managed sessions, Windows-based authentications, etc. would be nice.

Then, you will need the mORMot core, i.e. mORMot.pas, and not only SynCrtSock.

It features sessions, Windows based auth, caching, logging, RESTul routing, service safe parameters marshaling and execution model, and so on.

Eric · 2012-12-03 11:19:48

ab wrote:

Then, you will need the mORMot core, i.e. mORMot.pas, and not only SynCrtSock.

These are not part of the stable release 7z (http://synopse.info/fossil/wiki?name=Downloads), is it normal?

Also is there a minimal SSPI demo? Something minimalistic above SynCrtSock, that would allow IE or Chromium to perform NTLM authentication.

It looks like the function ServerSSPIAuth would be the key for that?

Eric · 2012-12-03 12:59:20

Eric wrote:

It looks like the function ServerSSPIAuth would be the key for that?

Got some of it working, however the Process event in THttpApiServer doesn't have connection information.

How do you know when a connection is lost or if a connection has already been authenticated?

AFAICT in mORMot the above is based on an ID and expiring cache, does it mean there is no way to track the connection at the HTTP level?

If so, does it mean NTLM should only be used to establish a connection cookie or something like that?

ab · 2012-12-03 13:26:25

mORMot.pas units and such have been introduced in trunk.
See http://synopse.info/fossil/wiki?name=Get+the+source and http://blog.synopse.info/post/2012/11/2 … e3TomORMot article.

Windows authentication is handled not at HTTP connection level, but at RESTful authentication level, i.e. for all transmission protocols.
See http://blog.synopse.info/post/2012/11/2 … redentials

mORmot does not use any cookie or such: it was designed to be a "clean" RESTFul approach - in fact, REST does not know anything about cookies and such low-level HTTP details.
We implemented a RESTful URI-level authentication scheme - so it is part of mORMot.pas not part of SynCrtSock.pas.
http://blog.synopse.info/post/2011/05/2 … entication

At http.sys level, there is no direct/included NTLM authentication available, as stated by http://www.dupuis.me/node/22
I did not test it, but perhaps newer version of http.sys handle Kerberos authentication directly within the kernel-mode stack. See http://msdn.microsoft.com/en-us/library … L.90).aspx

Eric · 2012-12-03 14:59:44

ab wrote:

At http.sys level, there is no direct/included NTLM authentication available, as stated by http://www.dupuis.me/node/22

I've used that one and http://www.innovation.ch/personal/ronald/ntlm.html to get it working.

However http ntlm authentication happens at the connection level, the problem is that SynCrtSock doesn't provide any connection-level data, so it works only for the first HTTP 1.1 connection from a client, and can't work when multiple clients are involved.
The authentication headers are send through only during the challenge phase, so after the challenge, there is no way to figure out in SynCrtSock if the request comes from an already authenticated connection or from another, not-yet authenticated connection.

So basically, I'm missing a tcp connection handle of some sort (could be any opaque number that uniquely identifies a connection at a given time), and a connection close notification (at which point the previous handle could be recycled) to have the http-level ntlm authentication work.

http-level authentication means you can have single sign-on with IE and Chrome browser (no need to enter user login or password), when the browser is running from an authenticated used session in the domain.

Eric · 2012-12-03 15:10:24

The missing elements are the ConnectionId field of the HTTP_REQUEST structure, and to be notified of a disconnect, one needs to call HttpWaitForDisconnect.

Also would it be possible in a future version of the SynCrtSock to use a record to pass the In/Out fields?

This would allow attaching extra fields (like ConnectionId or the pre-parsed headers), as well as simplify the call (only one pointer on the stack rather than the 8 ones currently required)

At the moment it seems the only way to access ConnectionID is to reimplement THttpApiServer.Execute in a subclass, but of course, that means any future fixes/improvements to THttpApiServer wouldn't be carried over.

Last edited by Eric (2012-12-03 15:12:49)

ab · 2012-12-03 17:58:26

Eric wrote:

Also would it be possible in a future version of the SynCrtSock to use a record to pass the In/Out fields?

Yes, I'll do this. My only concern is that it is specific to the http.sys server, and there is another class available (which uses a thread pool and IOCP - did you try it, by the way? I suspect it has also very good scaling abilities).
Using an opaque structure? I'll see what can be done. Perhaps the easier could be a "Sender" parameter.
A similar technique is already what we made for the service callbacks signature.
I'll do the same at TSQLRestServer.URI() level.

How do you retrieve the socket from this ConnectionId field?

Eric · 2012-12-04 09:26:01

ab wrote:

Using an opaque structure? I'll see what can be done. Perhaps the easier could be a "Sender" parameter.

Sender might not be opaque enough, an Int64 as the ConnectionId is probably enough, and in other classes, it could be a pointer cast to an integer, an integer counter, etc.

ab wrote:

How do you retrieve the socket from this ConnectionId field?

I don't think you're supposed to, and you don't need to AFAIK, all that is needed is knowing if requests are coming down an already authenticated http connection, so the opaque ID is enough.

The connection close notification is to be able to release authentication resources asap rather than wait on a time out.

Chaa · 2012-12-10 08:59:00

It would be very nice to have a connection id for all server implementations.

For servers based on WM_COPYDATA messages it might be client window handle.
For classic TCP/IP server it might be a client IP address and port (sockaddr_in in Accept).
For named pipe based servers it might be GetNamedPipeClientComputerName/GetNamedPipeClientProcessId/GetNamedPipeClientSessionId.

ab · 2012-12-10 20:29:55

Good idea.

We could extend the Remote-Ip header to contain this id.
It is already working for http servers.

Eric · 2012-12-12 16:29:09

At the moment I'm experimenting with a "repackaged" Process method, the strings moved to records:

  TSynHttpServerRequest = record
      InURL, InMethod, InHeaders, InContent, InContentType: RawByteString;
      ConnectionID : Int64;
  end;

  TSynHttpServerResponse = record
     OutContent, OutContentType, OutCustomHeader: RawByteString;
  end;

  TOnHttpServerRequest = function(
      const inRequest : TSynHttpServerRequest;
      var outResponse : TSynHttpServerResponse ) : cardinal of object;

The call overhead is non-measurably lower, but having records will open up adding methods for common stuff (f.i. add standard expiration/last-modified headers, etc.)

Also, currently the compression is set for the whole server, but a lot of data is already compressed (such as png or jpg images), so maybe it could be a boolean flag in the response instead?
Other data could benefit from the deflate being cached on the server-side (thus returning pre-compressed content, rather than re-compressing each time).

ab · 2012-12-12 20:43:49

1. I would rather use a class than a record as parameter.
It will allow to create an abstract class, then overridden version depending on the exact server implementation.
(this is how I finally implemented stubs Executes() method - see http://synopse.info/forum/viewtopic.php?pid=5873#p5873 )

We do not have to forget that SynCrtSock has two HTTP servers, for instance:
- THttpApiServer based on http.sys;
- THttpServer calling directly the socket library, via a thread pool and IOCP.
I suspect even the 2nd one is very efficient (more than Indy, by design).

Having parameters sent as class can work with both kind of server implementation, in a "generic/abstract" way.

2. See how function CompressDataAndGetHeaders() is implemented: it will compress only TEXT/.... or APPLICATION/JSON... or APPLICATION/XML... content. And only if size > COMPRESS_MIN_SIZE, i.e. 1KB in the current version (it refers to IP packet size to guess if compression is worth it).
So I do not understand your concern about png/jpg for instance.

For mORMot, i.e. pure Delphi client, we prefer using our SynLZ compression scheme, which is much faster than deflate/zip - so caching will probably won't be worth the price, in this case.
Server-side cache can be done, but perhaps using direct file sending and file-based cache. There is improvement possible, here, for sure, but only for static content I suspect. Caching is always difficult to implement without breaking any expected behavior.

Eric · 2012-12-13 07:02:54

ab wrote:

1. I would rather use a class than a record as parameter.

That would be more rich, but involves more extensive changes. The record is just a way of repackaging the existing stack-based variables, so it has no overhead and is automatically managed.

ab wrote:

We do not have to forget that SynCrtSock has two HTTP servers

Didn't test the second one (that's what we already had, so I was more interested in http.sys), though I made the changes to record to keep the unit in compiling state.

ab wrote:

I suspect even the 2nd one is very efficient (more than Indy, by design).

The one we have on sockets + thread pool is within a few percents of IIS & Apache & THttpAPIServer, in practice, above a certain performance level, I suppose the sockets API and Windows networking stack are more a bottlenecks than the http server is.

ab wrote:

2. See how function CompressDataAndGetHeaders() is implemented:
it will compress only TEXT/.... or APPLICATION/JSON... or APPLICATION/XML... content.
[...]So I do not understand your concern about png/jpg for instance.

Hmm, didn't spot that, I just assumed from the name it was compressing everything, maybe it should be renamed to CompressTextDataAndGetHeaders() to make it more obvious and one doesn't have to look at the implementations?

For mORMot, i.e. pure Delphi client, we prefer using our SynLZ compression scheme

Been using LZRW1 for that here, any idea how it compares? (I'll probably give it a whirl, there is a lzrwk1h unit hanging in the Delphi world, which we used a long time ago, but the implementation was fully rewritten since)

There is improvement possible, here, for sure, but only for static content I suspect.

Would mostly be for static html, js & css. The js & css files are typically minified and aggregated server-side here, so each page ideally has only one js and one css file linked in.
Server-side, the js & css files are non-minified, commented versions. This allows to switch to client-side debuggable js with a simple switch that turns off the minification+aggregation.

ab · 2012-12-13 09:37:56

The THttpServer class uses a thread pool and IOCP.
Did your own server uses IOCP?
You are right, I suspect http.sys version will scale better, especially when serving static content.

I'm quite sure LZRW1 will be slower than our SynLZ implementation, which is even faster than the well-known LZO for compression time (but LZO decompress faster than SynLZ).
So for server side compression, SynLZ sounds like the best match.
See http://blog.synopse.info/post/2010/06/2 … sion-units

Using a parameter defined as a class has a very low overhead, in fact (from real profiling on our stubbing routines).
I'll see how to implement it.

Eric · 2012-12-14 06:49:17

ab wrote:

Did your own server uses IOCP?

Initially it was using events & signaling, got changed a couple years ago when I introduced IOCP-based worker threads and futures (aka promises).
Currently our DB layer is asynchronous too, and a few background caching and UI tasks have been converted too.

I'm quite sure LZRW1 will be slower than our SynLZ implementation

I tested here, and indeed it is! well done!
The compression is also better in all scenarios I tested (sometimes not by much, but always better).
I've just replaced LZRW1 with SynLZ for the internal test middle-ware here, it should be seeing a few gigabytes of LAN traffic in the next days

Unless I misunderstood SynLZ is LZab? (lempel-ziv-arnaud-bouchez i.e designed by you?)

a hashing limitation makes SynLZ sometimes unable to pack continuous blocks of same byte

What does that mean exactly? That it will crash or corrupt, or just that the compression ratio won't be good?

Using a parameter defined as a class has a very low overhead, in fact (from real profiling on our stubbing routines).
I'll see how to implement it.

Thinking on it, maybe the current THttpApiServer class could remain in a similar or "low level no bells nor whistles" fashion, and a new subclass be introduced with the extra http response/request class?

This would also have the benefit of "opening up" THttpApiServer a bit for subclassing (currently it can't really be sub-classed, as all the logic is private or in SynCrtSock's implementation)
And with the above in mind, it would be nice if the Process event weren't called directly, but rather called from a DoProcess virtual method, with would facilitate subclassing and adding custom processing before/after/in-place-of the request (such as standard http logs)

ab · 2012-12-14 09:08:58

Eric wrote:

Currently our DB layer is asynchronous too, and a few background caching and UI tasks have been converted too.

This is impressive.
I would like to implement such a behavior in mORMot.

Eric wrote:

Unless I misunderstood SynLZ is LZab? (lempel-ziv-arnaud-bouchez i.e designed by you?)

Yes, this is a LZ "classic" algorithm, but what makes it unique is the implementation pattern, like hashing on both sides, resulting in very good compression speed, and "symmetric" decompression speed. You have both pascal and very optimized assembler versions. Pascal version is a bit slower, but also stable.

Eric wrote:

What does that mean exactly? That it will crash or corrupt, or just that the compression ratio won't be good?

I've have seen some border-side effects where compression ratio is not as good as it should (e.g. for the last data blocks).

Eric wrote:

Thinking on it, maybe the current THttpApiServer class could remain in a similar or "low level no bells nor whistles" fashion, and a new subclass be introduced with the extra http response/request class?

Good idea.
I'll see what can be done.

Eric · 2012-12-18 15:54:55

Any suggestions for a web server stress tool?

I had been using JMeter so far, but the current HttpAPI Server + DWScript is too fast on simple queries compared to JMeter, and I'm seeing a 10 to 1 cpu usage (10 for JMeter, 1 for the server), and I don't have 10 client machines to use to balance the client load

mpv · 2012-12-19 10:20:35

The only good tools I found for stress testing is <a href="http://en.wikipedia.org/wiki/ApacheBench"> ApacheBench </a>. Work good under Unix/linux envirovment.
For Windows there is some limitation:
1) work only for localhost
2) only up to 1000 concurrent connection
3) bad performance under Windows virtual machine (can't test on Linux virtual machine yet)

General limitation:
1) Impossible to test mORMot with authentification turned on, so I disable auth in mORMot for stress testing (ab support only basic HTTP auth)
2) Can test only one request type in one time - but it enough to stress testing
3) Can test only HTTP mORMot server

For best mORMot performance you must set number of mORMot threads = number of core x2 (not CPU but core)
If use TSynLog, when set TSynLog.Family.PerThreadLog := ptOneFilePerThread; in other case (ptMergedInOneFile, ptIdentifiedInOnFile) you get custom exception in hard load multithreading (tested in version 1.17)

Another tools I use is MS Visual Studio - it have "test project" feature, but in this case I need 5 client machines mORMot is VERY FAST!

Eric · 2012-12-19 11:20:47

mpv wrote:

3) bad performance under Windows virtual machine

Yep, noticed that.

Also it is single-threaded, and in my tests it reached 100% CPU utilisation of its core (ie. 25% on my quad-core), without having the Synopse web server maximized... so it was also measuring its own performance as a client rather than that of the server.

I tried to alleviate it by running it from different client machines at the same time, but that's not very practical.

mORMot is VERY FAST!

Yep!

Problem is, it's so fast we can't claim it's faster than the rest, because test clients aren't fast enough to differentiate...
Making a custom test client could alleviate that, but then it would raise issues of fairness of the test...

mpv · 2012-12-19 12:20:25

Try it under Linux - it much faster. In my case I have 2 computer - one with mORMot server + database engine and one with Linux - in case I test mORMot with database access ApacheBench utilize 100% of mORMot server CPU in this configuration.
I run something like this:
ab.exe -c900 -n50000 -p getOneRecFromBigTable.json -T"application/json" "http://mash-w7:888/m3/AS/runList"

Last edited by mpv (2012-12-19 12:22:45)

ab · 2012-12-19 13:40:55

mpv wrote:

If use TSynLog, when set TSynLog.Family.PerThreadLog := ptOneFilePerThread; in other case (ptMergedInOneFile, ptIdentifiedInOnFile) you get custom exception in hard load multithreading (tested in version 1.17)

Could you give some feedback about ptIdentifiedInOnFile?
It could be the best way of getting multi-thread information, in one log file.
Where are exceptions raised? If the critical section used to "protect" the writing is not enough (it should be IMHO), should we enhance the section coverage?
Of course, ptOneFilePerThread is perfectly thread-safe and scaling on heavy load: there is no lock nor contention possible (but the hard drive access at OS level).

Eric wrote:

Problem is, it's so fast we can't claim it's faster than the rest, because test clients aren't fast enough to differentiate...

Fair enough.

I suspect that in order to reach server limits, you would need a lot of small queries (less than 200 bytes).

Testing with some Linux boxes (or virtual machines) could be a good idea.
You can even run it without installing them.

mpv · 2012-12-19 18:41:27

Could you give some feedback about ptIdentifiedInOnFile?

I test latest version a little bit. Seems ptIdentifiedInOnFile is more stable then old PerThreadLog = False ( PerThreadLog = False always put server down in 24 thread and million request stress test)
But I found some strange issue - now ApacheBench don't work if I do HTTP request without Connection: Keep-Alive in header (in prev 1.17 version everything work fine) - I'll check it carefully tomorrow
Another issue - there is Accept-Encoding header in server response - it's must not be present in response - I write ticket.

ab · 2012-12-19 22:13:39

I've fixed the ticket.
But I still do not understand what changed since revision 1.17 in fact, which may have break something.

mpv · 2012-12-20 10:37:02

ApacheBench don't work if I do HTTP request without Connection: Keep-Alive in header

I compile mORMot with old sources (1.17) - and the problem is the same. So mORMot is OK, 99% that it because of some Windows updates applied to my computer. On other computer everything work as expected on all mORMot versions.
Also I confirm what using ptIdentifiedInOnFile is stable and very good for understanding - match better then log per thread. Thanks for good solutions!

yogiyang · 2012-12-21 06:56:19

Eric wrote:

Actually I hope to get away with a "stupid" parser that would leverage the full SQL parser of SQLite, something like:
start "parsing" sql when encountering "select"
grab everything up to the end of the statement, noting variable references and expression in passing (maybe prefixed with a '@'), and replace them with parameters
feed the SQL to SQLite, have it prepare the query, report errors
assume sqlite3_column_decltype is meaningful enough to be used for strong typing

Eric,

I don't know but this ready to use SQL parser probably help you. http://jansfreeware.com/jansql.zip

HTH

Yogi Yang

Eric · 2013-01-04 16:43:58

First version is out there

You'll need trunk latest DWScript & Synopse files, Win2008 or Vista or better (http.sys 2.0 requirement)

I made precompiled binary with mini demo site for those that want to test directly:
http://code.google.com/p/dwscript/downl … Server.zip
For config options and server doc, see http://code.google.com/p/dwscript/wiki/WebServer

At the moment telling notepad++ that .dws files are highlighted as html works "almost" well enough.

Demo includes a basic memory + cpu usage monitor, a minimal DB (SQLite) exemple, and some special cases.
Throttling (CPU and Bandwidth), SSL and NCSA logging are supported too.

mORMot Open Source

#1 2012-11-26 16:43:15

HTTP Server / DWScript

#2 2012-11-27 07:44:14

Re: HTTP Server / DWScript

#3 2012-11-27 08:45:49

Re: HTTP Server / DWScript

#4 2012-11-27 09:33:00

Re: HTTP Server / DWScript

#5 2012-11-27 11:05:30

Re: HTTP Server / DWScript

#6 2012-11-27 11:26:22

Re: HTTP Server / DWScript

#7 2012-11-27 12:31:12

Re: HTTP Server / DWScript

#8 2012-11-27 12:53:01

Re: HTTP Server / DWScript

#9 2012-11-27 15:19:00

Re: HTTP Server / DWScript

#10 2012-11-27 15:29:45

Re: HTTP Server / DWScript

#11 2012-11-27 15:35:52

Re: HTTP Server / DWScript

#12 2012-11-27 15:48:31

Re: HTTP Server / DWScript

#13 2012-11-27 15:51:45

Re: HTTP Server / DWScript

#14 2012-11-27 16:09:59

Re: HTTP Server / DWScript

#15 2012-11-27 16:26:45

Re: HTTP Server / DWScript

#16 2012-11-28 09:10:35

Re: HTTP Server / DWScript

#17 2012-11-28 15:46:06

Re: HTTP Server / DWScript

#18 2012-11-28 16:02:06

Re: HTTP Server / DWScript

#19 2012-11-28 16:25:23

Re: HTTP Server / DWScript

#20 2012-11-29 09:15:10

Re: HTTP Server / DWScript

#21 2012-11-29 09:30:41

Re: HTTP Server / DWScript

#22 2012-11-29 10:03:57

Re: HTTP Server / DWScript

#23 2012-11-29 13:19:41

Re: HTTP Server / DWScript

#24 2012-11-29 14:08:14

Re: HTTP Server / DWScript

#25 2012-11-29 14:30:24

Re: HTTP Server / DWScript

#26 2012-12-03 11:19:48

Re: HTTP Server / DWScript

#27 2012-12-03 12:59:20

Re: HTTP Server / DWScript

#28 2012-12-03 13:26:25

Re: HTTP Server / DWScript

#29 2012-12-03 14:59:44

Re: HTTP Server / DWScript

#30 2012-12-03 15:10:24

Re: HTTP Server / DWScript

#31 2012-12-03 17:58:26

Re: HTTP Server / DWScript

#32 2012-12-04 09:26:01

Re: HTTP Server / DWScript

#33 2012-12-10 08:59:00

Re: HTTP Server / DWScript

#34 2012-12-10 20:29:55

Re: HTTP Server / DWScript

#35 2012-12-12 16:29:09

Re: HTTP Server / DWScript

#36 2012-12-12 20:43:49

Re: HTTP Server / DWScript

#37 2012-12-13 07:02:54

Re: HTTP Server / DWScript

#38 2012-12-13 09:37:56

Re: HTTP Server / DWScript

#39 2012-12-14 06:49:17

Re: HTTP Server / DWScript

#40 2012-12-14 09:08:58