mORMot Open Source

@itSDS

Thanks for these details.
For ETag hash precalculation by the TAsset class is use crc32c function call (as it implemented in TSQLRestServerURIContext.Returns in mORMot.pas, line 38826).

Seems like you crc32c pre-calculation when you run assetslz.exe on Pre-build stage for file Tests\Assets\index.html is differ from expected by bdd-scenario: "6FECA093"

@ab

Is it possible that crc32c call (implemented as crc32cfast or crc32csse42) can return different results for x86/x64 platforms or on different CPU or differ for Debug/Release mode?

In the bdd-scenarios I used constant $6FECA093 to compare it against ETag's pre-calculated for index.html file when synlz-archive created on Pre-Build stage.
But on itSDS' platform the crc32c returns another constant $79CACBF7 for the same Tests\Assets\index.html. I saw that you using the same technique when run some tests and compare results against constant crc/hash. How is it possible that crc32c result is differ form expected constant?

@itSDS,

Thanks! I've include this tool fix into Release 1.3.

Unfortunately the current Mustache Template Engine implementation is heavily coupled with files layout and not diskIO-agnostic even on abstract levels.
I provided my suggestions of such refactoring through inheritance, but Arnaud decide to go in more decoupled way through delegation.
When he finished I can create boilerplate delegate to feed Mustache Engine with View and Partials from mem-cached boilerplate assets, but not from files.

As for now I do the next thing to get the same result:

1. Segregate all non-static *.html and *.partial (or any other mustache-related templates) into Views directory (put it near Assets dir).

2. Embed all these assets into exe with another three additional Pre-Build events

   ..\Tools\assetslz.exe Views Views.synlz
   ..\Tools\resedit.exe $(INPUTNAME).res rcdata VIEWS Views.synlz
   DEL Views.synlz

3. In the server startup code I unpacked all Views somewhere on production folder (when you restart your server all unpacked files will not be overwritten if not changed)

   var
     Views: TAssets;
   ...
     Views.Init;
     Views.DecompressFromResource('Views');
     Views.SaveAssets(GetViewsDir);
   
     FApplication := TMyApplication.Create;
     FApplication.Start(FServer, GetViewsDir);

4. Then create TMVCApplication with Mustache Views and Partials loaded from this folder:

     TMyApplication = class(TMVCApplication, IMyApplication)
     protected
       function CreateViews(const AFolder: string): TMVCViewsAbtract;
     public
       procedure Start(Server: TSQLRestServer; const Folder: string = ''); reintroduce;
   ...
   procedure TMyApplication.Start(Server: TSQLRestServer; const Folder: string);
   begin
     inherited Start(Server, TypeInfo(IMyApplication));
     FMainRunner := TMVCRunOnRestServer.Create(Self, fRestServer, '',
       CreateViews(Folder), [{publishStatic, }bypassAuthentication]);
   ...
   function TMyApplication.CreateViews(const AFolder: string): TMVCViewsAbtract;
   var
     Params: TMVCViewsMustacheParameters;
   begin
     FillChar(Params, SizeOf(Params), 0);
     with Params do
     begin
       Folder := AFolder;
       FileTimestampMonitorAfterSeconds := 0;
       ExtensionForNotExistingTemplate := '';
       Helpers := TSynMustache.HelpersGetStandardList;
     end;
     Result := TMVCViewsMustache.Create(FFactory.InterfaceTypeInfo, Params,
       (fRestModel as TSQLRestServer).LogClass);
   end;

@ab

Can you add mORMot version console logout during test suite execution? Maybe right after "Generated with: * compiler" line.

Release 1.2

Minor Accept-Encoding fix and code refactoring:

• fix "Accept-Encoding" parsing when gzip in the end of encodings list

• make Pre-Build events notice more visible in tests and demo

• minor code refactoring

@isSDS,

Thanks for feedback: I've changed the Pre-Build Notice to be more visible.

What version of mORMot framework did you use (you can take it from SynopseCommit.inc)?
As I understand, you test it with Rad Studio Berlin 10.1 - this mean that you didn't use http://synopse.info/files/mORMot.7z stable version (it's not compatible with 10.1 due to SynCommons (line 22002) GetDelphiCompilerVersion bug, missed {$else} condition for unknown compilers).

I do my tests on Win 8.1 Pro + Rad Studio XE8 22.0.19908.869

• mORMot 1.18.2688 + Boilerplate 1.0.0 = All BDD Scenarios Passed

• mORMot 1.18.2688 + Boilerplate 1.0.0 = All BDD Scenarios Passed

• mORMotNightlyBuild.zip 1.18.2734 + Boilerplate 1.1.0 = All BDD Scenarios Passed

• mORMotNightlyBuild.zip 1.18.2734 + Boilerplate 1.1.0 = All BDD Scenarios Passed

Can you localize what exact scenarios of TBoilerplateHTTPServerShould.EnableCacheByETag failed?
To do this just comment procedure EnableCacheByETag; and add it to the top of class definition with following protected:

  TBoilerplateHTTPServerShould = class(TSynTestCase)
    procedure EnableCacheByETag;
  protected
    procedure CallInherited;
    procedure ServeExactCaseURL;
    procedure LoadAndReturnAssets;
    procedure SpecifyCrossOrigin;
    procedure SpecifyCrossOriginForImages;
    procedure SpecifyCrossOriginForFonts;
    procedure SpecifyCrossOriginTiming;
    procedure DelegateBadRequestTo404;
    procedure DelegateForbiddenTo404;
    procedure DelegateNotFoundTo404;
    procedure SetXUACompatible;
    procedure SetP3P;
    procedure ForceMIMEType;
    procedure ForceTextUTF8Charset;
    procedure ForceUTF8Charset;
    procedure ForceHTTPS;
    procedure SupportWWWRewrite;
    procedure SetXFrameOptions;
    procedure SupportContentSecurityPolicy;
    procedure DelegateBlocked;
    procedure SupportStrictSSL;
    procedure PreventMIMESniffing;
    procedure EnableXSSFilter;
    procedure DeleteXPoweredBy;
    procedure FixMangledAcceptEncoding;
    procedure SupportGZipByMIMEType;
    procedure ForceGZipHeader;
    procedure SetCacheNoTransform;
    procedure SetCachePublic;
//    procedure EnableCacheByETag;
    procedure EnableCacheByLastModified;
    procedure SetExpires;
    procedure SetCacheMaxAge;
    procedure EnableCacheBusting;
    procedure SupportStaticRoot;
    procedure DelegateRootToIndex;
    procedure DeleteServerInternalState;
    procedure DelegateIndexToInheritedDefault;
    procedure Delegate404ToInherited_404;
  end;

This will give you an ability to experiment only with this feature.
All debugger interruptions show failed test cases in stack trace windows.

I will check BDD scenarios additionally with Berlin 10.1.

@ab,
If some checks are failed in test case - how show their Check(condition, text) this text during console logout?
Because "!  - Enable cache by E tag: 4 / 15 FAILED  28.34ms" is not detailed enough.

Hi, ab,

Sure, why not, but you have to help me resolve the next three concerns:

1. You will use Boilerplate, not BoilerPlate naming variant - this is one solid word fully aligned with huge boilerplate community

2. Please confirm that you also helps me to update (if needed) both tools required for resource embedding assetslz and resedit - they both must serve all suggested compilers (my concerns mostly following Linux support, I have no understanding how *.res resources created/updated/embedded/loaded there). Without them big piece of valuable mem-cached code will be lost. Also without this tools it will be unable to build and deploy applications on cloud services in Instant Deployment and single-file-distribution manner. Just image fully operable HTTP Server hosted on cloud without needs in any mounted disks (SQLite 3 / Logs / Shutdown Session States are easily segregated to other instances). So synlz-packed mem-cached path is important for me.

3. There is another big issue, which still required unnecessary DiskIO operations on server: you designed TMVCViewsMustache and even higher level TMVCViewsAbtract classes in mORMotMVC.pas with DiskIO / files existence in-mind (properties like .ViewTemplateFolder, .ViewStaticFolder, .ViewTemplateFileTimestampMonitor) - such decision even on an abstract level not smoothly aligned with mem-cached boilerplate paradigm. Right now to fully use of TMVCApplication functionality, I have to decompress all Mustache-related *.html and *.partial assets somewhere on production disk (usually it is [UserProfile]/AppData/Roaming/[MyApp] directory) and then load unpacked files again in memory with standard TMVCViewsMustache constructor. Maybe together we can redesign these two mustache classes faster or if you busy I can create TMVCViewsMustache descendant (smth. like TMVCAssetsMustache or TMVCViewsBoilerplateMustache) which can work with mem-cached mustache templates/partials directly. But descendant still contains not-needed properties related to files, folders, all this diskIO not mem-cached stuff.

Oh, please be informed that all test-coverage build in modern user-centered behavior-driven design manner (features, scenarios, cases, steps, *should, given*, when*, then*, you know - all this stuff...), not plain TDD. Will it be an issue? I want to keep this tests for future safe refactoring and degradation prevention.

The HTML Boilerplate is actively evolving (jQuery 3.0.0 integration is coming) so how I can keep it actual and aligned with this best HTTP practices and libraries If you merge all this code into mORMot trunk?

Thanks

edwinsn,

Yes, you can safely replace your TSQLHttpServer with inherited TBoilerplateHTTPServer.

I'm not sure about Linux because I used RT_RCDATA resource embedding into exe,

Please let me know when you check it under Linux or help to to make it more cross-platform

I tested it and build it with XE8.

I will be appreciate, if you test it with other compilers, I'm not sure when Pre-Build events pipeline was introduced in Delphi.

Hi ab,

Now all resources can be synlz-packed, embedded and returned in memcached manner with TBoilerplateHTTPServer class instance.

Even more: the assets can be pre-compressed and cached as files on production server, this allows delegate file transfers to low-level HTTP api (like http.sys) even gzipp'ed variants!

And now you can embed and unpack on production your .map / .mab project files (single file run/distribution strategy).

Unfortunately there is no way to ask TSynLog* get info not from disk but from or in-memory *.mab file during stack-trace logout.
Maybe ab will add this feature in the next releases.

Hi ab,

Thanks for fast replay,
Unfortunatelly _ObjFast(['names', _ArrFast(['ab', 'Eugene']) not appropriate because OneFieldValues() works with TRawUTF8DynArray, but not a static content.
Seem like usage of .InitArrayFrom is that what I need, but unfortunately it doesn't exist in 1.18

I'm just playing with Auth* layer of framework and found that returning a simple list of logons in form of {'logons' : ['L1', 'L2', ...]} is not trivial.

Here is a tiny example:

  TTestServer = class(TSQLRestServerFullMemory)
    procedure Logons(Ctxt: TSQLRestServerURIContext);
  end;

procedure TTestServer.Logons(Ctxt: TSQLRestServerURIContext);
var
  Logons: TRawUTF8DynArray;
  Values: variant;
begin
  OneFieldValues(TSQLAuthUser, 'LogonName', '', Logons);
//  TDocVariantData(Values).InitArrayFrom(Logons, JSON_OPTIONS_FAST);
  Ctxt.ReturnsJson(_ObjFast(['logons', Values]));
end;

  FModel := TSQLModel.Create([]);
  FServer := TTestServer.Create(FModel, True);
  FServer.ServiceMethodByPassAuthentication('logons');
  FServer.CreateMissingTables;

I commented error line due to:
[dcc32 Error] TestServer.pas(26): E2003 Undeclared identifier: 'InitArrayFrom'

So, when I'm requesting /root/logons in browser, I'm expecting JSON logons in the form mentioned above.
Do you have any replacements for not existed InitArrayFrom?

Thanks

Hi esmondb,

If you mean useHttpApiRegisteringURI HTTP server (THttpApiServer) based on http.sys windows kernel library - it doesn't gzip files too.
This is because of performance optimization introduced by mORmot overlord. He preferred delegate file transfer to windows kernel as the best possible performance solution.
But http.sys doesn't compress files too.

As for your IIS suggestion to use StaticCompressionModule (based on Inetsrv\Compstat.dll) this is IIS specific feature available in IIS only, but not in mORMot framework.
mORMot doesn't rely or require IIS presence on windows to run.

It's interesting that 30 - MVC Server example based on mustache templates gzip all content:

Request:
  Host: localhost:8092
  User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  Accept-Language: en-US,en;q=0.5
  Accept-Encoding: gzip, deflate
  Etag: "23C0189A"

Responce:
  Accept-Encoding: gzip
  Content-Encoding: gzip
  Content-Length: 4258
  Content-Type: text/html; charset=UTF-8
...

So the mustache HTTP processor have to compute all page html/js/css content from templates and give it back as a cached data stream but not as a static file.
This decision excludes us from kernel windows http.sys sending but gives us gziped computed content based on mustache.

Interesting that blog.css request to /blog/.static/blog.css is gzipped too from 2.35 Kb file content to 0.98 Kb of actual network traffic.
The answer is that blog.css bypass windows kernel file sending and Context.OutContentType in THttpApiServer.Execute not equal HTTP_RESP_STATICFILE but "text/css".

Interesting, how is .static mustache data cached (like blog.css file):

• Computed each time in every request to check against crc32c ETag value

• Has some timeout to save CPU for unnecessary computation and do it periodically

• Checks .static/* file date/time modification to update memory cache and recompute ETag

• Subscribed to file system and hooked when file modified to update internal cache and recompute ETag

• Has some lazy and permanent crc32c computation which is done only ones per first request and then always give the same static data from memory

Here is the Microsoft note for previous API version:

From the IIS perspective, according to this: there are no ways to remove Microsoft-HTTPAPI/2.0 postfix.

Hi, ab,

I use single exe-file for prod server + single db-file for database - interface and implementation is merged in one simple, light, portable solution (just run exe-file on amazon node or any other dedicated service and Wuala! is less than a second production deployment has been complete).

If this solution will be available to clients without load-balancer: any hacker will know OS, framework, and used core windows library just by reading the "Server: " line in any browser.

Do you think that there are no security concerns here and Windows (or http.sys or mORMot) zero-day vulnerabilities are not applicable?

I have an answer here. What is your experience and feedback from other developers following not changeable "Server: " post-fix in production systems?
It will be nice if I'm just being to paranoid (I have no so many real hackers attack to be ensure that most of security breaches are closed).

Thanks

P.S. as I understand this synopse.info forum interface is "server:lighttpd/1.4.26" but forum functional implementation is hidden somewhere deeper.