Memcached Boilerplate HTTP Server for Synopse mORMot Framework

Eugene Ilyin · 2016-06-08 23:32:25

Hi all,

I'm happy to introduce you Boilerplate HTTP Server for Synopse mORMot Framework

GitHub: https://github.com/eugeneilyin/mORMotBP

Releases: mORMotBP/releases

Features

Fully aligned with HTML5 Boilerplate HTTP configs with more than 50+ options and properties.
Designed for Delphi 6 up to Delphi 10.4.1 Sydney, Kylix 3 (over CrossKylix), and FPC, targeting Windows or Linux, 32-bit or 64-bit architectures.
Embed all static assets into application file as highly compressed synlz archive (see assetslz and resedit tools). This allows to build single file distribution fully aligned with Instant Deployment approach.
Save your cloud hosting Disk IO operations by return all static assets from mem-cached repository.
Don't spend CPU cycles to assets compression, all your assets will be pre-compressed by assetslz tool.
Support GZip Level 9 maximum compression.
Support Zopfli compression (save up to 5-15% of traffic and delivery time compared to max GZip compression).
Support Brotli compression as per RFC 7932 (save another 15%-25% of traffic and delivery time compared to Zopfli compression).
Support Dynamic Brotli compression (save about 25% CPU usage compared to GZip on 64-bit systems and 10% less delivery time and traffic utilization).
Ability to delegate compressed files transferring to low-level HTTP.sys high-performance library (see .StaticRoot property) and free your server threads for more interesting work.
Support ETag/Last-Modified or more user-friendly Last-Modified/If-Modified-Since cache strategies.
Server-side Expires or Cache-Control: max-age cache strategies.
Different cache busting strategies (see bpoEnableCacheBusting and bpoEnableCacheBustingBeforeExt options).
Fix well-known mangled Accept-Encoding values in HTTP headers.
Block access to files that can expose sensitive information (see bpoDelegateBlocked option).
Apply many HTTP headers corrections following HTML5 Boilerplate settings.
Support Content Security Policy Level 2 / Level 3 (see CSP.pas unit for details).
Support external computable Assets and URL requests with content/file transfers and redirects.
Support 1490 MIME Types file extensions combined from IANA, Apache, and Mozilla.
You can safely replace anywhere your TSQLHttpServer with
TBoilerplateHTTPServer = class(TSQLHttpServer)

All code is fully covered with tests supported by the mORMot Test Suite.

Fill free to use or give any feedback!

Last edited by Eugene Ilyin (2020-11-19 21:52:16)

ab · 2016-06-09 07:07:33

This is an awesome project!

On which Delphi version is it tested?
My guess is that there may be some issue with Unicode versions of Delphi, in which string=UnicodeString.

I'll fork and report.

Thanks a lot for sharing.
This is awesome.

Eugene Ilyin · 2016-06-09 11:13:17

I tested it and build it with XE8.

I will be appreciate, if you test it with other compilers, I'm not sure when Pre-Build events pipeline was introduced in Delphi.

edwinsn · 2016-06-09 12:54:48

Thanks for sharing. Can it be used with other html framework other than html 5 Boilerplate?

Last edited by edwinsn (2016-06-09 12:55:20)

AOG · 2016-06-09 13:29:08

I just tested it with OpenUI5.
Works 100% !
With a 500% speed gain !!

Eugene Ilyin · 2016-06-09 13:48:04

Hi, edwinsn,

Boilerplate is just a template.
Good starting point for your web app.

You can use it with any favorite JS (Ember, Redux, Angular 2) or CSS frameworks (Modernizr is already embedded).

What mORMotBP do - is fix mangled GZip encoding, prevent clickjacking, GZip more different content types (with more compression level) then standard mORMot HTTP API, embed all your assets into exe, handle strict ssl, serve cross-origins for images or fonts, delegate assets transferring from application to low-level API, prebuild ETags or content-type definitions to save some CPU time, save your disk IO, give you different html cache strategies "from the box", etc...

For example if you find that usage of ETag can bring you Deployment risks and possible lawsuit from customers in USA or EU, you can switch to another effective cache strategy by changing bpoEnableCacheByETag to bpoEnableCacheByLastModified, etc.

So you can develop your web app locally in debug mode with Mustache templates in Assets folder and then embed everything into single production-ready exe.

Look into BoilerplateHTTPServer.pas TBoilerplateOptions for full list of available fixups and properties.

edwinsn · 2016-06-09 15:43:36

It's defined as

 TBoilerplateHTTPServer = class(TSQLHttpServer)

I see now! Very useful features.

Just one more question - does it work under Linux?

Eugene Ilyin · 2016-06-09 16:26:53

edwinsn,

Yes, you can safely replace your TSQLHttpServer with inherited TBoilerplateHTTPServer.

I'm not sure about Linux because I used RT_RCDATA resource embedding into exe,

Please let me know when you check it under Linux or help to to make it more cross-platform

Eugene Ilyin · 2016-06-09 19:50:07

Release 1.1.0

minor "Cache-Control" parameters order changes
added bpoDelegateIndexToInheritedDefault to delegate "/" to Default() method
added bpoDelegate404ToInherited_404 to delegate "/404" requests to _404() method

The two new features allows you to delegate "/" and "/404.html" rendering to IMVCApplication .Default() and ._404() methods accordingly.

ab · 2016-06-09 21:03:20

What do you think of merging TBoilerPlateHttpServer to the main mORMot trunk?

Your code is of very high value.

Using TBoilerPlateHttpServer as an alternative to TSQLHttpServer could definitively make sense!

To be included in the trunk, I would have to ensure compatibility with all compilers (Delphi and FPC) and also platforms (Windows and Linux).

Eugene Ilyin · 2016-06-09 22:04:29

Hi, ab,

Sure, why not, but you have to help me resolve the next three concerns:

1. You will use Boilerplate, not BoilerPlate naming variant - this is one solid word fully aligned with huge boilerplate community

2. Please confirm that you also helps me to update (if needed) both tools required for resource embedding assetslz and resedit - they both must serve all suggested compilers (my concerns mostly following Linux support, I have no understanding how *.res resources created/updated/embedded/loaded there). Without them big piece of valuable mem-cached code will be lost. Also without this tools it will be unable to build and deploy applications on cloud services in Instant Deployment and single-file-distribution manner. Just image fully operable HTTP Server hosted on cloud without needs in any mounted disks (SQLite 3 / Logs / Shutdown Session States are easily segregated to other instances). So synlz-packed mem-cached path is important for me.

3. There is another big issue, which still required unnecessary DiskIO operations on server: you designed TMVCViewsMustache and even higher level TMVCViewsAbtract classes in mORMotMVC.pas with DiskIO / files existence in-mind (properties like .ViewTemplateFolder, .ViewStaticFolder, .ViewTemplateFileTimestampMonitor) - such decision even on an abstract level not smoothly aligned with mem-cached boilerplate paradigm. Right now to fully use of TMVCApplication functionality, I have to decompress all Mustache-related *.html and *.partial assets somewhere on production disk (usually it is [UserProfile]/AppData/Roaming/[MyApp] directory) and then load unpacked files again in memory with standard TMVCViewsMustache constructor. Maybe together we can redesign these two mustache classes faster or if you busy I can create TMVCViewsMustache descendant (smth. like TMVCAssetsMustache or TMVCViewsBoilerplateMustache) which can work with mem-cached mustache templates/partials directly. But descendant still contains not-needed properties related to files, folders, all this diskIO not mem-cached stuff.

Oh, please be informed that all test-coverage build in modern user-centered behavior-driven design manner (features, scenarios, cases, steps, *should, given*, when*, then*, you know - all this stuff...), not plain TDD. Will it be an issue? I want to keep this tests for future safe refactoring and degradation prevention.

The HTML Boilerplate is actively evolving (jQuery 3.0.0 integration is coming) so how I can keep it actual and aligned with this best HTTP practices and libraries If you merge all this code into mORMot trunk?

Thanks

Chaa · 2016-06-10 07:33:44

I think, that two-file-distribution not much more complicated than single-file-distribution.

Why not use two files: mORMotBPDemo.exe and mORMotBPDemo.assets?

It may be a simpler and crossplatform solution.

ab · 2016-06-10 07:39:57

I guess is that in the .exe you may have your services implementation, which would benefit of being in synch with the actual assets.

Or just put all assets in a .zip file, which could be appended to the executable if needed to have a single file.
SynZip is able to open a .zip file located at the end of the .exe.
It may be an alternative to synlz compression.

@Eugene
1. Boilerplate is just fine - my misTake
2. The idea is to have a dedicated unit, alternative to TSQLHttpServer, with its associated tools. Lazarus or Kylix are able to bind Windows resources, under Linux, directly.
3. The mustache templates would easily be located from resources, or any other mean of storage, using an event function. I will do this modification, which is a need when working with assets.
4. Yes, BDD is easier to follow than plain TDD in spaghetti mode. It is not an issue to use BDD naming: it is a welcomed evolution!
5. Idea is that you could easily fork mORMot, and maintain "your" units within the trunk, just like mpv does for his nice SpiderMonkey stuff.

Eugene Ilyin · 2016-06-10 11:02:31

@ab,

0. Ok then, let's merge this project into main mORMor framework trunk. Assets folder contains all web specific assets required by boilerplate best web app practices (it's a compilation of many years best practices made by web masters community for each element in index.html, 404.html or files/folders)

1. Please let me know when you make (partials, helpers, views) Abstract Mustache Views really abstract from any storage representation (RT_RCDATA, Files, DB, JSON). I mean refactor to more low-concrete level all diskIO specific artifacts like .ViewStaticFolder and others. Will inheritance here will be more appropriate then event delegation?

TMVCViewsAbtract or TCustomMVCViews (Delphi like naming convention)
    .fFactory
    .fLogClass
    .Render(methodIndex, Context, View); virtual; abstract;
    .Create(aInterface: PTypeInfo; aLogClass: TSynLogClass);
    .Factory: TInterfaceFactory

TMVCViewsMustache = class(TMVCViewsAbtract)
    .fViewPartials: TSynMustachePartials;
    .fViewHelpers: TSynMustacheHelpers;
    .fViews: array of record // follows fFactory.Methods[]
      .Mustache: TSynMustache;
      .Template: RawUTF8;
      .MethodName: not TFileName but RawUTF8 or SockString;
    .RegisterExpressionHelpers

TMVCViewsMustacheFiles = class(TMVCViewsMustache)
    .fViews: array of record // follows fFactory.Methods[]
      .Locker: IAutoLocker; // not sure maybe it needs to be in TMVCViewsMustache or even higher
      .SearchPattern: TFileName;
      .FileName: TFileName;
      .ShortFileName: TFileName;
      .FileExt: TFileName;
      .FileTimeStamp: TDateTime;
      .FileTimeStampCheckTick: Int64;
    .ViewTemplateFolder
    .ViewStaticFolder

TMVCViewsMustacheMemory = class(TMVCViewsMustache)

...

TMVCViewsMustacheDB = class(TMVCViewsMustache)

TMVCViewsMustacheJSON = class(TMVCViewsMustache)

TMVCViewsMustacheDoc = class(TMVCViewsMustache)

Segregate the next methods to separate routines (in Web App, SPA's we needn't to have low-level tables on such abstract layer)

    CreateExpressionHelpersCommon or CreateCommonExpressionHelpers
    CreateExpressionHelpersForCrypto
    CreateExpressionHelpersForTables

To use like this:

    MyViews.RegisterExpressionHelpers(CreateCommonExpressionHelpers);
    MyViews.RegisterExpressionHelpers(CreateExpressionHelpersForCrypto);
    MyViews.RegisterExpressionHelpers(CreateExpressionHelpersForTables(...));

What do your think?

2. How SynCommons.ResourceSynLZToRawByteString works on Lazarus and Kylix? If I add/create smth in .rc, .res files on pre-compilations stage, this content will be included/accessible on Linux?

ab · 2016-06-10 11:14:02

1. Perhaps a dedicated class, doing only storage/retrieval of content, would be preferred; and/or a callback event handler.

2. You have AFAIR https://sourceware.org/binutils/docs/bi … ndres.html which does the .rc compilation into .res on Linux.
Then FPC can link the .res as on windows, and ResourceSynLZToRawByteString would work as expected under Linux.
Lazarus has very feature-complete compilation stages, or you may use a separated batch file.

itSDS · 2016-06-12 09:25:04

Running mormot BP Test results in lots of errors

What is wrong ??

Microsoft Windows [Version 10.0.10586]
(c) 2015 Microsoft Corporation. Alle Rechte vorbehalten.

c:\Projekte\delphi\vcl\mORMot\Boilerplate\Tests>mORMotBPTests.exe

   Boilerplate tests suite
  -------------------------


1. Register

 1.1. Boilerplate HTTP server should:
  - Call inherited: 1 assertion passed  8.59ms
!  - Serve exact case URL: 6 / 9 FAILED  9.75ms
!  - Load and return assets: 2 / 3 FAILED  8.38ms
  - Specify cross origin: 3 assertions passed  7.53ms
!  - Specify cross origin for images: 3 / 6 FAILED  8.82ms
!  - Specify cross origin for fonts: 3 / 6 FAILED  9.48ms
!  - Specify cross origin timing: 2 / 4 FAILED  8.79ms
!  - Delegate bad request to 404: 2 / 4 FAILED  22.06ms
!  - Delegate forbidden to 404: 2 / 4 FAILED  9.84ms
!  - Delegate not found to 404: 1 / 4 FAILED  9.36ms
!  - Set XUA compatible: 4 / 8 FAILED  8.49ms
!  - Set P 3 P: 2 / 4 FAILED  7.01ms
  - Force MIME type: 2 assertions passed  8.43ms
  - Force text UT F 8 charset: 4 assertions passed  10.48ms
  - Force UT F 8 charset: 2 assertions passed  7.74ms
!  - Force HTTPS: 1 / 4 FAILED  7.54ms
!  - Support WWW rewrite: 2 / 14 FAILED  9.43ms
!  - Set X frame options: 4 / 8 FAILED  8.62ms
!  - Support content security policy: 4 / 8 FAILED  8.54ms
!  - Delegate blocked: 2 / 6 FAILED  10.30ms
!  - Support strict SSL: 6 / 12 FAILED  11.85ms
!  - Prevent MIME sniffing: 4 / 8 FAILED  7.55ms
!  - Enable XSS filter: 4 / 8 FAILED  8.32ms
!  - Delete X powered by: 2 / 4 FAILED  8.75ms
!  - Fix mangled accept encoding: 10 / 12 FAILED  9.07ms
!  - Support G zip by MIME type: 25 / 28 FAILED  10.35ms
!  - Force G zip header: 3 / 6 FAILED  12.14ms
!  - Set cache no transform: 2 / 4 FAILED  7.54ms
!  - Set cache public: 8 / 16 FAILED  9.08ms
!  - Enable cache by E tag: 8 / 15 FAILED  8.82ms
!  - Enable cache by last modified: 8 / 15 FAILED  8.38ms
!  - Set expires: 14 / 16 FAILED  10.52ms
!  - Set cache max age: 14 / 16 FAILED  9.18ms
!  - Enable cache busting: 4 / 7 FAILED  7.32ms
!  - Support static root: 5 / 6 FAILED  8.55ms
!  - Delegate root to index: 4 / 10 FAILED  8.16ms
  - Delete server internal state: 2 assertions passed  8.46ms
!  - Delegate index to inherited default: 2 / 5 FAILED  9.96ms
!  - Delegate 404 to inherited - 404: 2 / 5 FAILED  8.69ms
  Total failed: 165 / 299  - Boilerplate HTTP server should FAILED  396.98ms


Generated with: Delphi 10.1 Berlin compiler

Time elapsed for all tests: 399.05ms
Tests performed at 12.06.2016 11:22:43

Total assertions failed for all test suits:  165 / 299
! Some tests FAILED: please correct the code.

Done - Press ENTER to Exit

itSDS · 2016-06-12 10:57:38

^^ Found the Error / Forgot the Pre-Build Steps

now the Result is better - but not perfect !

   Boilerplate tests suite
  -------------------------


1. Register

 1.1. Boilerplate HTTP server should:
  - Call inherited: 1 assertion passed  8.14ms
  - Serve exact case URL: 9 assertions passed  20.87ms
  - Load and return assets: 3 assertions passed  28.41ms
  - Specify cross origin: 3 assertions passed  7.13ms
  - Specify cross origin for images: 6 assertions passed  23.34ms
  - Specify cross origin for fonts: 6 assertions passed  22.53ms
  - Specify cross origin timing: 4 assertions passed  16.43ms
  - Delegate bad request to 404: 4 assertions passed  11.51ms
  - Delegate forbidden to 404: 4 assertions passed  12.48ms
  - Delegate not found to 404: 4 assertions passed  11.69ms
  - Set XUA compatible: 8 assertions passed  23.85ms
  - Set P 3 P: 4 assertions passed  15.53ms
  - Force MIME type: 2 assertions passed  6.90ms
  - Force text UT F 8 charset: 4 assertions passed  7.41ms
  - Force UT F 8 charset: 2 assertions passed  7.79ms
  - Force HTTPS: 4 assertions passed  18.79ms
  - Support WWW rewrite: 14 assertions passed  32.38ms
  - Set X frame options: 8 assertions passed  26.92ms
  - Support content security policy: 8 assertions passed  25.33ms
  - Delegate blocked: 6 assertions passed  17.29ms
  - Support strict SSL: 12 assertions passed  24.30ms
  - Prevent MIME sniffing: 8 assertions passed  16.83ms
  - Enable XSS filter: 8 assertions passed  25.72ms
  - Delete X powered by: 4 assertions passed  15.35ms
  - Fix mangled accept encoding: 12 assertions passed  32.90ms
  - Support G zip by MIME type: 28 assertions passed  63.92ms
  - Force G zip header: 6 assertions passed  15.30ms
  - Set cache no transform: 4 assertions passed  24.48ms
  - Set cache public: 16 assertions passed  44.11ms
!  - Enable cache by E tag: 4 / 15 FAILED  28.34ms
  - Enable cache by last modified: 15 assertions passed  23.28ms
  - Set expires: 16 assertions passed  85.08ms
  - Set cache max age: 16 assertions passed  88.45ms
  - Enable cache busting: 7 assertions passed  21.07ms
  - Support static root: 6 assertions passed  22.56ms
  - Delegate root to index: 10 assertions passed  23.03ms
  - Delete server internal state: 2 assertions passed  7.18ms
  - Delegate index to inherited default: 5 assertions passed  18.51ms
  - Delegate 404 to inherited - 404: 5 assertions passed  16.70ms
  Total failed: 4 / 299  - Boilerplate HTTP server should FAILED  970.83ms


Generated with: Delphi 10.1 Berlin compiler

Time elapsed for all tests: 971.97ms
Tests performed at 12.06.2016 12:56:51

Total assertions failed for all test suits:  4 / 299
! Some tests FAILED: please correct the code.

Done - Press ENTER to Exit

Eugene Ilyin · 2016-06-12 15:00:31

@isSDS,

Thanks for feedback: I've changed the Pre-Build Notice to be more visible.

What version of mORMot framework did you use (you can take it from SynopseCommit.inc)?
As I understand, you test it with Rad Studio Berlin 10.1 - this mean that you didn't use http://synopse.info/files/mORMot.7z stable version (it's not compatible with 10.1 due to SynCommons (line 22002) GetDelphiCompilerVersion bug, missed {$else} condition for unknown compilers).

I do my tests on Win 8.1 Pro + Rad Studio XE8 22.0.19908.869

mORMot 1.18.2688 + Boilerplate 1.0.0 = All BDD Scenarios Passed
mORMot 1.18.2688 + Boilerplate 1.0.0 = All BDD Scenarios Passed
mORMotNightlyBuild.zip 1.18.2734 + Boilerplate 1.1.0 = All BDD Scenarios Passed
mORMotNightlyBuild.zip 1.18.2734 + Boilerplate 1.1.0 = All BDD Scenarios Passed

Can you localize what exact scenarios of TBoilerplateHTTPServerShould.EnableCacheByETag failed?
To do this just comment procedure EnableCacheByETag; and add it to the top of class definition with following protected:

  TBoilerplateHTTPServerShould = class(TSynTestCase)
    procedure EnableCacheByETag;
  protected
    procedure CallInherited;
    procedure ServeExactCaseURL;
    procedure LoadAndReturnAssets;
    procedure SpecifyCrossOrigin;
    procedure SpecifyCrossOriginForImages;
    procedure SpecifyCrossOriginForFonts;
    procedure SpecifyCrossOriginTiming;
    procedure DelegateBadRequestTo404;
    procedure DelegateForbiddenTo404;
    procedure DelegateNotFoundTo404;
    procedure SetXUACompatible;
    procedure SetP3P;
    procedure ForceMIMEType;
    procedure ForceTextUTF8Charset;
    procedure ForceUTF8Charset;
    procedure ForceHTTPS;
    procedure SupportWWWRewrite;
    procedure SetXFrameOptions;
    procedure SupportContentSecurityPolicy;
    procedure DelegateBlocked;
    procedure SupportStrictSSL;
    procedure PreventMIMESniffing;
    procedure EnableXSSFilter;
    procedure DeleteXPoweredBy;
    procedure FixMangledAcceptEncoding;
    procedure SupportGZipByMIMEType;
    procedure ForceGZipHeader;
    procedure SetCacheNoTransform;
    procedure SetCachePublic;
//    procedure EnableCacheByETag;
    procedure EnableCacheByLastModified;
    procedure SetExpires;
    procedure SetCacheMaxAge;
    procedure EnableCacheBusting;
    procedure SupportStaticRoot;
    procedure DelegateRootToIndex;
    procedure DeleteServerInternalState;
    procedure DelegateIndexToInheritedDefault;
    procedure Delegate404ToInherited_404;
  end;

This will give you an ability to experiment only with this feature.
All debugger interruptions show failed test cases in stack trace windows.

I will check BDD scenarios additionally with Berlin 10.1.

@ab,
If some checks are failed in test case - how show their Check(condition, text) this text during console logout?
Because "! - Enable cache by E tag: 4 / 15 FAILED 28.34ms" is not detailed enough.

Eugene Ilyin · 2016-06-12 16:07:28

@itSDS,

Test it on Win 8.1 Pro + Delphi 10.1 Berlin Version 24.0.22858.6822 + Synopse mORMot Framework 1.18.2734 + Boilerplate HTTP Server 1.1.0 = All BDD Scenarios passed.

itSDS · 2016-06-12 18:37:26

Hi Eugene, your plan to put alltogether in one exe is perfekt and for me it's enough if it runs under Delphi (Windows) no need for linux and fpc
I made a little modifikation to assetslz, because in my Folder there is a subfolder named .static which will be excluded if you just check for a dot at the beginning.

procedure FindFiles(const DirName: string; var Files: TFileNameDynArray);
var
  F: TSearchRec;
begin
  if FindFirst(DirName + '*', faAnyFile, F) <> 0 then Exit;
  try
    repeat
      if (F.Name = '.') then Continue;
      if (F.Name = '..') then Continue;
      if (F.Attr and faDirectory) > 0 then
        FindFiles(DirName + F.Name + PathDelim, Files)
      else begin
        SetLength(Files, Length(Files) + 1);
        Files[High(Files)] := DirName + F.Name;
      end;
    until FindNext(F) <> 0;
  finally
    FindClose(F);
  end;
end;

Which option do i have th set if i wan't the HTTPServer to get the Views of my MVC-Application from the assets ?

itSDS · 2016-06-12 18:44:22

Hi Eugene concering the Test:

I use the latest build of mormot downloaded with git ! (for the test above it was '1.18.2733')
You should display the mormot version in the test output so you don't have to ask
My Berlin Versin is as yours : 24.0.22858.6822
And Boilerplate was also a 1.1.0 taken from your GIT-Repository

Eugene Ilyin · 2016-06-12 20:27:52

Release 1.2

Minor Accept-Encoding fix and code refactoring:

fix "Accept-Encoding" parsing when gzip in the end of encodings list
make Pre-Build events notice more visible in tests and demo
minor code refactoring

Eugene Ilyin · 2016-06-12 20:34:56

Release 1.3

Fix bug in assetslz tool when asset name started with '.'

Eugene Ilyin · 2016-06-12 21:11:34

@itSDS,

Thanks! I've include this tool fix into Release 1.3.

itSDS wrote:

Which option do i have th set if i wan't the HTTPServer to get the Views of my MVC-Application from the assets ?

Unfortunately the current Mustache Template Engine implementation is heavily coupled with files layout and not diskIO-agnostic even on abstract levels.
I provided my suggestions of such refactoring through inheritance, but Arnaud decide to go in more decoupled way through delegation.
When he finished I can create boilerplate delegate to feed Mustache Engine with View and Partials from mem-cached boilerplate assets, but not from files.

As for now I do the next thing to get the same result:

Segregate all non-static *.html and *.partial (or any other mustache-related templates) into Views directory (put it near Assets dir).

Embed all these assets into exe with another three additional Pre-Build events

..\Tools\assetslz.exe Views Views.synlz
..\Tools\resedit.exe $(INPUTNAME).res rcdata VIEWS Views.synlz
DEL Views.synlz

In the server startup code I unpacked all Views somewhere on production folder (when you restart your server all unpacked files will not be overwritten if not changed)

var
  Views: TAssets;
...
  Views.Init;
  Views.DecompressFromResource('Views');
  Views.SaveAssets(GetViewsDir);

  FApplication := TMyApplication.Create;
  FApplication.Start(FServer, GetViewsDir);

Then create TMVCApplication with Mustache Views and Partials loaded from this folder:

  TMyApplication = class(TMVCApplication, IMyApplication)
  protected
    function CreateViews(const AFolder: string): TMVCViewsAbtract;
  public
    procedure Start(Server: TSQLRestServer; const Folder: string = ''); reintroduce;
...
procedure TMyApplication.Start(Server: TSQLRestServer; const Folder: string);
begin
  inherited Start(Server, TypeInfo(IMyApplication));
  FMainRunner := TMVCRunOnRestServer.Create(Self, fRestServer, '',
    CreateViews(Folder), [{publishStatic, }bypassAuthentication]);
...
function TMyApplication.CreateViews(const AFolder: string): TMVCViewsAbtract;
var
  Params: TMVCViewsMustacheParameters;
begin
  FillChar(Params, SizeOf(Params), 0);
  with Params do
  begin
    Folder := AFolder;
    FileTimestampMonitorAfterSeconds := 0;
    ExtensionForNotExistingTemplate := '';
    Helpers := TSynMustache.HelpersGetStandardList;
  end;
  Result := TMVCViewsMustache.Create(FFactory.InterfaceTypeInfo, Params,
    (fRestModel as TSQLRestServer).LogClass);
end;

itSDS wrote:

You should display the mormot version in the test output so you don't have to ask wink

@ab

Can you add mORMot version console logout during test suite execution? Maybe right after "Generated with: * compiler" line.

Last edited by Eugene Ilyin (2016-06-12 21:19:44)

Eugene Ilyin · 2016-06-12 21:17:52

@itSDS

Concerning failure of EnableCacheByETag feature BDD-scenarios: as you see I cann't reproduce it on my env. even with Berlin 10.1.
Can you localize the issue or give me additional feedback what scenarios are failed on what SynTestCase.Check's.

itSDS · 2016-06-13 19:25:48

Hi Eugene here are my Debug Outputs:

First Stop - Stack Trace

SynLog.TSynLog.DebuggerNotify(sllFail,'#% %',(...))
SynTests.TSynTestCase.TestFailed('_Out ''ETag'' expected=''"6FECA093"'', actual=''"79CACBF7"''')
BoilerplateTests.TBoilerplateHTTPServerSteps.ThenOutHeaderValueIs('ETag','"6FECA093"')
BoilerplateTests.TBoilerplateHTTPServerShould.EnableCacheByETag
SynTests.TSynTests.Run
SynTests.TSynTests.RunAsConsole('',[sllError,sllLastError,sllException,sllExceptionOS])
mORMotBPTests.mORMotBPTests
:744138f4 KERNEL32.BaseThreadInitThunk + 0x24
:77475de3 ; 
:77475dae ;

Second Stop

SynLog.TSynLog.DebuggerNotify(sllFail,'#% %',(...))
SynTests.TSynTestCase.TestFailed('_Out ''ETag'' expected='''', actual=''"79CACBF7"''')
BoilerplateTests.TBoilerplateHTTPServerSteps.ThenOutHeaderValueIs('ETag','')
BoilerplateTests.TBoilerplateHTTPServerShould.EnableCacheByETag
SynTests.TSynTests.Run
SynTests.TSynTests.RunAsConsole('',[sllError,sllLastError,sllException,sllExceptionOS])
mORMotBPTests.mORMotBPTests
:744138f4 KERNEL32.BaseThreadInitThunk + 0x24
:77475de3 ; 
:77475dae ;

Eugene Ilyin · 2016-06-13 21:59:16

@itSDS

Thanks for these details.
For ETag hash precalculation by the TAsset class is use crc32c function call (as it implemented in TSQLRestServerURIContext.Returns in mORMot.pas, line 38826).

Seems like you crc32c pre-calculation when you run assetslz.exe on Pre-build stage for file Tests\Assets\index.html is differ from expected by bdd-scenario: "6FECA093"

@ab

Is it possible that crc32c call (implemented as crc32cfast or crc32csse42) can return different results for x86/x64 platforms or on different CPU or differ for Debug/Release mode?

In the bdd-scenarios I used constant $6FECA093 to compare it against ETag's pre-calculated for index.html file when synlz-archive created on Pre-Build stage.
But on itSDS' platform the crc32c returns another constant $79CACBF7 for the same Tests\Assets\index.html. I saw that you using the same technique when run some tests and compare results against constant crc/hash. How is it possible that crc32c result is differ form expected constant?

Eugene Ilyin · 2016-06-14 01:20:43

@itSDS

I've just tried all possible variants with Berlin 10.1:

program CRC32CTests;

{$APPTYPE CONSOLE}

uses
  SynCommons;

var
  Data: RawByteString;

begin
  Data := StringFromFile('..\..\index.html');
  WriteLn('crc32cfast  : ' + CardinalToHex(crc32cfast(0, PAnsiChar(Data), Length(Data))));
  WriteLn('crc32csse42 : ' + CardinalToHex(crc32csse42(0, PAnsiChar(Data), Length(Data))));
  ReadLn;
end.

32-bit:

crc32cfast  : 6FECA093
crc32csse42 : 6FECA093

64-bit:

crc32cfast  : 6FECA093
crc32csse42 : 6FECA093

Is it possible that you change content of index.html somehow?

Please, could you run the program above on your env.?
Still cann't reproduce your 79CACBF7 ETag value.

Eugene Ilyin · 2016-06-14 01:34:46

Release 1.4

make TAsset to be packed record for better x86/x64 platforms compatibility

itSDS · 2016-06-15 08:42:28

Hi Eugene, i checked my file with the one from your ZIP and they are different.
The Reason is: I use the GIT CheckOut and there is a CR + LF Setting, adding CR+LF (#13#10) at the end.
Your index.html has only lf (#10) at the end.

Reason for error found.

Please remember, on Windows Platforms the line end code is cr+lf

Eugene Ilyin · 2016-06-15 13:16:02

Release 1.5

fix EnableCacheByETag test scenarios

Eugene Ilyin · 2016-06-15 13:17:37

@itSDS,

Now I reproduced this issue.
Please try and confirm Release 1.5 - all BDD-scenarious should be passed now.

Thanks

itSDS · 2016-06-18 10:28:03

Hi Eugene i tried to test it just a Moment ago. But atm i get Access Violation in SynCommons RecordLoad wth 1.18.2754 - don't know why ...

Eugene Ilyin · 2016-06-18 12:05:55

@itSDS,
Try clear and the rebuild the project.

Usually tests failed on SynCrtSock.pas (3937)

  // THttpServerGeneric thread preparation: launch any OnHttpThreadStart event
  NotifyThreadStart(self);
  // main server process loop
  if Sock.Sock>0 then

Seldom after call NotifyThreadStart the Sock become nil and AV has raised (all exceptions in this code are catch and ignored in the end of this method)
So if you run tests 5-8 times you will get this issue in this place (tested on XE8, Seattle, Berlin)

Just try to clear, then rebuild and let me know, thanks.

@ab,
Do you have any updates following integration into trunk?

itSDS · 2016-06-20 23:03:42

Hi found the error it has gone after i removed your modification for TAsset = packed record now i use TAsset = record and it works !

itSDS · 2016-06-20 23:05:54

Addition - I added 64Bit Platform -> 167/301 FAILED

Eugene Ilyin · 2016-06-21 02:04:15

@itSDS,

TAsset declared packed intentionally.
Other-vice you can't use assets compressed resource build with 32-bit assetslz.exe in 64-bit project and vice versa.

That's why you get 167/301 failed scenarios.
Just take it back and fully rebuild you project.
Everything should be ok now.

Eugene Ilyin · 2016-07-21 00:11:56

Release 1.6.0

add custom options registration (useful for different cache headers for different URLs)
add Vary header into http response for compressible resources - better compatibility with intermediate proxies

Eugene Ilyin · 2016-11-17 16:19:40

Release 1.7.0

add custom options registration for group of URLs
get rid of system PosEx for better compatibility with Delphi 2007 and below

Release 1.8.0

support redirections in /404 response
changed HTML_* to HTTP_* constants following the mORMot refactoring
support new HTTP context initialization spec

keinn · 2017-06-07 00:40:42

Very Nice !!! i finally get it worked .
Thanks very much .
but i find that some bpo* options dos NOT work , like bpoDeleteXPoweredBy. when i add this option, the response header still contain the X-Powered-By:mORMot.
and It seems that it does not cache The SynMustache Template Files, the Exe Still go to the Views dir to find Templates.
and another thing, does it work under Linux now?

Eugene Ilyin · 2017-11-19 04:21:33

keinn, sorry for the long delay with the response.

The "X-Powered-By" HTTP header line is added explicitly by the mORMot core code - this is helps to understand the popularity of mORMot in the public Internet services and give some gratitude to his authors.

If you insist on this line removal you can search for NOXPOWEREDNAME definition in SynCrtSock.pas
Or you can add NOXPOWEREDNAME into Project Options -> Delphi Compiler -> Conditional Defines under "All Configurations" Target

Eugene Ilyin · 2019-08-01 22:40:38

Hi all,

The mORMotBP 2.0 has been released!

High-level changes:

Now it contains 50+ properties and options which are fully aligned with HTML 5 Boilerplate v7.2.0.
Utilize your network smaller and deliver your resources and assets up to 15%-30% faster with new Zopfli and Brotli compressions.
Save your CPU cycles: now all assets are pre-compressed before embedding.
Delegate compressed assets transferring to low-level API and free your threads for more interesting work.
New cache busting strategies are available.

Following your requests:
mORMotBP now support on Delphi 6 up to Delphi 10.3 Rio, Kylix 3 (over CrossKylix), and FPC, targeting Windows or Linux, 32-bit or 64-bit architectures!

Release 2.0

Align all boilerplate assets to recent HTML 5 Boilerplate 7.2.0
All Delphi compilers support started from Delphi 6 (special BuildEvents IDE extenstion provided for old Delphi 6/7/2005/2006)
Free Pascal support (for Lazarus IDE pre-build.sh scipt provided to compress and embed static assets over "Run / Build File" IDE option)
Kylix 3 support (over CrossKilyx)
Zopfli compression support for static assets (save up to 5-15% of traffic and delivery time compared to max GZip Level)
Brotli compression support for static assets as per RFC 7932 (save another 15%-25% of traffic and delivery time compared to Zopfli)
All assets compressions (GZip/Zopfli, and Brotli) now precomputed and embedded, so you save your CPU cycles by skipping any static assets compression on production
Add additional cache bursting strategy. See bpoEnableCacheBustingBeforeExt
Following RFC 7946 the GeoJSON applications now use application/geo+json MIME type
MIME Type for RDF XML documents now application/rdf+xml following as per RFC 3870
Add support of .mjs files with EcmaScript modules (or JavaScript modules) MIME types
Add web assembly (.wasm) MIME type support
Woff fonts (.woff) now have updated font/woff MIME type
Woff version 2 fonts (.woff2) now have updated font/woff2 MIME type
True Type collection .ttc fonts now have separate font/collection MIME type
TTF fonts (.ttf) now have separate font/ttf MIME type
OTF fonts (.otf) now have separate font/otf MIME type
Add support for .ics (text/calendar), and .markdown, .md (text/markdown) MIME types
Upgrade the required 'charset=UTF-8' MIME type list
Upgrade Content Sequrity Policy (CSP)
New bpoEnableReferrerPolicy options
The GZippedMimeTypes has been removed (just pack your assets with updated assetslz tool)
Deprecation of Iframes cookies support in Internet Explorer
TAssets.SaveAssets remove regexp for assets matching (this excludes dependency over SynTable.pas)

Please see https://github.com/eugeneilyin/mORMotBP for details.

Eugene Ilyin · 2019-12-03 09:29:56

Release 2.2

Align assets to recent HTML 5 Boilerplate 7.3.0
New CSP unit to handle Content Security Policy Level 2 / Level 3
Add TBoilerplateHTTPServer.ContentSecurityPolicyReportOnly property
BoilerplateAssets content types normalization

Last edited by Eugene Ilyin (2019-12-03 21:37:38)

macfly · 2019-12-03 22:05:45

Thanks @Eugene for this great tool/project.

I'm going to start a new project and I want to use it.

I have a little doubt. Is it possible to include files outside the .res file?

For example, a css/js/etc ... loaded directly from the directory when requested for first time, and then served from memory in subsequent requests.

Eugene Ilyin · 2019-12-03 23:35:22

macfly

Usually I use mORMotBP in the opposite way

In one of my projects I create React.js/Emotion.js front-end, build a bundle with Webpack and then embed all assets into the single executable file (Linux with FPC or Windows with Delphi). On production I can choose: delegate assets transfer to low-level http.sys core driver (with .StaticRoot property) and save threds time to something else or to transfer assets right from the memory.

You can do the same trick with Mustache Views - debug and pack them into another Views.res file then extract them with one method call into some folder on production. This is good for delivery, version control, fast rollbacks and any production assets/resources embedding or extraction.

One big advantage on this library is that you precompress all your static assets with maximum possible compression each time you compile your project (you embed not only asset as it is (identity version), but maximum Zopfli compression version (best possible gzip) + maximum Brotli compression version + hash calculation done for each compressed version preliminary which can be used in different cache strategies later like ETag). So you save network costs, get faster html/css/js/json/xml, etc. transfer, save CPU usage, save HDD/SSD usage and deliver maximum possible compressed version of your assets to clients right from the server RAM. Such compression is not possible when you try to compress file on production, especially when you have large app/webworker js code.

Also this library show good results when work behind nginx or other http reverse proxy - because you needn't to spend resources on compression on the fly, you have a lot of HTTP headers fixes and get best HTTP Headers delivery practices and protection available in HTTP Bolerplate framework. Started from version 2.2 - all possible Content Security Policy directives Level 2 and Level 3 are supported (see CSP.pas unit) with Hashes, Nonces, etc. to protect your clients even more. The library code is concentrated on speed, direct call stack usage where possible and inlinings.

If you want to read assets from some production folder and feed them to mORMotBP server just look how assetslz tool works, then do the same thing on production and fill your mORMotBP server with any assets you want.

Something like this:

TMyHTTPServer = class(TBoilerplateHTTPServer)
public
  procedure ReadAssetsFromPath(const Path: string);
...
end;

procedure TMyHTTPServer.ReadAssetsFromPath(const Path: string);
var
  Index: Integer;
  Files: TFindFilesDynArray;
begin
  Files := FindFiles(Path, '*', '', True, True, True);
  for Index := Low(Files) to High(Files) do
    FAssets.AddAsset(Path, Files[Index].Name)
end;

Last edited by Eugene Ilyin (2019-12-05 10:16:25)

macfly · 2019-12-04 18:16:31

@Eugene

Thank you for the detailed explanation, contains valuable information.

I agree with all aspects.

My question was that also being able to load files on the fly is an important option.

(in some cases, and without abuse)

Nice to know that this is possible too.

Thanks for the examples.

Last edited by macfly (2019-12-04 18:18:36)

edwinsn · 2019-12-05 07:11:20

Great project! What if one want to use another css/html framework such as Bootstrap?

Eugene Ilyin · 2019-12-05 10:14:19

edwinsn,

This is HTTP Server, the library is not limited to any front-end solution, you can use anything you want: React, Vue, Angular, JQuery, Bootstrap, Vanilla JS.

The goals of this library is to bring best HTTP practices into mORMot Framework: protect against clickjacking, block access to sensitive information, prevent MIME sniffing, handle XSS filter, handle referrer policy, fix well known mangled accept encodings, handle www supression, MIME-types, Vary header, charsets, provides different cache strategies, support all content security policy directives, etc.

Besides handling HTTP you get best possible compression available for your assets, so you can deliver your app/service workers faster especially in mobile networks.

You can check the available options and properties or see CSP directives for details.

edwinsn · 2019-12-05 11:29:09

@Eugene, it's very useful, I shall study the source code, thanks!

macfly · 2019-12-05 14:53:41

@Eugene

What is the relationship between mORMotBP and a reverse proxy?

As some checks/processes will be done by nginx/apache, there may be duplicate processes, correct?

Recommend in this case, leave the reverse proxy settings "more relaxed" to privilege mORMotBP?

#1 2016-06-08 23:32:25