mORMot Open Source

You can e-mail directly to me (the e-mail is on the end of the library description): eugene.ilyin (at) gmail (dot) com

But right now, I'm working on increasing Brotli/Zopfli compression speed and mem reusage.

Maybe we should create a separate mORMotBP2 project on GitHub

P. S.
Or maybe we can put this library to mORMot as AB suggested couple of years ago

Fix confirmed, thanks!

Hi htits2008,
Just get the recent release, follow the instructions and build the Demo project, run HTTP server locally, open the root page and see the headers, GZip/Brotli compression, CSP directives, cache behaviour, static assets etc.
Then play with options and see the results or check how the test cases work.
Then try to build some your favorite front-end: Angular, React, Vue, Svelte, Solid, VanillaJS, play with SSR and experiment with mORMot on-the-fly mustache template engine, etc.
Then connect all of it with your favorite reverse proxy like nginx, openhttplite, CDN, etc.

Fill free to ask if something is not clear.

@mpv, you can easily migrate to k6

It's army swiss knife on the modern complex load testing scenarios:
Asserts, Thresholds, HTTP/1.1, HTTP/2, WebSocket, gRPC, Cookies, Crypto, Custom metrics, Encodings, Environment variables, JSON, HTML forms, files.

Configuation-as-a-script and very pleasant to write scenarios.
Not sticked to Linux only.

Here is a "hello world" HTTP load test under Windows:

 
import http from "k6/http";

export default function() {
  let response = http.get("http://127.0.0.1:8080/");
};

k6.exe run -u 256 -i 1000000 script.js

@ab,
Please consider the another alternative to fix it in the recent Delphi compilers with Managed Record inside TAutoFree.
One of the benefit from-the-box is reverse auto-invocation order of Finalize() in routines epilogue.

Ok, I will print " Id and EntityId, not ID and EntityID ! " on A4 and meditate for an hour per day on it

Unfortunatelly Object Pascal Style Guide and Free Pascal guidelines are not clear about acronyms notation in names.

Maybe Kotlin (successor of all Android dev) balance is the best approach?

@damiand,

There is an easy way to avoid this mORMotHTTPServer exception on FPC:
Detach DBServer/DBServers from your HTTPServer before destruction, like so:

{$IFDEF FPC} HTTPServer.RemoveServer(Server); {$ENDIF}

I've updated the Demo sample, please see this small amendment here.

No exceptions on FPC 3.2.0 / Lazarus 2.0.10 / mORMot 1.18.6186 on Win64.
Please let me know if the issue is gone or still exist on the latest mORMot revision when you add this line to the end of your source code.

@vlad,
If you have a single project (not dynamically generated bunch of projects)
Please use RFG - Real Favicon Generator to generate all necessary favicons.
This service is free and generates all state-of-the-art set of required icons aligned with the most recent guidelines.

Then you can pack all favicons into a single file, pre-compute GZipped and Brotli versions for each asset in maximum available compression and make all client's browsers happy.

To embed necessary HTML tags into generated HTML template you can use mORMot mustache template engine with embedding of isolated favicons.partial file:

{{#favicons}}
  <link rel="image_src" type="image/png" href="/apple-touch-icon-1024x1024.png"/>
  <link rel="apple-touch-icon" href="/apple-touch-icon-1024x1024.png">
  <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
  <link rel="mask-icon" href="/safari-pinned-tab.svg" color="{{{themeColor}}}">
  <meta name="msapplication-TileColor" content="{{{themeColor}}}">
  <meta name="theme-color" content="{{themeColor}}">
{{/favicons}}

As from the best caching practices I recommend to embed hash-versioning into the file names and strip hashed when the asset is requested:

  <link rel="image_src" type="image/png" href="/apple-touch-icon-1024x1024.7a6ed56a.png">
  <link rel="apple-touch-icon" href="/apple-touch-icon-1024x1024.7a6ed56a.png">
  <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.19f7af26.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.29731e46.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.686f4164.png">
  <link rel="mask-icon" href="/safari-pinned-tab.77664866.svg" color="{{{themeColor}}}">
  <meta name="msapplication-TileColor" content="{{{themeColor}}}">
  <meta name="theme-color" content="{{themeColor}}">

If you do so, you can make such assets immutable:

Cache-Control: public, no-transform, max-age=31536000, immutable

Or when CDN/NGINX reverse-proxy used:

Cache-Control: public, no-transform, max-age=31536000, immutable, stale-while-revalidate=2592000, stale-if-error=2592000

To pre-compress, compute ETag hashes, embed as a single resource right into executable, and then distribute all icons assets you can use mORMotBP which is a direct class from TSQLHttpServer.

@ab

We can find all collisions between units interface section identifiers in two different ways (please let me know if you find an easier way to do it):

API Reference collisions identification

Deep crawl Embarcadero site for all identifiers (not sure that every identifier is present in docs):
http://docwiki.embarcadero.com/Librarie … /Unit_List
http://docwiki.embarcadero.com/Librarie … /en/System
http://docwiki.embarcadero.com/Librarie … em.Actions
http://docwiki.embarcadero.com/Libraries/Sydney/en/Vcl
http://docwiki.embarcadero.com/Librarie … /en/Winapi
http://docwiki.embarcadero.com/Libraries/Sydney/en/Xml (maybe)

Deep crawl Free Pascal site for all identifiers by the next paths (again not sure that every identifier is present in docs):
https://www.freepascal.org/docs-html/cu … index.html
https://www.freepascal.org/docs-html/cu … index.html
https://www.freepascal.org/docs-html/cu … index.html

Deep crawl mORMot API referense site (is this reference actual, auto-generated and equal recent fossil version?):
https://synopse.info/files/html/Synopse … _FRAMEWORK

Extract all identifiers in interface section of all listed units and build the collision list.

Source code collisions identification

Parse all *.pas files of Embarcadero / Free Pascal / mORMot sources with one of the next Delphi language AST builders:

• http://www.ssw.uni-linz.ac.at/Research/Projects/Coco/

• https://github.com/jacobthurman/Castalia-Delphi-Parser

• https://github.com/SkliarOleksandr/DelphiAST

• https://github.com/RoutineOp/TeeParser

• https://github.com/Turbo87/PasParse

• https://github.com/Kryuski/GOLD-Parsing … For-Delphi

• https://github.com/DaveBlakeman/DelphiUsesGraph

• https://github.com/wanderlan/llvm-pascal

Extract all identifiers from AST-interface-section-nodes and build the collision list.

P. S.
I agree with other developers that the SynTrim naming is short, elegant, colision free, and do not require conditional defines or other magic to use.
As for fully signature identical methods the replacement with more efficient code can be named equally.

Thanks Pavel, it's great alternative to default MIME types list!

Release 2.3

• Upgrade Assets to HTML Boilerplate v8.0.0

• Align options with Apache Server Configs v4.0.0

• Upgrade Brotli compression to v1.0.9

• Support Dynamic Brotli compression (save about 25% CPU usage compared to GZip on 64-bit systems and 10% less delivery time and traffic utilization).

• Add support of 1490 MIME Types file extensions from IANA, Apache, and Mozilla (see BoilerplateAssets.KnownMIMETypes as an alternative to SynCommons.GetMimeContentType)

• bpoDelegateUnauthorizedTo404 set content for HTTP 401 "Unauthorized" response code equals to /404

• bpoDelegateNotAcceptableTo404 set content for HTTP 406 "Not Acceptable" response code equals to /404

• bpoDelegateHidden block access to all hidden files and directories except for the visible content from within the /.well-known/ hidden directory

• bpoDisableTRACEMethod prevents TRACE requests being made via JavaScript

• TStrictSSL supports strictSSLIncludeSubDomainsPreload

• New DNSPrefetchControl property controls DNS prefetching

• TAssets.SaveToFile now forces file directories before save and returns boolean success value

• assetslz didn't store compressed content if it size is greater than the size of the identity content (which prevents unnecessary bundle increase)

Thanks

@macfly,

No, I think that issues is that ab is busy now with v2 migration.
No rush - just pinging him from time to time

#307 is fully backward capatible and solve 2 issues:
1. Hardcoded Accept: */*
2. Add QueryDataAvailable before ReadData (see separate forum topic ralated to it)

Both changes are minor (add couple of lines) and related only to Windows API.

Let me disagree with you: mORMot HTTP Client Requests are good for feeds fetching and cancelling, cancellation of long requests, support of Brotli compression and many other applications compared to simple call of remote REST endpoint.

I can solve almost anything with inheritance, except hardcoded values or missing of QueryDataAvailable which is required.

#307 just fix this without any regression or changes in app code.

Hi ab,
Do you need some assistance from my side to accept #307?

Hi sakura,

Seems like HTTP 402 is reserved as per RFC 7231:

And in real practice this HTTP code has very unclear semantic:

May be it will be better to revert #303, what do you think?

Thanks ab,

All allocated memory now released property after the 1.18.5944 release.

Thanks macfly,

The issue is in internal buffer management of TTextWriter.

When the data is bigger than provided external stack-allocated buffer - the TTextWriter code is switching to heap memory allocation
in SynCommons:55649

      if twoBufferIsExternal in fCustomOptions then // use heap, not stack
        exclude(fCustomOptions,twoBufferIsExternal) else
        FreeMem(fTempBuf); // with big content comes bigger buffer
      GetMem(fTempBuf,fTempBufSize);

As you see the twoBufferIsExternal flag is excluded from fCustomOptions to indicate that we must release memory later in TTextWriter.Destroy or during the future internal buffer reallocations.

But! Externally the TDocVariant.ToJSON code restores fCustomOptions from some strange backup variable (?)
As the result twoBufferIsExternal flag is brings back to fCustomOptions and all allocated by the TTextWriter heap memory is never released.

SynCommons:48215

      W.Add(']');
    end;
    W.fCustomOptions := backup;
  end else

Weird solution

Restore from this backup variable all flags except twoBufferIsExternal.

    W.fCustomOptions := backup - [twoBufferIsExternal] + W.fCustomOptions * [twoBufferIsExternal];

Better solution

Refactor TDocVariant.ToJSON code and not use the protected section of TTextWriter class by the external TDocVariant class.

As for now any JSON requests, serialization, or data transfer larger than ~40KB (~4Kb gzip) provides memory leaks on server

1. Besides D5 (maybe D6) drop, can we drop Kylix support also (due to FPC domination)? Write library for mORMot framework with Kylix support (because mORMot support it) is not fun.

2. Sermer is perfect (hope that 2.* will exists as much as 1.18 )

3. Please keep synking with git - this is the easiest way to have own git branches with required framework tuning.

4. Some whats new section/blog/file will be nice (maybe as git releases with notes) to skip sources analysis for changes for the last couple of months to see what new functions created/updated.

5. Speed techniques for different aspects of framework: mORMot has 4-8 ways do to the same work, I always have feeling that my code calls/usage are not optimal for my cases. Of couser some functions docs have "good/better/best" hints, but not for all day-by-day calls. Like string manipulations, search, arrays, JSON, TDocVariantData, etc.

Please check #274 with JSONToVariant added.

Tried to make it more mORMot`ish and remove all intermediate procs (now this code contains the fastest available calls). Please confirm.

interface

/// retrieve a variant value from a JSON as per RFC 8259, RFC 7159, RFC 7158
// - follows TTextWriter.AddVariant() format (calls GetVariantFromJSON)
// - will instantiate either an Integer, Int64, currency, double or string value
// (as RawUTF8), guessing the best numeric type according to the textual content,
// and string in all other cases, except TryCustomVariants points to some options
// (e.g. @JSON_OPTIONS[true] for fast instance) and input is a known object or
// array, either encoded as strict-JSON (i.e. {..} or [..]), or with some
// extended (e.g. BSON) syntax
// - warning: by default dvoAllowDoubleValue is set and 32-bit floating-point
// conversion is tried, with potential loss of precision during the conversion
// - warning: the JSON buffer will be modified in-place during process - use
// a temporary copy or the overloaded functions with RawUTF8 parameter
// if you need to access it later
function _JsonStrictInPlace(const JSON: PUTF8Char;
  Options: TDocVariantOptions = [dvoReturnNullForUnknownProperty];
  const AllowDouble: Boolean = True): Variant;
  {$ifdef HASINLINE}inline;{$endif}

/// retrieve a variant value from a JSON as per RFC 8259, RFC 7159, RFC 7158
// - follows TTextWriter.AddVariant() format (calls GetVariantFromJSON)
// - will instantiate either an Integer, Int64, currency, double or string value
// (as RawUTF8), guessing the best numeric type according to the textual content,
// and string in all other cases, except TryCustomVariants points to some options
// (e.g. @JSON_OPTIONS[true] for fast instance) and input is a known object or
// array, either encoded as strict-JSON (i.e. {..} or [..]), or with some
// extended (e.g. BSON) syntax
// - this overloaded procedure will make a temporary copy before JSON parsing
// and return the variant as result
// - warning: by default dvoAllowDoubleValue is set and 32-bit floating-point
// conversion is tried, with potential loss of precision during the conversion
function _JsonStrict(const JSON: RawUTF8;
  Options: TDocVariantOptions = [dvoReturnNullForUnknownProperty];
  const AllowDouble: Boolean = True): Variant;
  {$ifdef HASINLINE}inline;{$endif}

/// retrieve a variant value from a JSON as per RFC 8259, RFC 7159, RFC 7158
// - this global function is an handy alias to:
// ! _JsonStrict(JSON,JSON_OPTIONS_FAST,AllowDouble);
// - follows TTextWriter.AddVariant() format (calls GetVariantFromJSON)
// - will instantiate either an Integer, Int64, currency, double or string value
// (as RawUTF8), guessing the best numeric type according to the textual content,
// and string in all other cases, except TryCustomVariants points to some options
// (e.g. @JSON_OPTIONS[true] for fast instance) and input is a known object or
// array, either encoded as strict-JSON (i.e. {..} or [..]), or with some
// extended (e.g. BSON) syntax
// - this overloaded procedure will make a temporary copy before JSON parsing
// and return the variant as result
// - warning: by default dvoAllowDoubleValue is set and 32-bit floating-point
// conversion is tried, with potential loss of precision during the conversion
function _JsonStrictFast(JSON: RawUTF8;
  const AllowDouble: Boolean = True): Variant;
  {$ifdef HASINLINE}inline;{$endif}

implementation

function _JsonStrictInPlace(const JSON: PUTF8Char; Options: TDocVariantOptions;
  const AllowDouble: Boolean): Variant;
var wasString: boolean;
    Val, Dest: PUTF8Char;
begin
  if JSON = nil then
    ZeroFill(@result) // varEmpty
  else if IdemPChar(JSON, 'NULL') then
    TVarData(result).VType := varNull
  else begin
    Dest := JSON;
    if AllowDouble then
      Dest := TDocVariantData(result).InitJSONInPlace(
        Dest, Options + [dvoAllowDoubleValue])
    else
      Dest := TDocVariantData(result).InitJSONInPlace(Dest, Options);
    if Dest = nil then begin
      Dest := JSON;
      Val := GetJSONField(Dest,Dest,@wasString);
      GetVariantFromJSON(Val,wasString,result,nil,AllowDouble);
    end;
  end;
end;

function _JsonStrict(const JSON: RawUTF8; Options: TDocVariantOptions;
  const AllowDouble: Boolean): Variant;
var tmp: TSynTempBuffer;
begin
  tmp.Init(JSON); // temp copy before in-place decoding
  try
    Result := _JsonStrictInPlace(tmp.buf, Options, AllowDouble);
  finally
    tmp.Done;
  end;
end;

function _JsonStrictFast(JSON: RawUTF8; const AllowDouble: Boolean): Variant;
begin
  Result := _JsonStrict(JSON, JSON_OPTIONS_FAST, AllowDouble);
end;

Btw, from JSON.org:

json
  element

element
  ws value ws

value
  object
  array
  string
  number
  "true"
  "false"
  "null"

And they also support standart exponent notation for numbers, which I am getting frequently from different 3rd party APIs:

@jaclas,

Seems like the UnCompressMem routine in PasZip has an issue and proceed only the first 2 bytes of the compressed data then exit with the success result.

The solution is simple: use SynZip unit based on zlib library instead of PasZip unit. It's recent, better managed, gives you control over compression level and let you choose the compression method (Deflate, GZip or ZLib).

You needn't to change anything in your code (parameters order and types are the same).

program DeflateTest;

{$APPTYPE CONSOLE}

uses
  SynZip,
//  PasZip, // Uncomment to check PasZip CompressMem
  SysUtils;

var
  Index, Size: Integer;
  Data, Compressed, Decompressed: array of AnsiChar;

begin
  SetLength(Data, 1000);
  SetLength(Compressed, 1000);
  SetLength(Decompressed, 1000);

  for Index := Low(Data) to High(Data) do
    Data[Index] := AnsiChar(Ord('A') + Index mod 10);

  Size := CompressMem(Data, Compressed, Length(Data), Length(Compressed));
  Writeln('Compressed size: ', Size);

  Size := UnCompressMem(Compressed, Decompressed, Size, Length(Decompressed));
  Writeln('Decompressed size: ', Size);
  Writeln('Equality: ', CompareMem(Data, Decompressed, Length(Data)));
  Readln;
end.

Don't set the compressed buffer size equals to the source data size.
You must add some extra bytes for the cases where the compressed sequence requires more bytes than the source data (it is possible when you try to compress high-entropy random data or better compressed data like synlz archives, png, pdf, docx, etc.). Additionally you have to reserve some extra bytes for the compression header overhead.

If you change AnsiChar(65 + j) to AnsiChar(Random(255)) you will see that compression failed because the compressed data requires 1005 bytes (in average case) as minimum (set compressed buffer to 2000 for example and check the actual compressed size).

The maximum possible memory required for (Gzip, Zlib or Deflate) compression is provided in SynZip.pas:5368 unit in CompressInternal:

procedure CompressInternal(var Data: ZipString; Compress, ZLib: boolean);
...
    DataLen: integer;
begin
...
  DataLen := length(Data);
...
    SetString(Data,nil,DataLen+256+DataLen shr 3); // max mem required

So 12.5% overhead with additional 256 bytes for the header (or Size + Size shr 3 + 256) is a good heuristic for the max size of any compressed data (it's includes your possible plans to switch from Deflate to GZip):

begin
  SetLength(Data, 1000);
  for Index := Low(Data) to High(Data) do
    Data[Index] := AnsiChar(Ord('A') + Index mod 10);

  Size := Length(Data);
  SetLength(Compressed, Size + Size shr 3 + 256);
  Size := CompressMem(Data, Compressed, Length(Data), Length(Compressed));
  Writeln('Compressed size: ', Size);

  SetLength(Decompressed, Length(Data));
  Size := UnCompressMem(Compressed, Decompressed, Size, Length(Decompressed));
  Writeln('Decompressed size: ', Size);
  Writeln('Equality: ', CompareMem(Data, Decompressed, Length(Data)));
  Readln;
end.