mORMot Open Source

I finally know the reason.  My FPC + Linux combination doesn't work is because:

The .mab file I generated relies on the .dbg file, and the SmartLink option is automatically disabled during its creation. However, after generating the .mab file and subsequently switching to Release Mode (which explicitly enables SmartLink), the previously created .mab file becomes invalid. This occurs even when using the -Xm compilation flag to generate the .map file – the map2mab tool appears incompatible with Free Pascal Compiler (FPC)-generated .map files, throwing an "Error: EInOutError Read past end of file" during processing.

The root cause has been identified - SmartLink optimization invalidates the .mab file. This leads to another  question:

Could we make TDebugFile to extract line number and symbol information directly from FPC-generated .map files (created via the -Xm compilation flag)? or make map2mab work?

I knew why. Because I put two formatted log contents into one. The first one is without a thread ID, but the other one had.

I committed a PR to fix some errors regarding the code page based on the results tested yesterday.

I did some experiments, and the conclusion is as follows:
None of these three approaches seems to fully cover the test cases for the CJK encoding set of Python 3.12.

1. uconv (ICU lib) on debian

Or you can use another set of aliases:

The interesting thing about the HZ-GB2312 encoding is:

Although uconv claims to support this encoding, compared to hz.txt, there are two additional characters at the beginning of hz.out: "~}"

2. iconv on debian

3. Windows API

If we use the official list provided by Microsoft, the results are as follows:

https://github.com/L-Angel/compress-demo

This is a comparison of several common compression algorithms, including Snappy and zstd. The conclusion is:

zstd is similar to snappy. And according to the test of ab, synlz is better than snappy. https://blog.synopse.info/?post/2017/08 … form-SynLZ

OK, It works now. :-)

GetLastError returns 87: INVALID_PARAMETER

I compared the implementation code of FPC and mORMot with the WideCharToMultiByte API document, and found that:

The case statement in FullCodePage uses the same values as the API documentation, but the documentation states that the dwFlags parameter needs to be set to 0 when the codepage is equal to the specified value. However, mORMot and FullCodePage return value actually set lpDefaultChar to nil, which is optional according to the API documentation, and FPC has always set this parameter to nil.

In the end, I forced the lpDefaultChar parameter to nil, and the error disappeared.

It's the FPC "feature" :

if the source code saved as UTF8 , e.g. using {$codepage utf8} or saving in utf8 with BOM, the FPC will storage the string constants that contain non-ASCII characters as UTF-16 in the executable.

So, in your FreeBSD, writeln('ü') will output a UnicodeString actually but the console output codepage is UTF-8, so you got a '?' as the result.

But if you don't use UTF8 as source file encoding format, the FPC using CP_ACP ， e.g. in linux it is UTF-8, to storage string constants.

Which API do you use? If it is WideCharToMultiByte, I checked Microsoft's instructions, there is the following description:

If the above instructions are adjusted, if you still fail to succeed, I guess it may be the problem of not installing a language package.

Following the guidelines below to install the simplified Chinese language package:

https://support.microsoft.com/en-us/win … fc697cfca8

FPC doesn't seem to call WinAPI, and on my Windows, I get the correct result with SetCodePage. I traced the FPC code a bit, and it seems that the final conversion code is this:

procedure Unicode2AnsiMove(source:punicodechar;var dest:RawByteString;cp : TSystemCodePage;len:SizeInt);

It is located in the "...\fpcsrc\rtl\objpas\fpwidestring.pp" unit file.

You can try the TSqlDBConnectionProperties.GetFields() method ab mentioned. This method outputs the field definition to the Fields parameter, which is a TSqlDBColumnDefineDynArray type. Each element contains the following information:

  TSqlDBColumnDefine = packed record
    /// the Column name
    ColumnName: RawUtf8;
    /// the Column type, as retrieved from the database provider
    // - returned as plain text by GetFields method, to be used e.g. by
    // TSqlDBConnectionProperties.GetFieldDefinitions method
    // - SqlCreate will check for this value to override the default type
    ColumnTypeNative: RawUtf8;
    /// the Column default width (in chars or bytes) of ftUtf8 or ftBlob
    // - can be set to value <0 for CLOB or BLOB column type, i.e. for
    // a value without any maximal length
    ColumnLength: PtrInt;
   ...
  end;

Windows 11 64bit and Linux aarch64, server side;
Linux aarach64, client side.

I don't know if there is something wrong with me here. You can reproduce the situation on my end like this:

1. Use the official example httpServerRaw, only modify DoRequest to add some delay

function TSimpleHttpAsyncServer.DoOnRequest(Ctxt: THttpServerRequestAbstract): cardinal;
...
// POST = echo
Ctxt.OutContent := Ctxt.InContent;
Ctxt.OutContentType := TEXT_CONTENT_TYPE;
Writeln(Ctxt.ConnectionID, Ctxt.InContent);
Sleep(5000);
...

2. In TSimpleHttpAsyncServer.Create, use a thread pool of only 1 thread:

fHttpServer := THttpAsyncServer.Create(
'55555', nil, nil, '', 1, 30000,
[hsoNoXPoweredHeader,
hsoNoStats,
hsoHeadersInterning
hsoThreadSmooting
//, hsoLogVerbose
]);

3. On the client side, simply create 100 threads to send requests using TSimpleHttpClient:

procedure TtestHttpRequestThread.Execute;
...
FHttpClient := TSimpleHttpClient.Create(True);
Response := FHttpClient.Request(Uri, 'POST', Headers, Format(Params, [FID]), mimetype, 0);
Response := FHttpClient.Request(Uri, 'POST', Headers, Format(Params, [FID]), mimetype, 0);
FHttpClient.Free;

4. When the sending request is complete, press Ctrl-C to exit the client program.
On the server side, you can see that the server has finally processed all 100 requests.

I'm in a similar situation.

Using the latest version of FPC Trunk 3.3.1-17174-g72a3729ca0 [2024/12/31] for x86_64 to compile mormot2tests, TTestCoreBase.FastStringCompare fails to run. Specifically, the shr code of strspn is incorrectly optimized. Only adding {$OPTIMIZATION OFF} works, but this makes FastStringCompare test twice as fast.

How can I make a TRestHttpServer to listen both of IPv4/6 ？

TRestHttpServer.Create(HttpPort, [SampleServer], '+', HTTP_DEFAULT_MODE, 4);

This will using http.sys and listen both 0.0.0.0 and [::].

TRestHttpServer.Create(HttpPort, [SampleServer], '+', WEBSOCKETS_DEFAULT_MODE, 4);

This will only listen 0.0.0.0

TRestHttpServer.Create('[::]:'+HttpPort, [SampleServer], '+', WEBSOCKETS_DEFAULT_MODE, 4);

This will only listen [::].

I tried on Windows 11, here is the result:

1. FPC Default MM

16 GB, 1 MB: 16384
Allocate: 16 GB in 21.29s i.e. 769.5 MB/s
FreeSingle: 16 GB in 613.26ms i.e. 26 GB/s
Allocate: 16 GB in 21.38s i.e. 766.1 MB/s
FreeMulti: 16 GB in 35.68ms i.e. 448.4 GB/s

16 GB, 4 MB: 4096
Allocate: 16 GB in 4.87s i.e. 3.2 GB/s
FreeSingle: 16 GB in 583.41ms i.e. 27.4 GB/s
Allocate: 16 GB in 4.45s i.e. 3.5 GB/s
FreeMulti: 16 GB in 33.54ms i.e. 477 GB/s

16 GB, 64 MB: 256
Allocate: 16 GB in 4.43s i.e. 3.6 GB/s
FreeSingle: 16 GB in 535.99ms i.e. 29.8 GB/s
Allocate: 16 GB in 3.89s i.e. 4.1 GB/s
FreeMulti: 16 GB in 42.48ms i.e. 376.6 GB/s

16 GB, 128 MB: 128
Allocate: 16 GB in 4.30s i.e. 3.7 GB/s
FreeSingle: 16 GB in 533.62ms i.e. 29.9 GB/s
Allocate: 16 GB in 3.90s i.e. 4 GB/s
FreeMulti: 16 GB in 68.81ms i.e. 232.4 GB/s

16 GB, 256 MB: 64
Allocate: 16 GB in 4.35s i.e. 3.6 GB/s
FreeSingle: 16 GB in 523.76ms i.e. 30.5 GB/s
Allocate: 16 GB in 3.86s i.e. 4.1 GB/s
FreeMulti: 16 GB in 134.51ms i.e. 118.9 GB/s

16 GB, 1 GB: 16
Allocate: 16 GB in 4.21s i.e. 3.8 GB/s
FreeSingle: 16 GB in 495.69ms i.e. 32.2 GB/s
Allocate: 16 GB in 3.83s i.e. 4.1 GB/s
FreeMulti: 16 GB in 36.50ms i.e. 438.2 GB/s

2. fpcx64mm

16 GB, 1 MB: 16384
Allocate: 16 GB in 3.99s i.e. 4 GB/s
FreeSingle: 16 GB in 564.80ms i.e. 28.3 GB/s
Allocate: 16 GB in 4s i.e. 3.9 GB/s
FreeMulti: 16 GB in 1.12s i.e. 14.2 GB/s

16 GB, 4 MB: 4096
Allocate: 16 GB in 3.96s i.e. 4 GB/s
FreeSingle: 16 GB in 610.13ms i.e. 26.2 GB/s
Allocate: 16 GB in 3.92s i.e. 4 GB/s
FreeMulti: 16 GB in 985.57ms i.e. 16.2 GB/s

16 GB, 64 MB: 256
Allocate: 16 GB in 3.86s i.e. 4.1 GB/s
FreeSingle: 16 GB in 518.28ms i.e. 30.8 GB/s
Allocate: 16 GB in 3.80s i.e. 4.2 GB/s
FreeMulti: 16 GB in 808.84ms i.e. 19.7 GB/s

16 GB, 128 MB: 128
Allocate: 16 GB in 3.85s i.e. 4.1 GB/s
FreeSingle: 16 GB in 488.24ms i.e. 32.7 GB/s
Allocate: 16 GB in 3.83s i.e. 4.1 GB/s
FreeMulti: 16 GB in 776.20ms i.e. 20.6 GB/s

16 GB, 256 MB: 64
Allocate: 16 GB in 3.89s i.e. 4.1 GB/s
FreeSingle: 16 GB in 541.66ms i.e. 29.5 GB/s
Allocate: 16 GB in 3.85s i.e. 4.1 GB/s
FreeMulti: 16 GB in 726.30ms i.e. 22 GB/s

16 GB, 1 GB: 16
Allocate: 16 GB in 3.83s i.e. 4.1 GB/s
FreeSingle: 16 GB in 518.28ms i.e. 30.8 GB/s
Allocate: 16 GB in 3.88s i.e. 4.1 GB/s
FreeMulti: 16 GB in 786.35ms i.e. 20.3 GB/s

The performance of FPC Default MM is very good, except for 2 issues: first, the speed of allocation in units of 1MB is indeed slow, but the speed of allocation in units of 4MB and above is comparable to fpcx64mm; second, memory is only released when the program is closed.
fpcx64mm Same as the problem described by @okoba, the speed of FreeMulti is not fast but slow.

see source code:

/// create a HTTP/HTTPS Server instance, to serve REST requests
...
// - for a HTTPS server, use secTLS and set CertificateFile, PrivateKeyFile,
// and PrivateKeyPassword expected values, or specify secTLSSelfSigned
constructor Create(const aServers: array of TRestServer; const aPort: RawUtf8;
  aThreadPoolCount: integer = 32; aSecurity: TRestHttpServerSecurity = secNone;
  aOptions: TRestHttpServerOptions = HTTPSERVER_DEFAULT_OPTIONS;
  const CertificateFile: TFileName = ''; const PrivateKeyFile: TFileName = '';
  const PrivateKeyPassword: RawUtf8 = ''; const CACertificatesFile: TFileName = '');
    reintroduce; overload;

from the documentation

Yes, that's right, the comment of the stop() method says so：

    /// inherited class should override this abstract method with proper process
    // - should do nothing if the daemon was not running (e.g. already stopped)
    procedure Stop; virtual; abstract;

Here is some samples too:

   v := _Json('{"a":["1"],"b":"2"}');
   v.a.Delete(0);
   v.a.Add('02');
   v.a._ := '03'; // same as v.a.Add('03');

Sorry, I found that the WhiteIP property of THttpAccsptBan seems to only be able to contain one IP address that should not be banned.

    /// a 32-bit IP4 which should never be banned
    // - is set to cLocalhost32, i.e. 127.0.0.1, by default
    property WhiteIP: cardinal
      read fWhiteIP write fWhiteIP;