#1 Re: mORMot 2 » Multi thread free memory using fpcx64mm » Today 12:43:11

okoba wrote:

Default memory manager of FPC does not free it until closing the program or on reuse.

Yes, I meet the same issue. My program calls the web API 10,000 times, the memory always increases until I close the program. It reports no memory leaks. But when using fpcx64mm, the amount of memory is a fixed value.

P.S. the OS is windows. In aarach64, the memory manager of FPC is the same as fpcx64mm.

#2 Re: mORMot 2 » RestHttpServer with https » Today 12:31:03

It looks like your certificate file isn't signed by a valid CA or might be damaged.

#3 Re: mORMot 2 » RestHttpServer with https » Yesterday 15:11:59

see source code:

/// create a HTTP/HTTPS Server instance, to serve REST requests
...
// - for a HTTPS server, use secTLS and set CertificateFile, PrivateKeyFile,
// and PrivateKeyPassword expected values, or specify secTLSSelfSigned
constructor Create(const aServers: array of TRestServer; const aPort: RawUtf8;
  aThreadPoolCount: integer = 32; aSecurity: TRestHttpServerSecurity = secNone;
  aOptions: TRestHttpServerOptions = HTTPSERVER_DEFAULT_OPTIONS;
  const CertificateFile: TFileName = ''; const PrivateKeyFile: TFileName = '';
  const PrivateKeyPassword: RawUtf8 = ''; const CACertificatesFile: TFileName = '');
    reintroduce; overload;

from the documentation

#4 Re: mORMot 2 » RestHttpServer with https » 2024-11-17 02:14:11

server:
  HttpServer := TRestHttpServer.Create(HttpPort, [SampleServer], '+', useBidirAsync, 4, secTLSSelfSigned);
client:
  HttpClient := TRestHttpClient.Create('localhost', HttpPort, Model, true);

#5 Re: mORMot 2 » Help for uses IDocList » 2024-11-17 01:41:31

var
  d1, d2: IDocDict;
  l: IDocList;
begin
  l := DocList(j);
  for d1 in l do
    for d2 in d1.L['phones'] do
      writeln(d2['number']);
end;    

#6 Re: mORMot 2 » Arnaud will be away for Holidays » 2024-11-14 02:13:31

Congratulations to mORMot 2 for winning the championship with a time of 1.261s in the 1BRC in Pascal competition! mORMot is the best! Pascal is the best!

https://github.com/ObjectPascal-Communi … me-ov-file

#7 Re: mORMot 2 » TSynDaemon/TSynDaemonSettings memory leak » 2024-11-12 05:02:00

ab wrote:

IIRC the Stop method should be reentrant.

Yes, that's right, the comment of the stop() method says so:

    /// inherited class should override this abstract method with proper process
    // - should do nothing if the daemon was not running (e.g. already stopped)
    procedure Stop; virtual; abstract;

#8 Re: mORMot 2 » TSynDaemon/TSynDaemonSettings memory leak » 2024-11-11 05:42:11

Please try this PR: https://github.com/synopse/mORMot2/pull/319


There are two reasons for this issue:

1. No call to TSampleDaemon.Free()
2. Even if called, an exception will occur because Free() will call Stop(), and this function has already been executed in the previous CommandLine() when the user presses the Enter key.

Solution:

1. In the Stop() method, check the Server status and skip execution if it is not running.
2. Call TSampleDaemon.Free() at the end of the program.

#9 Re: mORMot 2 » Different behaviour in variant to json » 2024-11-10 16:10:55

Here is some samples too:

   v := _Json('{"a":["1"],"b":"2"}');
   v.a.Delete(0);
   v.a.Add('02');
   v.a._ := '03'; // same as v.a.Add('03');

#10 Re: mORMot 2 » ClientRestoreCallbacks in TWebSocketProcessSettings » 2024-11-10 09:35:24

If you try this commit: https://github.com/synopse/mORMot2/comm … c7a417d929, you will see that the server-side OnWSClose() can now be called regardless of whether the client is normally or abnormally disconnected, so that the TMyService.CallbackReleased() event inherited from TInterfacedObject or TInterfacedObjectRest will be called correctly.

Of course, there is no problem with your current approach, which allows for greater flexibility in handling disconnected events.

#11 Re: mORMot 2 » THttpProxyServer Inquiry » 2024-11-10 09:20:42

Kabiri wrote:

Thank you @zen010101
I just didn't understand about the whitelist.

Sorry, I found that the WhiteIP property of THttpAccsptBan seems to only be able to contain one IP address that should not be banned.

    /// a 32-bit IP4 which should never be banned
    // - is set to cLocalhost32, i.e. 127.0.0.1, by default
    property WhiteIP: cardinal
      read fWhiteIP write fWhiteIP; 

#12 mORMot 2 » Feature Request: add Multicast support in TNetSocketWrap » 2024-11-09 03:07:32

zen010101
Replies: 0

TNetSocketWrap encapsulates the basic Socket API, but seems to have missed the API related to UDP Multicast, such as AddMulticast, DropMulticast, MulticastTTL, and so on.

If @ab have time, you can fill in this missing piece. Thank you very much!

#13 Re: mORMot 2 » THttpProxyServer Inquiry » 2024-11-09 02:56:07

Regarding whitelist, you can check out THttpServer.Banned property which is THttpAcceptBan type that has some features about whitelist.

#14 Re: mORMot 2 » THttpProxyServer Inquiry » 2024-11-09 02:50:08

Maybe you call call Stop method first, then Start again.


You can take a look at the source code of the AfterServerStarted() method, and then you can manually control fServer.Route function. The core statement is :

    old := fServer.ReplaceRoute(new); 

#15 Re: mORMot 2 » TAESPRNG manual seed » 2024-11-09 02:45:08

Seed is a procedure instead of a property. See it comments:

/// would force the internal generator to re-seed its private key
    // - avoid potential attacks on backward or forward security
    // - would be called by FillRandom() methods, according to SeedAfterBytes
    // - this method is thread-safe

So, no need to set the SEED manually, just using those properties:

/// after how many generated bytes Seed method would be called
    // - default is 32 MB - i.e. 21-bit CTR rounds which seems paranoid enough
    // - if set to 0 - e.g. for TSystemPrng - no seeding will occur
    property SeedAfterBytes: PtrUInt
      read fSeedAfterBytes;
    /// how many Pbkdf2HmacSha512 count is applied by Seed to the entropy
    // - default is 16 rounds, which is more than enough for entropy gathering,
    // since GetEntropy output comes from a SHAKE-256 generator in XOF mode
    property SeedPbkdf2Round: cardinal
      read fSeedPbkdf2Round;
    /// the source of entropy used during seeding - faster gesUserOnly by default
    property SeedEntropySource: TAesPrngGetEntropySource
      read fSeedEntropySource;

#16 Re: mORMot 2 » SynZip in c# » 2024-11-06 03:29:07

ebekman wrote:

The next 2 bytes is the header of zlib 0xC5 0x39. The zlib documentation says that SUM of this two bytes it must be exactly divisible by 31 (% 31)
At this part "Header checksum illegal" error is raised.

Im stuck in this point

Using https://github.com/jzebedee/LibDeflate.NET instead of zlib to Uncompress the data.

#17 Re: mORMot 2 » SynZip in c# » 2024-11-05 03:32:18

The following c# code is just an example and does not mean that it can be executed:

https://gist.github.com/zen010101/e8236 … c34777b9ee

#18 Re: mORMot 2 » SynZip in c# » 2024-11-05 02:43:03

program demoUncompressString;
uses
  mormot.core.buffers,
  mormot.core.zip;

const
  b64Buffer = 'ZxsAA....==';
var
  binBuffer: RawByteString;
  xmlStr: RawByteString;
begin
  binBuffer := Base64ToBin(b64Buffer);
  xmlStr := UncompressString(binBuffer);
  ...
end.

It very simple task.

#19 Re: mORMot 2 » example httpServerFiles THttpProxyServer.destroy exception » 2024-11-04 14:56:37

ab wrote:

Could you create a Pull Request with this fix?
I am not with my dev computer, because I am on holidays. wink

Done

#21 Re: mORMot 2 » ClientRestoreCallbacks in TWebSocketProcessSettings » 2024-10-29 03:27:46

dcoun wrote:

I think I managed to arrive to a working solution.

Maybe you can share your solution so that everyone can be inspired. wink

#22 Re: mORMot 2 » JSON de-serialization - string quoted numbers » 2024-10-28 17:59:52

ertank wrote:

I wonder if mORMot2 can de-serialize these numbers in double data type as defined in the record.
If it is possible, I do appreciate a small RecordLoadJson() example showing how I can build necessary parameters and pass them to the function.

mORMot can deal with such formatted JSON string. Here is demonstration code for using RecordLoadJson() to process it.

https://gist.github.com/zen010101/75acf … 1c7476285f

#23 mORMot 2 » Why limit the max number of ServerThreadPoolCount to 4? » 2024-10-28 15:32:57

zen010101
Replies: 1

While I was researching the mORMot source, I found the following code:

constructor TWebSocketServer.Create(const aPort: RawUtf8;
  const OnStart, OnStop: TOnNotifyThread; const ProcessName: RawUtf8;
  ServerThreadPoolCount, KeepAliveTimeOut: integer;
  ProcessOptions: THttpServerOptions);
begin
  ....
  if ServerThreadPoolCount > 4 then
    ServerThreadPoolCount := 4; // don't loose threads for nothing
  // start the server
  inherited Create(aPort, OnStart, OnStop, ProcessName, ServerThreadPoolCount, ....
  ....
end;

Why do we need to limit ServerThreadPoolCount? Although I found the following explanation in the comments of Create, to be honest, I still don’t quite understand it:

....
    // - in the current implementation, the ServerThreadPoolCount parameter will
    // use two threads by default to handle shortliving HTTP/1.0 "connection: close"
    // requests, and one thread will be maintained per keep-alive/websockets client
    ....
    constructor Create(const aPort: RawUtf8;
      const OnStart, OnStop: TOnNotifyThread; const ProcessName: RawUtf8;
      ServerThreadPoolCount: integer = 2; KeepAliveTimeOut: integer = 30000;
      ProcessOptions: THttpServerOptions = []); override;

#24 Re: mORMot 2 » i have ian issue with concurrent calls to a function in SOA » 2024-10-28 15:10:56

"but in the console of my test, I get a timeout error if I call the function more than six times

What does this sentence mean?
Is there any difference between the so-called "console of my test" and "///then, it calls the function TestThread"?
In "console of my test", did you also initiate the test using thread concurrency?

My suggestions are:

Server Side:
1. Output the `YourRestHttpServer.HttpServer.HttpQueueLength` property
2. Output the `TWebSocketServerRest(YourRestHttpServer.HttpServer).ThreadPool.WorkThreadCount` property (or using THttpServer(YourRestHttpServer.HttpServer).ThreadPool....)


Client Side:
1. Set global variable `HTTP_DEFAULT_RECEIVETIMEOUT := 120000`
2. Set  `YourRestHttpClientWebsockets.RetryOnceOnTimeout  := true`

#26 Re: mORMot 2 » ClientRestoreCallbacks in TWebSocketProcessSettings » 2024-10-28 08:21:18

Maybe AB has a easier solution with ClientRestoreCallbacks wink

#27 Re: mORMot 2 » ClientRestoreCallbacks in TWebSocketProcessSettings » 2024-10-28 02:51:33

If you have trouble identifying a specific client using RemoteIP, I think you can use ConnectionID to do it.

In the case of Project31ChatClient sample routine, we can use the following code to obtain the Client ConnectionID:

procedure Run;
var Client: TRestHttpClientWebsockets;
    pseudo,msg: string;
    Service: IChatService;
    callback: IChatCallback;
begin
    ...
    Client.WebSocketsUpgrade(PROJECT31_TRANSMISSION_KEY);
    ...
    Client.ServiceDefine([IChatService],sicShared);
    if not Client.Services.Resolve(IChatService,Service) then
      raise EServiceException.Create('Service IChatService unavailable');
    try
      ...
      callback := TChatCallback.Create(Client,IChatCallback);
      Service.Join(pseudo,callback);
      ConsoleWrite('ConnectionID: %', [Client.WebSockets.WebSockets.ConnectionID], ccGreen);   // LOOK AT HERE
      ...
    finally
      callback := nil; // will unsubscribe from the remote publisher
      Service := nil;  // release the service local instance BEFORE Client.Free
    end;
  finally
    Client.Free;
  end;
end;

Now that you have the ConnectionID, pass it to the server when joining() or other Register-alike() service, then on the server side, you can establish the corresponding relationship between the ConnectionID and the client Callback Interface.

#28 Re: mORMot 2 » ClientRestoreCallbacks in TWebSocketProcessSettings » 2024-10-27 17:46:38

In #2 floor, I mentioned one method:

      HttpServer.WebSocketsEnable(Server,PROJECT31_TRANSMISSION_KEY, false,
      [pboSynLzCompress], nil, Server.DoWSClose);


That is, inject the parameter aOnWSClosed: TOnWebSocketProtocolClosed in WebSocketsEnable();

In addition, there are two event properties in PWebSocketProcessSettings returned by WebSocketsEnable() that can also be used for your needs:

    /// callback run when a WebSockets client is just connected
    // - triggerred by TWebSocketProcess.ProcessStart
    OnClientConnected: TNotifyEvent;
    /// callback run when a WebSockets client is just disconnected
    // - triggerred by TWebSocketProcess.ProcessStop
    OnClientDisconnected: TNotifyEvent;
    /// if the WebSockets Client should be upgraded after socket reconnection 

Now that you have these events, they will be automatically called, and when the connection between the client and the server is abnormally interrupted, all you need to do in advance is to establish the correlation between the RemoteIP and the Callback 。

#29 Re: mORMot 2 » ClientRestoreCallbacks in TWebSocketProcessSettings » 2024-10-27 17:31:01

The following method will be called by the server, when a client callback instance is released (either explicitly, or if the connection is broken), so could be used to unsubscribe to the notification, simply by deleting the callback from the internal fConnected[] array:

procedure TChatService.CallbackReleased(const callback: IInvokable; const interfaceName: RawUTF8);
begin
  if interfaceName='IChatCallback' then
    InterfaceArrayDelete(fConnected,callback);
end;

The framework will in fact recognize the following method definition in any interface type for a service (it will check the method name, and the method parameters):

   procedure CallbackReleased(const callback: IInvokable; const interfaceName: RawUTF8);

When a callback interface parameter (in our case, IChatCallback) will be released on the client side, this method will be called with the corresponding interface instance and type name as parameters. You do not have to call explicitly any method on the client side to unsubscribe a service: assigning nil to a callback variable, or freeing the class instance owning it as a field on the subscriber side, will automatically unregister it on the publisher side.

This is the answer given in the documentation. https://synopse.info/files/html/Synopse … l#TITL_173

#32 Re: mORMot 2 » open pdf files in browsers of client without downloading them » 2024-10-27 16:25:49

Content-Type is set to application/pdf, and Content-Disposition is set to inline, which tells the browser that the file should be opened directly in the browser rather than downloading it.

#33 Re: mORMot 2 » i have ian issue with concurrent calls to a function in SOA » 2024-10-27 04:31:27

Theoretically, the execution speed of your service code should not be very slow, and it should not hinder or block the processing thread of the HTTP request. Moreover, even if it does get blocked, for example, you could replace the service code with Sleep(10000), but by default, the thread pool handling requests has 32 work-threads, so it shouldn't block the client's service calls. It's really strange. Could you provide some logs to see what the server-side WebSocket is doing?

#34 Re: mORMot 2 » ClientRestoreCallbacks in TWebSocketProcessSettings » 2024-10-27 04:14:27

You can refer to this example:  https://github.com/synopse/mORMot/blob/ … Server.dpr

However, CallbackReleased can only be automatically triggered when the Client is closed normally (Interfaces are freed normally) and the network is ok. If it is due to abnormal reasons, such as the client program being killed by the OS, then this callback function will not be called.

I asked a similar question here:  https://synopse.info/forum/viewtopic.php?id=7037

After research, my solution is as follows:

First, create a subclass of TRestServer and add the callback function DoWSClose:

  TMyServer = class(TRestServerFullMemory)
  public
    procedure DoWSClose(Protocol: TWebSocketProtocol);
  end;
...
procedure TMyServer.DoWSClose(Protocol: TWebSocketProtocol);
begin
  Writeln('ConnectID: ', Protocol.ConnectionID, ' is closed. IP: ', Protocol.RemoteIP);
end;

In this callback function, you can obtain ConnectionID, RemoteIP and other information through Protocol parameters.

Second, pass in the new DoWSClose callback function in the WebSocketEnable method call:

  Server := TMyServer.CreateWithOwnModel([]);
  try
    Server.Server.CreateMissingTables;
    Server.ServiceDefine(TChatService,[IChatService],sicShared).
      SetOptions([],[optExecLockedPerInterface]). // thread-safe fConnected[]
      ByPassAuthentication := true;
    HttpServer := TRestHttpServer.Create('8888',[Server],'+',useBidirSocket);
    try
      HttpServer.WebSocketsEnable(Server,PROJECT31_TRANSMISSION_KEY, false,
      [pboSynLzCompress], nil, Server.DoWSClose).
        SetFullLog; // full verbose logs for this demo
...

Eventually, you will find that no matter what reason the Client disconnects from the Server, this callback function will be called. Of course, you need to bind the relationship between RemoteIP and Client Callback Interface and any other data you need to centrally manage in Join() or other similar Register() services. This way, when you are informed of the interruption, you gain control over handling the objects in the relevant list. Additionally, you should pay attention to thread-safety issues.

#35 Re: mORMot 2 » AESSHA256 in mORMot 2 » 2024-10-26 03:24:38

Search "{ ****************** Deprecated Weak AES/SHA Process } " in mormot.crypt.core.pas and you can see this:

/// SHA-256 hash calculation with padding if shorter than 255 bytes
// - WARNING: this algorithm is DEPRECATED, and supplied only for backward
// compatibility of existing code (CryptDataForCurrentUser or TProtocolAes)
// - use Pbkdf2HmacSha256() or similar functions for safer password derivation
procedure Sha256Weak(const s: RawByteString; out Digest: TSha256Digest);

 

#36 Re: mORMot 2 » A strange error » 2024-10-26 03:01:53

ab wrote:

Note that with heavily multi threaded process, i.e. a lot of concurrent Join() then InterfaceArrayAdd() would start randomly trigger GPF because its internal f2 storage is not protected by a lock.

Even if optExecLockedPerInterface option is used when creating IChatService ?

    Server.ServiceDefine(TChatService,[IChatService],sicShared).
      SetOptions([],[optExecLockedPerInterface]). // thread-safe fConnected[]
      ByPassAuthentication := true;  

#37 Re: mORMot 2 » A strange error » 2024-10-25 09:32:01

ab wrote:

TSynObjectListLocked is for TObject not interface.
As its name states.

Thank you for your reminder. By searching the source code, I feel that the TSynLogCallbacks object in mormot.core.log.pas is what I am looking for. I can refer to it to implement my own Callback Interface List.

However, I still feel strange, why does a certain statement in the following code "magically" avoid the AV exception ?

procedure TChatService.Join(const pseudo: string;
  const callback: IChatCallback);
begin
  if fConnected = nil then
    fConnected := TSynObjectListLocked.Create(false);  
  fConnected.Add(callback);
  InterfaceArrayAdd(f2,callback); // This statement magically avoids the AV exception generated by the `Intf := fConnected[i]` statement in the BlaBla Method below.
end;

#38 Re: mORMot 2 » A strange error » 2024-10-24 19:14:33

If the program has InterfaceArrayAdd(), then

intf := fConnected[i] 

will not report an error;
If you comment it out, an error will be reported.

Your explanation seems to be the exact opposite of what I stated.

#39 mORMot 2 » A strange error » 2024-10-24 18:38:14

zen010101
Replies: 7

In 31-WebSockets\Project31ChatServer.dpr, I try to use TSynObjectListLocked in TChatService to save the IChatCallback because I like to control the lock myself and will access certain global variables in the service in the future. But during the debugging process, I found that an AV error was generated when calling callback in TchatService.BlaBla. To be precise, it was an exception generated in FPC_INTF_ASSIGN.

The strange thing is that when I retain the original array of IChatCallback and register it with the Client, and also retain the InterfaceArrayAdd(f2,callback);, the program will not generate an AV exception. I really didn't understand the reason.

Here is the gist: https://gist.github.com/zen010101/87a35 … c211996889

#40 Re: mORMot 1 » Sharde nothing architecture » 2024-10-23 06:24:40

This idea has it been realized yet?

#42 mORMot 2 » How to capture the interrupt event of the Websocket client program » 2024-10-20 16:04:51

zen010101
Replies: 0

In the mORMot 1 sample "Project31Chat" server-side, how can the interrupt event of the chat client program be captured, for example, when Ctrl-C is pressed?

I found that "ProcessStart/Stop" will be called, but TChatService.CallbackReleased not.

#43 Re: mORMot 1 » WinHTTP, http.sys and WebSockets » 2024-10-20 08:31:27

mpv wrote:

We finish a implementation of WebSockets using HTTP API 2.0 and commit it to branch "WinWebSocket" (BTW merge operation on fossil is very good).
The sample `31 - WebSockets\Project31WinHTTPEchoServer.dpr` give a basic  usage.

We test out implementation using synthetic tests and successfully create a 50 000 concurrent webSocket connections. One server-side connection required 5kb of memory (and do not create a separate thread)
Also we deploy a solution based on the THttpApiWebSocketServer to one of our production. The average load is 3000 concurrent WS connection with up to 5000 sometimes. It work 1 week w/o problems (but we use WS only to notify a client about something happens, all other communication is done using HTTP).

https://blog.synopse.info/?post/2014/08 … they-scale : Here, AB mentioned:

As a conclusion, I would recommend the following, for any project:

- Use WebSocket for client notifications only (with a fallback mechanism to long-polling - there are plenty of libraries around);
- Use RESTful / JSON for all other data, using a CDN or proxies for cache, to make it scale.


At the same time, in this post, mpv also mentioned:

but we use WS only to notify a client about something happens, all other communication is done using HTTP

I am quite curious about the above perspective, how is it implemented? Could you provide some example code?

#44 Re: mORMot 2 » Arnaud will be away for Holidays » 2024-10-16 19:03:04

Enjoy your time, recharge those batteries, and we'll see you soon with stories to share!

#45 Re: mORMot 2 » Accessing this URL using TWinHttp will report an error » 2024-10-16 16:30:04

We have several entries in mormot.net.client.pas to initiate an HTTP GET Request, but their default RedirectMax is a bit confusing:

1. IHttpClient/TSimpleHttpClient: 4 times, can be changed before calling Request
2. TWinINet.Get(): automatic, unknown max times value
3. TWinHttp.Get(): 0 times
4. OpenHttpGet() function: 5 times
5. HttpGet() function: 0 times if use winhttp or 5 times as same as OpenHttpGet
6. THttpClientSocketWGet.WGet/WGet() function: 0 times default, can be changed when called

It might be more friendly if we could unify the default values of RedirectMax for different objects or functions written in different periods.  smile

#46 Re: mORMot 2 » Accessing this URL using TWinHttp will report an error » 2024-10-16 08:35:58

All works but TWinHttp.Get('https://z.cn', '', True{IgnoreTlsCertificateError}, nil, nil, 3000)) not, it seems not reading the RedirectMax property value.

Neither HttpGet('https://z.cn', ''); .

#47 Re: mORMot 2 » lang-cmp example result vs. go version » 2024-10-16 07:19:13

You summed it up so right. So, mORMot is awesome!

#49 Re: mORMot 2 » lang-cmp example result vs. go version » 2024-10-16 03:42:46

ab wrote:

Which compiler did you use?
Win64 as target, I hope?

Yes, Win64 as target, using FPC trunk compiler.

Here is the result running in the same host but in Windows WSL ( Ubuntu ):
https://gist.githubusercontent.com/syno … tfile1.txt

#50 mORMot 2 » lang-cmp example result vs. go version » 2024-10-15 18:19:21

zen010101
Replies: 5

I compiled the mormot version and the go version, and the following are the latest test results. I'm surprised that the Go language version has improved so quickly.

https://gist.githubusercontent.com/syno … tfile1.txt

CPU: Intel(R) Core(TM) Ultra 9 185H
RAM: 32G
OS: Windows 11 23H2

Board footer

Powered by FluxBB