mORMot Open Source

disable inlining in FastNewString was it! Thanks a lot.

(mormot2tests doesn't compile, I don't know what's wrong. No error, just Error in ./build_fpc.sh on line 117)

Thanks for explain, make sense.

So when using plain THttpServer, what's your proposal to set ServerKeepAliveTimeOut:= 0 (disabled keep alive)?

I'm creating just a TRestHttpServer which could creates different kinds of HttpServers (Socket, Api, Async) and ServerKeepAliveTimeOut property isn't public over  TRestHttpServer.HttpServer.

--> THttpServer(aTRestHttpServerInstance).ServerKeepAliveTimeOut:= 0; <-- looks like a hack

@dcoun
As I remember I had the same issue with TSynLog for years. It's probably not related to mORMo2 but Windows:

https://stackoverflow.com/questions/687 … owsservice

Sure is a great word and what does it mean to you, when I am?
Better I show you what the log file says:

20250211 15375400  !  +            uHTTPServer.TSOneHttpServer(020f4f28).Create useHttpAsync (secNone) on port 8281
20250211 15375400  !  +                mormot.net.async.THttpAsyncConnections(03205f68).Create(THttpAsyncServerConnection,root,32)
20250211 15375401  " info  SetThreadName 7f69b8b6c640 140092047279680=R0:root
20250211 15375401  " trace uRestServerDB.TSOneRestServerDB(01dfe208) BeginCurrentThread(TAsyncConnectionsThread) root=root ThreadID=7f69b8b6c640 'R0:root' ThreadCount=1

Hello Arnaud,

I could find the location of our culprit today.
But I'm not able to understand it by now and have to test more.

Thanks
Daniel

@ab
Today we had "same" behavior like @itSDS (I'm looking for errors on our side first and since weeks), but it's not a client issue on our side.
1. service isn't responsive any more (/timestamp over curl gives timeout too).
2. absolut no cpu-usage in any thread
3. no exceptions or error in (verbose) log file before.
4. absolut no timeout in postgres.

Our service runs since yesterday evening and we had about 40' request per hour till the issue at 2:20 pm.
We are using mORMot commit #fe67be7b8 / heap.inc / linux (fpc 3.2.0).

I don't want to waste your time. It's a fpc issue I don't understand.

locale:
LANG=C.UTF-8
LC_CTYPE="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_TIME="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_ALL=

fpc:
compiled with -FcUTF-8
mormot.defines.inc inlcuded

file SOneSrv2.dpr:
Unicode text, UTF-8 (with BOM) text (also tested without BOM)

in .dpr just a writeln('ü') -> output: '?'

Thanks you,
Daniel

all my units have a options.inc include with your
{$I mormot.defines.inc}

Just a question from today when working under linux (system codepage utf8), unit.pas is set to utf8 using fpc 3.2.2:

When I make a function Example(const s: RawJSON/RawUtf8/UTF8String) call with:

Example('äöü') -> it's not working any more. I have to call
Example(StringToUtf8('äöü')) to get the right string.

Is this a new behavior from this update. I can't remember to have such issues in past.

I don't have a stack trace on my production system, just a log file (in verbose logging).

The mongdb call is a simple aggregate with a small result, called over a sicSingle service method. Not complex, just a very simple method.

I believe the root of the issue comes from our side, not from mORMot2. But I don't find any entries of interest in my logs.
The mongodb call methods are so simple, that there couldn't be any issue come from. I have 32 workers and every worker has its own mongodb connection in his own threadvar. No race condition, no blocking issue.
MongoDB calls are the main work in our service. If any worker connection would have an issue I would have a crash or errors instantly.

Atm I don't use your new rewrite version (just ef8764e2c).

To find out if async server is involved, should I switch to useHttpSocket and watch?

cmem

@ab Yes mem issue is outside of pipeline.
Today we had such an issue in our program (of course without pipeline):

SOneSrv2: malloc.c:4302: _int_malloc: Assertion `(unsigned long) (size) >= (unsigned long) (nb)' failed

We had no error in Postgres log at all. But we are using cmem.

Threadvar's are created in a special memory area (thread-local storage). At compile time the addresses of that space for a thread created at runtime isn't fix.
Therefore libraries linked at compile time (which packages are) could have an issue with threadvar's. Dynamically loaded libraries are fine, because they will loaded at runtime, when the threads run.

Because of this it doesn't matter whether a threadvar is declared in interface or implementation part.

I've implemented this task some time ago to check if I could handle it. It's not so easy, but perhaps the situation is more easy if I would have a real AD.

To verify the JWT you need the public key.
The public key in pem format you get from the x5c value. In my test case (at microsoft) the key changes every time (I can't remember the interval e.g. every 30 days).
To get it you have to call -> https://login.microsoftonline.com/'+ response.tenantId +'/discovery/v2.0/keys?appid=...
and compare the kid included in the JWT which you want to verify.

If you have the public key you can verify the JWT with this func:

function VerifyJWT(const aJWT: RawByteString; const aPublicKey: RawByteString; out JWT: Variant): Boolean;
  function VerifyWithmORMot(const aJWT: RawByteString; out JWT: Variant): Boolean;
  var
    Header, Payload, Signature: RawByteString;
    sigBinary, msgBinary: TBytes;
    idx, step: Integer;
    aBuf: PAnsiChar;
  begin
    Header:= '';
    Payload:= '';
    Signature:= '';
    idx:= 0;
    step:= 0;
    aBuf:= PAnsiChar(aJWT);
    while aBuf[idx] <> '' do begin
       INC(idx);
       if aBuf[idx] = '.' then begin
         INC(step);
         if step=1 then
           SetString(Header, aBuf, idx)
         else
         if step=2 then
           SetString(Payload, aBuf, idx)
         else
           SetString(Signature, aBuf, idx);
         INC(aBuf, idx+1);
         idx:= 0;

       end;
    end;

    if (idx > 0) and (step < 3) then
      SetString(Signature, aBuf, idx);

    JWT:= _Obj([]);
    sigBinary:= Base64UrlDecode(Signature);
    RawByteStringToBytes(Header+'.'+Payload, msgBinary);

    Result:= OpenSslVerify('', '', Pointer(msgBinary), Pointer(aPublicKey), Pointer(sigBinary), Length(msgBinary), Length(aPublicKey), Length(sigBinary));
    if Result then
      JWT:= Js2Var(Base64uriToBin(PAnsiChar(Payload), Length(Payload)));
  end;

begin
  Result:= VerifyWithmORMot(aJWT, JWT);
end;

I wrote again because I think it could be useful to stabilize the interfaces/network (mormot.core.interfaces / mormot.core.soa) part of the framework, so it is useful for all users:

My pov is, executing of interfaced methods isn't thread safe in some circumstances. Two parallel (sicSingle) service calls comes and their params will be mixed/override. In my case there are such corrupt bison's in mongodb queries.

@Ab could you please check TInterfaceMethodExecuteCached if there is anything what could go wrong under heaver load? All my wrk tests looks good but the live system has this error behavior and all other processes looks good, also the syslog looks good so far.

I've commented out this part of TInterfaceMethodExecuteCached .Aquire

if fCached.TryLock then
  begin
    // reuse this shared instance between calls
    SetOptions(opt);
    exec := self;
    fCachedWR.CancelAllAsNew;
    WR := fCachedWR;
  end

but its not enough or not the right place.
Are there any other changes in past that comes with https://blog.synopse.info/?post/2022/01 … e-Them-All the that I could test easily?

Perhaps it is not an issue of wrong locking but of not initialization of something reused because of caching?

Hi ab,

just fyi the error in my service from today and the syslog entries bring me to the nested option for lxc containers and I think I'll give it a try:

https://forum.proxmox.com/threads/lxc-a … -13.36173/

Do you know this thread and is it interesting to you?

https://gitlab.com/freepascal.org/fpc/s … sues/40677

Here are the relevant log file entries for the EThreadError:

20241016 13290923  -    08.340.038
20241016 13290923 EXC           EThreadError {Message:"Thread error"} [R16:root] at 428014
20241016 13290923 debug         uRestServerDB.TSOneRestServerDB(020bafc8) TServiceFactoryServer.InstanceFree: ignored EThreadError exception during IPersons._Release
20241016 13290923 srvr           192.168.1.214 Interface POST root/Persons.GetOneByID=200 out=3 KB in 4.32s
20241016 13290923 ret           mormot.rest.server.TRestServerRoutingRest(7fdb340cd238) {"result":[{"res": true, ...

The call of the sicSingle service method Persons.GetOneByID is ok from the pov of the client.

This error also comes in case of useHttpSocket (not only with useHttpAsync) and under current mORMot2 branch #38874e16c.

Yes I hope too.

This error is not really reproducible. I had it the first time since July, but we had several other errors over the last months (e.g. service not responsible or undefined exceptions or the Variant error with with _Safe(v)^.InitArrayFromCsvFile and memory corruption after that) and we have changed a lot of possible bugs on our side but without serious knowledge.

I can change logging to verbose mode and send the file to you (but it's very huge and I will give you an url for downloading per email) but also verbose logging wasn't enough in past. Perhaps adding logs in mormot.net.async or mormot.net.sock is the way to go.

If error is async related could I change to useHttpSocket for testing (in front of our service we are using nginx as reverse proxy)?

->Do you have more info?
Only this errors I had in logfile since last weeks:
20241015 15564437 EXC EThreadError {Message:"Thread error"} [R12:root]
20241015 15564446 EXC EThreadError {Message:"Thread error"} [R14:root]
20241015 15564448 EXC EThreadError {Message:"Thread error"} [R13:root]
20241015 16171856 EXC EThreadError {Message:"Thread error"} [R13:root]
20241015 16171859 EXC EThreadError {Message:"Thread error"} [R8:root]
But it seems to be transparent from the client side, because we have no error responses related to that errors.

Atm I use two services as upstream under nginx (linux as main service and a windows http.sys service as backup). The windows service seems to be more stable as the linux with async server.

->Where/when is it killed?
It's killed from os today. Here are the rows from syslog:
Oct 15 16:09:53 mssql SOneSrv2[358268]: malloc(): unaligned tcache chunk detected
Oct 15 16:09:53 mssql systemd[1]: SOneSrv2.service: Main process exited, code=killed, status=6/ABRT
Oct 15 16:09:53 mssql systemd[1]: SOneSrv2.service: Failed with result 'signal'.
Oct 15 16:09:53 mssql systemd[1]: SOneSrv2.service: Consumed 28min 29.400s CPU time.
Oct 15 16:10:23 mssql systemd[1]: SOneSrv2.service: Scheduled restart job, restart counter is at 1.

->What is the load?
It's difficult to answer.
Per day we have about 400-500 thousand requests with a average request time of 20ms, but there are also request with text search in MongoDB that could run 10s.
We are using an lxc container with 32 cores, 100G mem (60G free) and zfs. Our service has 32 worker threads and there are about 100-200 concurrent users.
The cpu-usage is mostly under 10% but for some seconds over 100% because MongoDB is heavily used in front of postgres. But I would say we don't have any overload here.

I wouldn't change so much. My confusing results from working with your tools and not understanding much of it in depth. Some points you understand after years.

But perhaps some small improvements could help.
That _Safe could return a read only fake DocVariantData is perhaps a little bit too much and contrary to the name „_safe“.

A clear after each Init is necessary because Variants are managed types, right?
Otherwise the compiler would free a Variant which may be still in using because of a duplicate Init

Ok, but what’s wrong with a re-initialize of a variant? I could re-initialize it who often I want?
It’s only one variant on the stack and not a class-instance of a create constructor.
I create a Variant with _ARR and will fill this Variant with the data of a csv.
Is the using of _Safe(of a Variant of type 271) in this case so clearly senseless?

@Márcio Baroni
Thanks a lot