mORMot Open Source

Threadvar's are created in a special memory area (thread-local storage). At compile time the addresses of that space for a thread created at runtime isn't fix.
Therefore libraries linked at compile time (which packages are) could have an issue with threadvar's. Dynamically loaded libraries are fine, because they will loaded at runtime, when the threads run.

Because of this it doesn't matter whether a threadvar is declared in interface or implementation part.

I've implemented this task some time ago to check if I could handle it. It's not so easy, but perhaps the situation is more easy if I would have a real AD.

To verify the JWT you need the public key.
The public key in pem format you get from the x5c value. In my test case (at microsoft) the key changes every time (I can't remember the interval e.g. every 30 days).
To get it you have to call -> https://login.microsoftonline.com/'+ response.tenantId +'/discovery/v2.0/keys?appid=...
and compare the kid included in the JWT which you want to verify.

If you have the public key you can verify the JWT with this func:

function VerifyJWT(const aJWT: RawByteString; const aPublicKey: RawByteString; out JWT: Variant): Boolean;
  function VerifyWithmORMot(const aJWT: RawByteString; out JWT: Variant): Boolean;
  var
    Header, Payload, Signature: RawByteString;
    sigBinary, msgBinary: TBytes;
    idx, step: Integer;
    aBuf: PAnsiChar;
  begin
    Header:= '';
    Payload:= '';
    Signature:= '';
    idx:= 0;
    step:= 0;
    aBuf:= PAnsiChar(aJWT);
    while aBuf[idx] <> '' do begin
       INC(idx);
       if aBuf[idx] = '.' then begin
         INC(step);
         if step=1 then
           SetString(Header, aBuf, idx)
         else
         if step=2 then
           SetString(Payload, aBuf, idx)
         else
           SetString(Signature, aBuf, idx);
         INC(aBuf, idx+1);
         idx:= 0;

       end;
    end;

    if (idx > 0) and (step < 3) then
      SetString(Signature, aBuf, idx);

    JWT:= _Obj([]);
    sigBinary:= Base64UrlDecode(Signature);
    RawByteStringToBytes(Header+'.'+Payload, msgBinary);

    Result:= OpenSslVerify('', '', Pointer(msgBinary), Pointer(aPublicKey), Pointer(sigBinary), Length(msgBinary), Length(aPublicKey), Length(sigBinary));
    if Result then
      JWT:= Js2Var(Base64uriToBin(PAnsiChar(Payload), Length(Payload)));
  end;

begin
  Result:= VerifyWithmORMot(aJWT, JWT);
end;

I wrote again because I think it could be useful to stabilize the interfaces/network (mormot.core.interfaces / mormot.core.soa) part of the framework, so it is useful for all users:

My pov is, executing of interfaced methods isn't thread safe in some circumstances. Two parallel (sicSingle) service calls comes and their params will be mixed/override. In my case there are such corrupt bison's in mongodb queries.

@Ab could you please check TInterfaceMethodExecuteCached if there is anything what could go wrong under heaver load? All my wrk tests looks good but the live system has this error behavior and all other processes looks good, also the syslog looks good so far.

I've commented out this part of TInterfaceMethodExecuteCached .Aquire

if fCached.TryLock then
  begin
    // reuse this shared instance between calls
    SetOptions(opt);
    exec := self;
    fCachedWR.CancelAllAsNew;
    WR := fCachedWR;
  end

but its not enough or not the right place.
Are there any other changes in past that comes with https://blog.synopse.info/?post/2022/01 … e-Them-All the that I could test easily?

Perhaps it is not an issue of wrong locking but of not initialization of something reused because of caching?

Hi ab,

just fyi the error in my service from today and the syslog entries bring me to the nested option for lxc containers and I think I'll give it a try:

https://forum.proxmox.com/threads/lxc-a … -13.36173/

Do you know this thread and is it interesting to you?

https://gitlab.com/freepascal.org/fpc/s … sues/40677

Here are the relevant log file entries for the EThreadError:

20241016 13290923  -    08.340.038
20241016 13290923 EXC           EThreadError {Message:"Thread error"} [R16:root] at 428014
20241016 13290923 debug         uRestServerDB.TSOneRestServerDB(020bafc8) TServiceFactoryServer.InstanceFree: ignored EThreadError exception during IPersons._Release
20241016 13290923 srvr           192.168.1.214 Interface POST root/Persons.GetOneByID=200 out=3 KB in 4.32s
20241016 13290923 ret           mormot.rest.server.TRestServerRoutingRest(7fdb340cd238) {"result":[{"res": true, ...

The call of the sicSingle service method Persons.GetOneByID is ok from the pov of the client.

This error also comes in case of useHttpSocket (not only with useHttpAsync) and under current mORMot2 branch #38874e16c.

Yes I hope too.

This error is not really reproducible. I had it the first time since July, but we had several other errors over the last months (e.g. service not responsible or undefined exceptions or the Variant error with with _Safe(v)^.InitArrayFromCsvFile and memory corruption after that) and we have changed a lot of possible bugs on our side but without serious knowledge.

I can change logging to verbose mode and send the file to you (but it's very huge and I will give you an url for downloading per email) but also verbose logging wasn't enough in past. Perhaps adding logs in mormot.net.async or mormot.net.sock is the way to go.

If error is async related could I change to useHttpSocket for testing (in front of our service we are using nginx as reverse proxy)?

->Do you have more info?
Only this errors I had in logfile since last weeks:
20241015 15564437 EXC EThreadError {Message:"Thread error"} [R12:root]
20241015 15564446 EXC EThreadError {Message:"Thread error"} [R14:root]
20241015 15564448 EXC EThreadError {Message:"Thread error"} [R13:root]
20241015 16171856 EXC EThreadError {Message:"Thread error"} [R13:root]
20241015 16171859 EXC EThreadError {Message:"Thread error"} [R8:root]
But it seems to be transparent from the client side, because we have no error responses related to that errors.

Atm I use two services as upstream under nginx (linux as main service and a windows http.sys service as backup). The windows service seems to be more stable as the linux with async server.

->Where/when is it killed?
It's killed from os today. Here are the rows from syslog:
Oct 15 16:09:53 mssql SOneSrv2[358268]: malloc(): unaligned tcache chunk detected
Oct 15 16:09:53 mssql systemd[1]: SOneSrv2.service: Main process exited, code=killed, status=6/ABRT
Oct 15 16:09:53 mssql systemd[1]: SOneSrv2.service: Failed with result 'signal'.
Oct 15 16:09:53 mssql systemd[1]: SOneSrv2.service: Consumed 28min 29.400s CPU time.
Oct 15 16:10:23 mssql systemd[1]: SOneSrv2.service: Scheduled restart job, restart counter is at 1.

->What is the load?
It's difficult to answer.
Per day we have about 400-500 thousand requests with a average request time of 20ms, but there are also request with text search in MongoDB that could run 10s.
We are using an lxc container with 32 cores, 100G mem (60G free) and zfs. Our service has 32 worker threads and there are about 100-200 concurrent users.
The cpu-usage is mostly under 10% but for some seconds over 100% because MongoDB is heavily used in front of postgres. But I would say we don't have any overload here.

I wouldn't change so much. My confusing results from working with your tools and not understanding much of it in depth. Some points you understand after years.

But perhaps some small improvements could help.
That _Safe could return a read only fake DocVariantData is perhaps a little bit too much and contrary to the name „_safe“.

A clear after each Init is necessary because Variants are managed types, right?
Otherwise the compiler would free a Variant which may be still in using because of a duplicate Init

Ok, but what’s wrong with a re-initialize of a variant? I could re-initialize it who often I want?
It’s only one variant on the stack and not a class-instance of a create constructor.
I create a Variant with _ARR and will fill this Variant with the data of a csv.
Is the using of _Safe(of a Variant of type 271) in this case so clearly senseless?

@Márcio Baroni
Thanks a lot

In TRttiCustomList.DoRegister Info.KIND has rkBOOL.
But Instance (Result) will not set after GlobalClass.Create and given to AddToPairs(result, Info);

In AddToPairs I can't see from where Instance is getting his KIND (it's just rkUnknown) until inc(Counts[Instance.Kind]);

I've tested and changed code in a very unprofessional way and maybe I'm lost, but I want to share my guess founded in mormon.core.rtti:

The address of the CS will be changed through line 8730 in TRttiCustomList.AddToPairs:

inc(Counts[Instance.Kind]);

After that the RegisterSafe.UnLock; in line 9002 goes with the wrong Pointer address in mutex_unlock_common and therefore it results in a segfault.

If I comment line 8730 the segfault is away and also my real app is running under FreeBSD 13.2 as under Win/Linux.

But I don't know, what's the meaning of line 9002 and the drawbacks of that.

Please look at the stack trace from lazarus. Is #1 und #2 double or normal?
I'm not able to debug constructive (perhaps because of inlining).

#0 mutex_unlock_common(0x9013d1708, false, 0x0) at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1 CTHREADS_$$_CLEAVECRITICALSECTION$formal at :0
#2 SYSTEM_$$_LEAVECRITICALSECTION$POINTER at :0
#3 DOREGISTER(0x801cef6a0, 0xe3d8d8) at ../../mORMot2/src/core/mormot.core.rtti.pas:8630
#4 SETGLOBALCLASS(0x801cef6a0, 0xf8af30) at ../../mORMot2/src/core/mormot.core.rtti.pas:8751
#5 INITIALIZEUNIT at ../../mORMot2/src/core/mormot.core.json.pas:11687
#6 MORMOT.CORE.JSON_$$_init$ at ../../mORMot2/src/core/mormot.core.json.pas:11699
#7 fpc_initializeunits at :0
#8 main at SOneSrv2.dpr:551

I've applied your commit, but
how can I remove the reference to clib in mormot.core.os.posix.inc? There are other functions (e.g. clock_gettime) needing clib.
Do I need cthreads anymore? We have disabled MemoryManager for testing.

@DonAlfredo
Thanks for this link. I've alfredy (joke) read about this issue. But the patch is fro 2019 and I've the R13.2 from Apr 2023 and should the backtrace not shown something of libc in such a case? But I will compare the patch with the current source files and perhaps that will help.

Thanks, with TLightLock app hangs again.

With TOSLightLock segfault again:

#0  mutex_unlock_common (m=0x90134b708, cv=false, mtx_defer=0x0)
    at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1  0x00000000004dcae3 in DOREGISTER (this=0x801de8370, INFO=0xeb7b68)
    at ../../mORMot2/src/core/mormot.core.rtti.pas:8630
#2  0x00000000004dd357 in SETGLOBALCLASS (this=0x801de8370, RTTICLASS=0x1004fa0)
    at ../../mORMot2/src/core/mormot.core.rtti.pas:8751
#3  0x000000000094f8bd in INITIALIZEUNIT ()
    at ../../mORMot2/src/core/mormot.core.json.pas:11687
#4  0x000000000094f979 in MORMOT.CORE.JSON_$$_init$ ()
    at ../../mORMot2/src/core/mormot.core.json.pas:11699
#5  0x0000000000421652 in fpc_initializeunits ()
#6  0x0000000000403b9f in main () at SOneSrv2.dpr:551

Registers:
rax            0x7fffffff          2147483647
rbx            0x90134b708         38674937608
rcx            0x801349008         34379960328
rdx            0x0                 0
rsi            0x0                 0
rdi            0x90134b708         38674937608
rbp            0x7fffffffd540      0x7fffffffd540
rsp            0x7fffffffd4e0      0x7fffffffd4e0
r8             0x6c6f6f42          1819242306
r9             0x80000             524288
r10            0x4a7b48            4881224
r11            0x0                 0
r12            0x7fffffffd9c0      140737488345536
r13            0x0                 0
r14            0x189be             100798
r15            0x801612000         34382880768
rip            0x800f06a33         0x800f06a33 <mutex_unlock_common+51>
eflags         0x10206             [ PF IF RF ]
cs             0x43                67
ss             0x3b                59
ds             0x3b                59
es             0x3b                59
fs             0x13                19
gs             0x1b                27
fs_base        0x800ef1120         34375405856

Please give me some more time to find out next issues, because the main issue resists and just a first step of initialization of the units seems to work…

Yes, I think that's the same issue like the docker one. But it's just my intension, because the av is released at the same time in the same unit initializations.

I've tested your new commit (thank's a lot for your investigation), but got the same segfault as without it:

mutex_unlock_common (m=0x901398708, cv=false, mtx_defer=0x0)
    at /usr/src/lib/libthr/thread/thr_mutex.c:976
976        if (__predict_false(PMUTEX_OWNER_ID(m) != id))
(gdb) bt 10
#0  mutex_unlock_common (m=0x901398708, cv=false, mtx_defer=0x0)
    at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1  0x0000000000431c19 in CTHREADS_$$_CLEAVECRITICALSECTION$formal ()
#2  0x000000000042560a in SYSTEM_$$_LEAVECRITICALSECTION$POINTER ()
#3  0x00000000004d9ea3 in DOREGISTER (this=0x801637188, INFO=0xeb4b58)
    at ../../mORMot2/src/core/mormot.core.rtti.pas:8629
#4  0x00000000004da717 in SETGLOBALCLASS (this=0x801637188, RTTICLASS=0x1001a30)
    at ../../mORMot2/src/core/mormot.core.rtti.pas:8750
#5  0x000000000094ca5d in INITIALIZEUNIT () at ../../mORMot2/src/core/mormot.core.json.pas:11687
#6  0x000000000094cb19 in MORMOT.CORE.JSON_$$_init$ ()
    at ../../mORMot2/src/core/mormot.core.json.pas:11699
#7  0x0000000000421792 in fpc_initializeunits ()
#8  0x0000000000403cdf in main () at SOneSrv2.dpr:550

I've testet with standard memory manager from fpc and with cmem.
No reallocations of large buffers of my own.
Now I've tested the two defines, but with no success.
(freebsd 13.2, fpc 3.2.2)

Following late binding calls doesn't work:

v._(0)

where v is an _ARR

This post seems related to the issue I have:
https://synopse.info/forum/viewtopic.php?pid=34034

And I know I can work around with using TDocVariant instead of
last binding.

But not using late binding is a regression.