mORMot Open Source

A clear after each Init is necessary because Variants are managed types, right?
Otherwise the compiler would free a Variant which may be still in using because of a duplicate Init

Ok, but what’s wrong with a re-initialize of a variant? I could re-initialize it who often I want?
It’s only one variant on the stack and not a class-instance of a create constructor.
I create a Variant with _ARR and will fill this Variant with the data of a csv.
Is the using of _Safe(of a Variant of type 271) in this case so clearly senseless?

@Márcio Baroni
Thanks a lot

In TRttiCustomList.DoRegister Info.KIND has rkBOOL.
But Instance (Result) will not set after GlobalClass.Create and given to AddToPairs(result, Info);

In AddToPairs I can't see from where Instance is getting his KIND (it's just rkUnknown) until inc(Counts[Instance.Kind]);

I've tested and changed code in a very unprofessional way and maybe I'm lost, but I want to share my guess founded in mormon.core.rtti:

The address of the CS will be changed through line 8730 in TRttiCustomList.AddToPairs:

inc(Counts[Instance.Kind]);

After that the RegisterSafe.UnLock; in line 9002 goes with the wrong Pointer address in mutex_unlock_common and therefore it results in a segfault.

If I comment line 8730 the segfault is away and also my real app is running under FreeBSD 13.2 as under Win/Linux.

But I don't know, what's the meaning of line 9002 and the drawbacks of that.

Please look at the stack trace from lazarus. Is #1 und #2 double or normal?
I'm not able to debug constructive (perhaps because of inlining).

#0 mutex_unlock_common(0x9013d1708, false, 0x0) at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1 CTHREADS_$$_CLEAVECRITICALSECTION$formal at :0
#2 SYSTEM_$$_LEAVECRITICALSECTION$POINTER at :0
#3 DOREGISTER(0x801cef6a0, 0xe3d8d8) at ../../mORMot2/src/core/mormot.core.rtti.pas:8630
#4 SETGLOBALCLASS(0x801cef6a0, 0xf8af30) at ../../mORMot2/src/core/mormot.core.rtti.pas:8751
#5 INITIALIZEUNIT at ../../mORMot2/src/core/mormot.core.json.pas:11687
#6 MORMOT.CORE.JSON_$$_init$ at ../../mORMot2/src/core/mormot.core.json.pas:11699
#7 fpc_initializeunits at :0
#8 main at SOneSrv2.dpr:551

I've applied your commit, but
how can I remove the reference to clib in mormot.core.os.posix.inc? There are other functions (e.g. clock_gettime) needing clib.
Do I need cthreads anymore? We have disabled MemoryManager for testing.

@DonAlfredo
Thanks for this link. I've alfredy (joke) read about this issue. But the patch is fro 2019 and I've the R13.2 from Apr 2023 and should the backtrace not shown something of libc in such a case? But I will compare the patch with the current source files and perhaps that will help.

Thanks, with TLightLock app hangs again.

With TOSLightLock segfault again:

#0  mutex_unlock_common (m=0x90134b708, cv=false, mtx_defer=0x0)
    at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1  0x00000000004dcae3 in DOREGISTER (this=0x801de8370, INFO=0xeb7b68)
    at ../../mORMot2/src/core/mormot.core.rtti.pas:8630
#2  0x00000000004dd357 in SETGLOBALCLASS (this=0x801de8370, RTTICLASS=0x1004fa0)
    at ../../mORMot2/src/core/mormot.core.rtti.pas:8751
#3  0x000000000094f8bd in INITIALIZEUNIT ()
    at ../../mORMot2/src/core/mormot.core.json.pas:11687
#4  0x000000000094f979 in MORMOT.CORE.JSON_$$_init$ ()
    at ../../mORMot2/src/core/mormot.core.json.pas:11699
#5  0x0000000000421652 in fpc_initializeunits ()
#6  0x0000000000403b9f in main () at SOneSrv2.dpr:551

Registers:
rax            0x7fffffff          2147483647
rbx            0x90134b708         38674937608
rcx            0x801349008         34379960328
rdx            0x0                 0
rsi            0x0                 0
rdi            0x90134b708         38674937608
rbp            0x7fffffffd540      0x7fffffffd540
rsp            0x7fffffffd4e0      0x7fffffffd4e0
r8             0x6c6f6f42          1819242306
r9             0x80000             524288
r10            0x4a7b48            4881224
r11            0x0                 0
r12            0x7fffffffd9c0      140737488345536
r13            0x0                 0
r14            0x189be             100798
r15            0x801612000         34382880768
rip            0x800f06a33         0x800f06a33 <mutex_unlock_common+51>
eflags         0x10206             [ PF IF RF ]
cs             0x43                67
ss             0x3b                59
ds             0x3b                59
es             0x3b                59
fs             0x13                19
gs             0x1b                27
fs_base        0x800ef1120         34375405856

Please give me some more time to find out next issues, because the main issue resists and just a first step of initialization of the units seems to work…

Yes, I think that's the same issue like the docker one. But it's just my intension, because the av is released at the same time in the same unit initializations.

I've tested your new commit (thank's a lot for your investigation), but got the same segfault as without it:

mutex_unlock_common (m=0x901398708, cv=false, mtx_defer=0x0)
    at /usr/src/lib/libthr/thread/thr_mutex.c:976
976        if (__predict_false(PMUTEX_OWNER_ID(m) != id))
(gdb) bt 10
#0  mutex_unlock_common (m=0x901398708, cv=false, mtx_defer=0x0)
    at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1  0x0000000000431c19 in CTHREADS_$$_CLEAVECRITICALSECTION$formal ()
#2  0x000000000042560a in SYSTEM_$$_LEAVECRITICALSECTION$POINTER ()
#3  0x00000000004d9ea3 in DOREGISTER (this=0x801637188, INFO=0xeb4b58)
    at ../../mORMot2/src/core/mormot.core.rtti.pas:8629
#4  0x00000000004da717 in SETGLOBALCLASS (this=0x801637188, RTTICLASS=0x1001a30)
    at ../../mORMot2/src/core/mormot.core.rtti.pas:8750
#5  0x000000000094ca5d in INITIALIZEUNIT () at ../../mORMot2/src/core/mormot.core.json.pas:11687
#6  0x000000000094cb19 in MORMOT.CORE.JSON_$$_init$ ()
    at ../../mORMot2/src/core/mormot.core.json.pas:11699
#7  0x0000000000421792 in fpc_initializeunits ()
#8  0x0000000000403cdf in main () at SOneSrv2.dpr:550

I've testet with standard memory manager from fpc and with cmem.
No reallocations of large buffers of my own.
Now I've tested the two defines, but with no success.
(freebsd 13.2, fpc 3.2.2)

Following late binding calls doesn't work:

v._(0)

where v is an _ARR

This post seems related to the issue I have:
https://synopse.info/forum/viewtopic.php?pid=34034

And I know I can work around with using TDocVariant instead of
last binding.

But not using late binding is a regression.

@sakura You should think about an unwanted exception in your sm and the manual inserted raise as an example for that.
Than it's interesting because I often see such external exceptions in my log files and misinterpreted these.

Nice Blog about new wire protocol! You could be also a salesman or a story-teller. Unusual for a asm optimizer

Perfect!

Test with aggregate has same issue as with RunCommand.

From expected DocumentCount of 51294 aggregate returns only 16905.

I think I could find out where the issue is. In SendAndGetCursor first 34398 Documents of reply are replaced by next 16905. But I miss the first 101 before 34398, that comes with the version of dc66841.

Here are my wishes:

1. I want to use RunCommand as under commit dc66841. Because I want to have full control about what is sending and what's the replies are.
2. I need your help because I don't understand all the details of SendAndGetCursor(msg, reply) especially working with BSONVariant and other details (e.g. why has BatchSize a value of 65536 -> I haven't set Batchsize at all)

After removing $db params in my Commands I get following error for aggregates:

'{"ok":0,"errmsg":"Requested getMore on namespace ''Rottler.$cmd'', but cursor belongs to a different namespace Rottler.Persons","code":13,"codeName":"Unauthorized"}'

aggregate was:

{aggregate:"Persons",pipeline:[{$match:{PersonTypeID:0,Parents.ParentID:{$in:[903087]},live:{$ne:false}}},{$sort:{LastProcessingDate:-1}},{$project:{_id:0,ID:"$_id",Number:1,Title:1,SecondName:1,FirstName:1,DayOfBirth:1,Sex:1,PersonTypeID:1,Remarks:1,LastProcessingDate:1,CostCenter:{$arrayElemAt:[{$map:{input:{$filter:{input:"$Parents",as:"p",cond:{$eq:["$$p.PersonTypeID",1]}}},as:"cc",in:"$$cc.Number"}},0]},LoginExpiryDate:1,Alive:1,LoginName:1,Parents.ParentID:1,ContactData:1,DSV:1,FirstSalesOrder:1,LastSalesOrder:1}},{$limit:200}],cursor:{},allowDiskUse:true,maxTimeMS:30000}

Problem seems to be in:

SendAndGetRepliesAndFree->call of getMore ($cmd seems to be to needed under new wire protocol (IIRC). It was a virtual collection for OP_QUERY).

When I remove +'.$cmd' in RunCommand Collections is not stetted:
'{getMore:5068573612759429335,collection:"",batchSize:65536,$db:"Rottler"}' -> because Request.CollectionName = '' in line 2786.

Before rewriting every call with RunCommand gets one replay. The getMore request was done by myself, when requested in the reply.
The replay looks like this:
('{"cursor":{"firstBatch":[{"PersonTypeID":4,"Number":"HLD","Alive":true,...

Now replay is:
'[{"PersonTypeID":4,"Number":"HLD","Alive":true,...

If I use your implementation with getMore I tried to set DatabaseName in RunCommand like this:

BsonVariantType.GetItem(command, 'aggregate', v);
  if not VarIsEmpty(v) then
  GetDocumentsAndFree(NewCommand(
    aDatabaseName + '.' + v, command, flags), returnedvalue)

But the replies read with getMore are still not correct this way.

Just a hint: your commit #0ba4230 don't work with JSON-Objects e.g. listCollections with params, because $db have to be the param after listCollections and not the last one.

wrong:
'{listCollections:"Rottler",filter:{name:"revenuesPayments"},$db:"Rottler"}'

right:
'{listCollections:"Rottler",$db:"Rottler",filter:{name:"revenuesPayments"}}'

You would have ti insert the $db param in the command string that comes from the user. That was not necessary in my opinion. I would add $db param only in case of VarIsStr. In all other cases,
the user must deliver the $db param in his command to work with new protocol. So I had to update all my encapsulate methods and added the $db param without the need to change mORMot2 routines.

The AddItem added the $db param also multiple times: '{listCollections:"Rottler",filter:{name:"revenuesPayments"},$db:"Rottler", $db:"Rottler"}'

Daniel

Hello Arnaud,

I've sent you an email with a new version of mormot.db.nosql.mongodb.pas.
But the PR #114 is the better one.

Daniel

Some errors produced by TMongoConnection.GetReply are not real errors but comes from old handling of replies (e.g. Request.MongoRequestID has wrong value).

Even my statistic request works!