MongoDB 6.0

danielkuettner · 2022-07-23 19:13:50

Hi Arnaud,

with MongoDB 6.0 I get following error:

"Unsupported OP_QUERY command" -> https://www.mongodb.com/docs/v6.0/relea … es-removed

It has to be OP_MSG instead.

Same with OP_UPDATE and OP_INSERT and OP_REPLY.

Regards,
Daniel

Last edited by danielkuettner (2022-07-23 20:45:12)

ab · 2022-07-24 06:54:53

Yes, the driver needs some rewrite.
The MongoDB protocol, as most of the MongoDB stack in fact (not the WirefTiger engine itself, which is well written), was always half-backed to my understanding - like something written by good but junior programmers. So they did upgrade the protocol, to ease its expressiveness (focus on JSON/BSON not binary frames). But the concern is that don't care much about protocol consistency and backward compatibility. Why disable something which worked well at wire level? Doesn't make sense. They could keep it forever. They even keep it for some low-level system collections queries. Sounds like if this technology is driven by managers, not techs, anymore.

AFAICT no native Delphi driver have been updated yet.
The Embarcadero driver may work, because they are not native, but use the official mongo C client library, so if you update the dll it "should work".

danielkuettner · 2022-07-24 07:59:05

Hi Arnaud,

my production service uses the 5.0.9 atm and I'm glad to get this issue in a test env. The little mORMot has grown to a bigger one and I wonder how you deal with it.
If I can do something, please let me know.

Daniel

danielkuettner · 2022-07-30 15:53:56

Hi Arnaud,

after making some PRs further changes are necessary to work with new WIRE protocol.
But creating PRs with GitHub is too painful.
If you want, I can send you my version of mormot.db.nosql.mongodb.pas.

Atm my service starts with auth + ssl, list collections and check indexes and insert an array of documents. And all other stuff looks good from my side, because we only used RunCommand and inserting new $db field is the only change we have to do at our side.

Daniel

Last edited by danielkuettner (2022-07-30 16:40:42)

ab · 2022-07-30 16:53:35

Yes send me an email if you prefer.

Or put it in a gist.

Thanks !

danielkuettner · 2022-07-30 17:55:52

Some errors produced by TMongoConnection.GetReply are not real errors but comes from old handling of replies (e.g. Request.MongoRequestID has wrong value).

Even my statistic request works!

Last edited by danielkuettner (2022-07-31 07:11:43)

danielkuettner · 2022-07-31 07:16:25

Some changes are necessary e.g. replacement of CursorID (not available in > 3.6).
I will send you an other version of mormot.db.nosql.mongodb.pas as soon as I've fixed last errors on my side.

Last edited by danielkuettner (2022-07-31 13:13:12)

danielkuettner · 2022-08-02 13:17:02

Hello Arnaud,

I've sent you an email with a new version of mormot.db.nosql.mongodb.pas.
But the PR #114 is the better one.

Daniel

Last edited by danielkuettner (2022-08-02 17:43:35)

danielkuettner · 2022-08-04 08:24:36

Hello Arnaud,

thanks for implementing new wire protocol. I'm testing your commit and it looks promising.

Daniel

ab · 2022-08-04 18:23:17

I worked the whole day on it.

There was a lot to rewrite, in fact...

The writes are now working as expected.
I am finishing the reads.
And will commit something stable tomorrow.

danielkuettner · 2022-08-05 08:27:24

Just a hint: your commit #0ba4230 don't work with JSON-Objects e.g. listCollections with params, because $db have to be the param after listCollections and not the last one.

wrong:
'{listCollections:"Rottler",filter:{name:"revenuesPayments"},$db:"Rottler"}'

right:
'{listCollections:"Rottler",$db:"Rottler",filter:{name:"revenuesPayments"}}'

You would have ti insert the $db param in the command string that comes from the user. That was not necessary in my opinion. I would add $db param only in case of VarIsStr. In all other cases,
the user must deliver the $db param in his command to work with new protocol. So I had to update all my encapsulate methods and added the $db param without the need to change mORMot2 routines.

The AddItem added the $db param also multiple times: '{listCollections:"Rottler",filter:{name:"revenuesPayments"},$db:"Rottler", $db:"Rottler"}'

Daniel

Last edited by danielkuettner (2022-08-05 09:00:51)

ab · 2022-08-05 18:22:00

The order of $db has no relevance.
I always add it at last position with no problem.
The only requirement is that the command is in the first position.

The new behavior is the same as with MongoShell or any other driver - and also of the documentation: you don't put $db in the command itself, it is added by the framework.
It should work seamless with any kind of commands, without any $db in your methods. Just delete the $db references in your code.
Please report any cumbersome problem.

I have just committed a full rewrite, and support of the OP_MSG new requirements.
On my side, it passes all my tests, including ORM/ODM tests.

danielkuettner · 2022-08-05 21:12:47

You are right. I had to remove all my $db params.
Sorry for that.

danielkuettner · 2022-08-06 09:19:30

After removing $db params in my Commands I get following error for aggregates:

'{"ok":0,"errmsg":"Requested getMore on namespace ''Rottler.$cmd'', but cursor belongs to a different namespace Rottler.Persons","code":13,"codeName":"Unauthorized"}'

aggregate was:

{aggregate:"Persons",pipeline:[{$match:{PersonTypeID:0,Parents.ParentID:{$in:[903087]},live:{$ne:false}}},{$sort:{LastProcessingDate:-1}},{$project:{_id:0,ID:"$_id",Number:1,Title:1,SecondName:1,FirstName:1,DayOfBirth:1,Sex:1,PersonTypeID:1,Remarks:1,LastProcessingDate:1,CostCenter:{$arrayElemAt:[{$map:{input:{$filter:{input:"$Parents",as:"p",cond:{$eq:["$$p.PersonTypeID",1]}}},as:"cc",in:"$$cc.Number"}},0]},LoginExpiryDate:1,Alive:1,LoginName:1,Parents.ParentID:1,ContactData:1,DSV:1,FirstSalesOrder:1,LastSalesOrder:1}},{$limit:200}],cursor:{},allowDiskUse:true,maxTimeMS:30000}

Problem seems to be in:

SendAndGetRepliesAndFree->call of getMore ($cmd seems to be to needed under new wire protocol (IIRC). It was a virtual collection for OP_QUERY).

When I remove +'.$cmd' in RunCommand Collections is not stetted:
'{getMore:5068573612759429335,collection:"",batchSize:65536,$db:"Rottler"}' -> because Request.CollectionName = '' in line 2786.

Before rewriting every call with RunCommand gets one replay. The getMore request was done by myself, when requested in the reply.
The replay looks like this:
('{"cursor":{"firstBatch":[{"PersonTypeID":4,"Number":"HLD","Alive":true,...

Now replay is:
'[{"PersonTypeID":4,"Number":"HLD","Alive":true,...

If I use your implementation with getMore I tried to set DatabaseName in RunCommand like this:

BsonVariantType.GetItem(command, 'aggregate', v);
if not VarIsEmpty(v) then
GetDocumentsAndFree(NewCommand(
aDatabaseName + '.' + v, command, flags), returnedvalue)

But the replies read with getMore are still not correct this way.

Last edited by danielkuettner (2022-08-06 16:46:11)

ab · 2022-08-06 20:46:47

Try to use the mORMot aggregate function, not a manual command.

danielkuettner · 2022-08-07 07:30:49

Sure I'll try it.

But do you want to say, RunCommand isn't supported from mORMot2 any more (after rewriting)? Although all was still working in commit dc66841?

danielkuettner · 2022-08-07 10:21:31

Test with aggregate has same issue as with RunCommand.

From expected DocumentCount of 51294 aggregate returns only 16905.

I think I could find out where the issue is. In SendAndGetCursor first 34398 Documents of reply are replaced by next 16905. But I miss the first 101 before 34398, that comes with the version of dc66841.

Here are my wishes:

1. I want to use RunCommand as under commit dc66841. Because I want to have full control about what is sending and what's the replies are.
2. I need your help because I don't understand all the details of SendAndGetCursor(msg, reply) especially working with BSONVariant and other details (e.g. why has BatchSize a value of 65536 -> I haven't set Batchsize at all)

Last edited by danielkuettner (2022-08-07 11:07:22)

ab · 2022-08-07 12:42:02

With the current implementation, RunCommand() is to run basic commands...
If you run it with commands returning a "cursor", then:
1) the "cursor" internal batch is extracted and added to result set;
2) the "getMore" command is run as many times as needed, to retrieve all the content into the result set.

So if you run the mORMot Find*() or Aggregate*() commands, everything should work as expected, and the matching values (of all batches) are returned as a single array.
As a benefit, those methods will increase the "batchSize" default value, which is very low, as 101 documents. This is OK for the MongoShell, but not from the software point of view. No real DB returns 101 rows and stops.
This comes also from the limitation of the BSON standard itself, which is limited to 16MB maximum per returned message, which is makes this number of 101 rows safe also in this point of view.
I don't understand why MongoDB decided to change the wire protocol with no more backward compatibility, and still suffer from such low-level limitations like 16MB or 101 rows.

I will extend RunCommand() to work as you expect.

danielkuettner · 2022-08-07 13:15:31

Ok, thanks, that sounds better.

I understand your statements totally and I agree with them, but just some thoughts:

1. also I use your aggregate call it doesn't get the rights count documents. There reply count are 101/34398/16795 (if no BatchSize is set).
But in SendAndGetRepliesAndFree Opaque will override every time OnEachReply is called. Here is the issue!

2. the first 101 batch is just for analyzing of the optimal batch count and I trust the algo at server side here. Because every document could have a special size I don't know the optimal BatchSize of my own database.
The second call brings than 38.000 and the last one 16.000. So there is not so much to optimize. If the batches are to large, than in a client-server architecture with many clients, the first client gets his reply some ms faster, but at costs of waiting all others.

Last edited by danielkuettner (2022-08-07 16:13:56)

ab · 2022-08-07 17:19:54

Please try with https://github.com/synopse/mORMot2/commit/91a8ff2a

danielkuettner · 2022-08-07 17:55:51

Perfect!

ab · 2022-08-08 17:22:40

Perhaps https://github.com/synopse/mORMot2/commit/0317ce0e could help too.

danielkuettner · 2022-08-09 11:01:23

Yes, this is could be useful in case of big inserts, I guess.

danielkuettner · 2022-08-13 08:15:16

Nice Blog about new wire protocol! You could be also a salesman or a story-teller. Unusual for a asm optimizer

mORMot Open Source

#1 2022-07-23 19:13:50

MongoDB 6.0

#2 2022-07-24 06:54:53

Re: MongoDB 6.0

#3 2022-07-24 07:59:05

Re: MongoDB 6.0

#4 2022-07-30 15:53:56

Re: MongoDB 6.0

#5 2022-07-30 16:53:35

Re: MongoDB 6.0

#6 2022-07-30 17:55:52

Re: MongoDB 6.0

#7 2022-07-31 07:16:25

Re: MongoDB 6.0

#8 2022-08-02 13:17:02

Re: MongoDB 6.0

#9 2022-08-04 08:24:36

Re: MongoDB 6.0

#10 2022-08-04 18:23:17

Re: MongoDB 6.0

#11 2022-08-05 08:27:24

Re: MongoDB 6.0

#12 2022-08-05 18:22:00

Re: MongoDB 6.0

#13 2022-08-05 21:12:47

Re: MongoDB 6.0

#14 2022-08-06 09:19:30

Re: MongoDB 6.0

#15 2022-08-06 20:46:47

Re: MongoDB 6.0

#16 2022-08-07 07:30:49

Re: MongoDB 6.0

#17 2022-08-07 10:21:31

Re: MongoDB 6.0

#18 2022-08-07 12:42:02

Re: MongoDB 6.0

#19 2022-08-07 13:15:31

Re: MongoDB 6.0

#20 2022-08-07 17:19:54

Re: MongoDB 6.0

#21 2022-08-07 17:55:51

Re: MongoDB 6.0

#22 2022-08-08 17:22:40

Re: MongoDB 6.0

#23 2022-08-09 11:01:23

Re: MongoDB 6.0

#24 2022-08-13 08:15:16

Re: MongoDB 6.0

Board footer