#1 Re: mORMot Framework » Fatal Error (Exception External:?) in Interfaced based Services » 2022-08-27 07:48:55

@sakura You should think about an unwanted exception in your sm and the manual inserted raise as an example for that.
Than it's interesting because I often see such external exceptions in my log files and misinterpreted these.

#2 Re: mORMot Framework » TryLoadLibray PostgreSql lib » 2022-08-25 16:07:39

Are you using 32-bit? Postgres14 comes only with a 64-bit client

#3 Re: mORMot Framework » MongoDB 6.0 » 2022-08-13 08:15:16

Nice Blog about new wire protocol! You could be also a salesman or a story-teller. Unusual for a asm optimizer smile

#4 Re: mORMot Framework » MongoDB 6.0 » 2022-08-09 11:01:23

Yes, this is could be useful in case of big inserts, I guess.

#6 Re: mORMot Framework » MongoDB 6.0 » 2022-08-07 13:15:31

Ok, thanks, that sounds better.

I understand your statements totally and I agree with them, but just some thoughts:

1.  also I use your aggregate call it doesn't get the rights count documents. There reply count are 101/34398/16795 (if no BatchSize is set).
But in SendAndGetRepliesAndFree Opaque will override every time OnEachReply is called. Here is the issue!

2. the first 101 batch is just for analyzing of the optimal batch count and I trust the algo at server side here. Because every document could have a special size I don't know the optimal BatchSize of my own database.
The second call brings than 38.000 and the last one 16.000. So there is not so much to optimize. If the batches are to large, than in a client-server architecture with many clients, the first client gets his reply some ms faster, but at costs of waiting all others.

#7 Re: mORMot Framework » MongoDB 6.0 » 2022-08-07 10:21:31

Test with aggregate has same issue as with RunCommand.

From expected DocumentCount of 51294 aggregate returns only 16905.

I think I could find out where the issue is. In SendAndGetCursor first 34398 Documents of reply are replaced by next 16905. But I miss the first 101 before 34398, that comes with the version of dc66841.

Here are my wishes:

1. I want to use RunCommand as under commit dc66841. Because I want to have full control about what is sending and what's the replies are.
2. I need your help because I don't understand all the details of SendAndGetCursor(msg, reply) especially working with BSONVariant and other details (e.g. why has BatchSize a value of 65536 -> I haven't set Batchsize at all)

#8 Re: mORMot Framework » MongoDB 6.0 » 2022-08-07 07:30:49

Sure I'll try it.

But do you want to say, RunCommand isn't supported from mORMot2 any more (after rewriting)? Although all was still working in commit dc66841?

#9 Re: mORMot Framework » MongoDB 6.0 » 2022-08-06 09:19:30

After removing $db params in my Commands I get following error for aggregates:

'{"ok":0,"errmsg":"Requested getMore on namespace ''Rottler.$cmd'', but cursor belongs to a different namespace Rottler.Persons","code":13,"codeName":"Unauthorized"}'

aggregate was:

{aggregate:"Persons",pipeline:[{$match:{PersonTypeID:0,Parents.ParentID:{$in:[903087]},live:{$ne:false}}},{$sort:{LastProcessingDate:-1}},{$project:{_id:0,ID:"$_id",Number:1,Title:1,SecondName:1,FirstName:1,DayOfBirth:1,Sex:1,PersonTypeID:1,Remarks:1,LastProcessingDate:1,CostCenter:{$arrayElemAt:[{$map:{input:{$filter:{input:"$Parents",as:"p",cond:{$eq:["$$p.PersonTypeID",1]}}},as:"cc",in:"$$cc.Number"}},0]},LoginExpiryDate:1,Alive:1,LoginName:1,Parents.ParentID:1,ContactData:1,DSV:1,FirstSalesOrder:1,LastSalesOrder:1}},{$limit:200}],cursor:{},allowDiskUse:true,maxTimeMS:30000}

Problem seems to be in:

SendAndGetRepliesAndFree->call of getMore ($cmd seems to be to needed under new wire protocol (IIRC). It was a virtual collection for OP_QUERY).

When I remove +'.$cmd' in RunCommand Collections is not stetted:
'{getMore:5068573612759429335,collection:"",batchSize:65536,$db:"Rottler"}' -> because Request.CollectionName = '' in line 2786.

Before rewriting every call with RunCommand gets one replay. The getMore request was done by myself, when requested in the reply.
The replay looks like this:
('{"cursor":{"firstBatch":[{"PersonTypeID":4,"Number":"HLD","Alive":true,...

Now replay is:
'[{"PersonTypeID":4,"Number":"HLD","Alive":true,...

If I use your implementation with getMore I tried to set DatabaseName in RunCommand like this:

BsonVariantType.GetItem(command, 'aggregate', v);
  if not VarIsEmpty(v) then
  GetDocumentsAndFree(NewCommand(
    aDatabaseName + '.' + v, command, flags), returnedvalue)

But the replies read with getMore are still not correct this way.

#10 Re: mORMot Framework » MongoDB 6.0 » 2022-08-05 21:12:47

You are right. I had to remove all my $db params.
Sorry for that.

#11 Re: mORMot Framework » MongoDB 6.0 » 2022-08-05 08:27:24

Just a hint: your commit #0ba4230 don't work with JSON-Objects e.g. listCollections with params, because $db have to be the param after listCollections and not the last one.

wrong:
'{listCollections:"Rottler",filter:{name:"revenuesPayments"},$db:"Rottler"}'

right:
'{listCollections:"Rottler",$db:"Rottler",filter:{name:"revenuesPayments"}}'

You would have ti insert the $db param in the command string that comes from the user. That was not necessary in my opinion. I would add $db param only in case of VarIsStr. In all other cases,
the user must deliver the $db param in his command to work with new protocol. So I had to update all my encapsulate methods and added the $db param without the need to change mORMot2 routines.

The AddItem added the $db param also multiple times: '{listCollections:"Rottler",filter:{name:"revenuesPayments"},$db:"Rottler", $db:"Rottler"}'

Daniel

#12 Re: mORMot Framework » MongoDB 6.0 » 2022-08-04 08:24:36

Hello Arnaud,

thanks for implementing new wire protocol. I'm testing your commit and it looks promising.

Daniel

#13 Re: mORMot Framework » MongoDB 6.0 » 2022-08-02 13:17:02

Hello Arnaud,

I've sent you an email with a new version of mormot.db.nosql.mongodb.pas.
But the PR #114 is the better one.

Daniel

#14 Re: mORMot Framework » MongoDB 6.0 » 2022-07-31 07:16:25

Some changes are necessary e.g. replacement of CursorID (not available in > 3.6).
I will send you an other version of mormot.db.nosql.mongodb.pas as soon as I've fixed last errors on my side.

#15 Re: mORMot Framework » MongoDB 6.0 » 2022-07-30 17:55:52

Some errors produced by TMongoConnection.GetReply are not real errors but comes from old handling of replies (e.g. Request.MongoRequestID has wrong value).

Even my statistic request works!

#16 Re: mORMot Framework » MongoDB 6.0 » 2022-07-30 15:53:56

Hi Arnaud,

after making some PRs further changes are necessary to work with new WIRE protocol.
But creating PRs with GitHub is too painful.
If you want, I can send you my version of mormot.db.nosql.mongodb.pas.

Atm my service starts with auth + ssl, list collections and check indexes and insert an array of documents. And all other stuff looks good from my side, because we only used RunCommand and inserting new $db field is the only change we have to do at our side.

Daniel

#17 Re: mORMot Framework » MongoDB 6.0 » 2022-07-24 07:59:05

Hi Arnaud,

my production service uses the 5.0.9 atm and I'm glad to get this issue in a test env. The little mORMot has grown to a bigger one and I wonder how you deal with it.
If I can do something, please let me know.

Daniel

#18 mORMot Framework » MongoDB 6.0 » 2022-07-23 19:13:50

danielkuettner
Replies: 23

Hi Arnaud,

with MongoDB 6.0 I get following error:

"Unsupported OP_QUERY command" -> https://www.mongodb.com/docs/v6.0/relea … es-removed

It has to be OP_MSG instead.

Same with OP_UPDATE and OP_INSERT and OP_REPLY.

Regards,
Daniel

#19 Re: mORMot Framework » Native TLS Support for mORMot 2 REST or WebSockets Servers » 2022-07-11 14:40:32

Now I've debugged in LogViewer and first, procedure THttpServerResp.Execute was called periodically, but after I've added a log entry for it, it won't called for a while and the error 10053 is also away in that time window. After it will be called, error 10053 is coming up.

#20 Re: mORMot Framework » Native TLS Support for mORMot 2 REST or WebSockets Servers » 2022-07-11 07:35:53

The header is always empty (='').

In mormot.net.sock line 3442 F.BufPtr has following value:

'HTTP/1.1 200 OK'#$D#$A'X-Powered-By: mORMot 1.18 synopse.info'#$D#$A'Server: mORMot (Windows)'#$D#$A'Content-Length: 0'#$D#$A'Content-Type: application/json; charset=UTF-8'#$D#$A'Accept-Encoding: synlz,gzip'#$D#$A'Connection: Keep-Alive'#$D#$A#$D#$A'0627055'

and size is 1024 (always).

In error case sock has following value:

($3A8, '127.0.0.1', '8091', '', '', $36EE460, nil, 30000, 91406, 134278, nil, 0, False, nlTcp, 'PUT /LogService/RemoteLog HTTP/1.1'#$D#$A'Host: 127.0.0.1:8091'#$D#$A'Accept: */*'#$D#$A'User-Agent: Mozilla/5.0 (Win; mORMot 2.0.3606 HCS)'#$D#$A'Keep-Alive: 20000'#$D#$A'Connection: Keep-Alive'#$D#$A'Content-Length: 24'#$D#$A'Content-Type: application/json; charset=UTF-8'#$D#$A#$D#$A'20220711 07315309  +    '#9'02.498.812'#$D#$A'20220711 07303732  -    02.498.840Answer Interface=200 out=15 B in 2.49s'#$D#$A'20220711 07303732 ret   '#9#9'mormot.rest.server.TRestServerRoutingRest(0a167520) {"result":[[]]}swer in=75 B'#$D#$A'20220711 07303501 call  '#9#9'uRestServerDB.TSOneRestServerDB(03028dd0) IIntranet.PollWebRTCAnswer{"Params":{"user":"daniel","accesskey":"fc4eb1e37cf603a315a40e13e533c3a6"}} ENetSock {Message:"THttpClientSocket.SockRecvLn error 10053 after 0 chars"} [TRemoteLog LogService] at 6ac915  '#$D#$A'20220711 07241519 srvr  '#9#9' 192.168.1.214 POST root/Intranet.PollWebRTCAnswer Interface=200 out=15 B in 23.03s'#$D#$A'20220711 07241519 ret   '#9#9'mormot.rest.server.TRestServerRoutingRest(0a167520) {"result":[[]]}'#$D#$A'20220711 07241519  -    '#9'23.037.594'#$D#$A'20220711 07241519  -    23.037.624cf603a315a40e13e533c3a6"}}'#$D#$A'20220711 06555037 srvr  '#9#9' 192.168.1.214 POST root/Intranet.PollWebRTCAnswer Interface=200 out=15 B in 4.81ms'#$D#$A'20220711 06555037 ret   '#9#9'mormot.rest.server.TRestServerRoutingRest(0a1670a0) {"result":[[]]}'#$D#$A'20220711 06555037  -    '#9'00.004.826'#$D#$A'20220711 06555037  -    00.004.851'#$D#$A'20220711 06562747 EXC   ENetSock {Message:"THttpClientSocket.SockRecvLn error 10053 after 0 chars"} [TRemoteLog LogService] at 6ac915  [TRemoteLog LogService] at 6ac915  0220711 06512160  -    '#9'00.654.069'#$D#$A'20220711 06512160  -    00.654.089'#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0

In such cases len = -1 (line 1858 mormot.net.sock)

Perhaps it has nothing to do with updates of mORMot2, but with the body of my request. But I don't know how to get it.

#21 Re: mORMot Framework » Native TLS Support for mORMot 2 REST or WebSockets Servers » 2022-07-10 18:58:36

After updating mORMot2 (last version was from April or May) I get errors (10053) when using remote-logger under Win32/Delphi 10.4.
The remote-logger runs on the same Win2019 machine as the mORMot2 service.
Under linux/fpc I can't see this error.
Perhaps the remote-logger is closing the socket because of a wrong header. Let me know if I can test something. The remote-logger is compiled from the samples of mORMot1.

#22 Re: mORMot Framework » regression for Variants in commits after 96c8a5d3 » 2022-03-24 18:45:34

Thanks!

I've tested your changes and it works in my project without issues.

#23 mORMot Framework » regression for Variants in commits after 96c8a5d3 » 2022-03-23 10:42:22

danielkuettner
Replies: 3

Hello Arnaud,

this code doesn't work with a commit later than 96c8a5d3 (2022-02-12):

var
  aCCGID: Integer;

V:= _OBJ([]);
aCCGID:= 1;
V.ID:= aCCGID;

Debugger shows a EJsonException instead of a parsed json.

(Windows11, Delphi 10.4, 32/64bit)

Daniel

#24 Re: mORMot Framework » ERR_CONTENT_LENGTH_MISMATCH using mORMot2 » 2021-08-12 06:34:17

Ctxt.OutContent is perfectly identical (mORMot1 vs. mORMot2)! Not what I expected.

Seems to be a wrong content-length in http header. How can I check this value? In Ctxt I haven't found it?

As a side note, content won't parsed under mORMot2 by

_Json(JSON, Result, [dvoReturnNullForUnknownProperty, dvoAllowDoubleValue, dvoValueCopiedByReference]);
--> var JSON: RawUTF8, Result: Variant)

whereas under mORMot1 it will.

#25 Re: mORMot Framework » ERR_CONTENT_LENGTH_MISMATCH using mORMot2 » 2021-08-12 04:51:34

I've looked for tools for that too and under linux there are also xxd and vimdiff or colordiff.

#26 Re: mORMot Framework » ERR_CONTENT_LENGTH_MISMATCH using mORMot2 » 2021-08-11 14:57:33

at Service level Result is ok, in Http.Request Ctxt.OutContent is wrong often (not always).
Although length of Ctxt.OutContent is correct, you can't convert Ctxt.OutContent to _JSON (json is invalid here).

With mORMot1 length is the same as under mORMot2, but you can convert Ctxt.OutContent to _JSON without error.

In TBaseWriter.AddNoJsonEscape P can't parsed to json with _Json (TDocVariantData(Value).InitJson(Json, Options) gives false), but P parsed with JSON-Lint says "valid json"

I've to compare the P (mORMot1<>2) tomorrow.

#27 Re: mORMot Framework » ERR_CONTENT_LENGTH_MISMATCH using mORMot2 » 2021-08-11 13:45:04

error still occurs after last update (14d8508bf1578e996542c44887e63efe91115972).

Ctxt.OutContent is not smaller, it's bigger, but seems to be wrong (when I can trust the Browsers).

I will follow your advices and give feedback...

#28 Re: mORMot Framework » ERR_CONTENT_LENGTH_MISMATCH using mORMot2 » 2021-08-11 12:50:37

1. Yes I'm using a reverse proxy (apache2) from the beginning of my career.

2.
Length of Res (Service-Result) = 123579733
Length of Ctxt.OutContent = 123579746 (this is +13 of 123579733 (because of +'{"result":[]}' what's correct))

3.
No, no sockets

4. No exceptions, only in Client(Browser) with ERR_CONTENT_LENGTH_MISMATCH

PS: a Length of 28403957 works, but this is just an example, not a gap

#29 Re: mORMot Framework » ERR_CONTENT_LENGTH_MISMATCH using mORMot2 » 2021-08-11 11:59:51

result := inherited Request(Ctxt);

in HttpServer.Request from a service call with standard RawJSON result (result:[JSON-Content from Service])

#30 Re: mORMot Framework » ERR_CONTENT_LENGTH_MISMATCH using mORMot2 » 2021-08-11 11:33:19

Same configuration, absolute no changes.
We are using mORMot2 for production since 2-3 weeks and all works well. This error comes some days ago. I don't thought about mORMot2 but WLAN/network issues.
But this comes definitely from/with mORMot2 but just for larger responses, small one works fine.

Thanks for your great work again!

PS: for jit I personally would prefer BESEN. What I've seen and heart from bero simply destroys my self-confidence

#31 mORMot Framework » ERR_CONTENT_LENGTH_MISMATCH using mORMot2 » 2021-08-11 08:32:20

danielkuettner
Replies: 15

Hi ab,

under Win64 (XE6) I got this error in all Browsers (Safari, Chrome, Firefox) at xhr response for big contents.
With mORMot(1.18) there is no such error.
Are there options I've to set for mORMot2?

Do you further informations?

Daniel

#32 mORMot Framework » Count of Names/Values of TDocVariantData » 2021-07-24 14:06:27

danielkuettner
Replies: 2

mORMot.core.variants set length to 16 (although real number of fields in json is lower e.g. 11).
Is it an error when Length(TDocVariantData.Names)-1 or High(TDocVariantData.Names) gives 15 or isn't it allowed to access Names/Values with mORMot2?

#34 Re: mORMot Framework » Dangerous IgnoreSSLCertificateErrors defaults » 2021-07-21 10:43:02

TMongoConnection.Open were the right place, but constructor hasn't a param for it.

#35 Re: mORMot Framework » Dangerous IgnoreSSLCertificateErrors defaults » 2021-07-19 16:24:34

Where can I set IgnoreSSLCertificateErrors to true for MongoClient?

#36 Re: mORMot Framework » Exception with external DB access » 2021-05-04 14:23:55

Did you call MapField('ID', 'YourExternalPrimaryKeyIDField')?

#37 Re: mORMot Framework » linker issue under linux with static libs » 2021-04-26 10:46:22

You also shouldn't drink alcohol. But sometimes it's worth doing it.

#39 Re: mORMot Framework » linker issue under linux with static libs » 2021-04-26 08:52:13

Thanks @mpv and @ab.

I had to disable

{.$L static/x86_64-linux/sha512-x64sse4.o}
{.$L static/x86_64-linux/crc32c64.o}

in SynCrypto.pas.

Now the linker is happy again

#40 Re: mORMot Framework » linker issue under linux with static libs » 2021-04-26 07:36:34

LD_LIBRARY_PATH was empty. I've set it to /home/daniel/mORMot2/static/x86_64-linux where sha512-x64sse4.o and others are located, but same errors.
My guess is that it comes from mixing mORMot with mORMot2. I've also tried to replace SynCrypto but its also used in mORMot.pas.

There seems to be a declaration of a variable or function in a header file, which is included in more than one *.c file.

#41 mORMot Framework » linker issue under linux with static libs » 2021-04-25 19:00:32

danielkuettner
Replies: 9

I'm using mORMot but for OpenSslVerify I've used mormon.crypt.openssl from mORMot2 with actual static libs from get_latest_static.sh.
After compiling following error comes from linker:

Linking fpc/bin/x86_64-linux/SOneSrv
/usr/bin/ld: //home/daniel/mORMot2/src/crypt/../../static/x86_64-linux/sha512-x64sse4.o: in function `sha512_sse4':
(.text+0x0): multiple definition of `sha512_sse4'; static/x86_64-linux/sha512-x64sse4.o:(.text+0x0): first defined here
/usr/bin/ld: //home/daniel/mORMot2/src/crypt/../../static/x86_64-linux/crc32c64.o: in function `crc32_iscsi_01':
(.text+0x0): multiple definition of `crc32_iscsi_01'; static/x86_64-linux/crc32c64.o:(.text+0x0): first defined here
/usr/bin/ld: //home/daniel/mORMot2/src/crypt/../../static/x86_64-linux/crc32c64.o: in function `crc32_iscsi_01_slver_01030015':
(.text+0x1566): multiple definition of `crc32_iscsi_01_slver_01030015'; static/x86_64-linux/crc32c64.o:(.text+0x1566): first defined here
/usr/bin/ld: //home/daniel/mORMot2/src/crypt/../../static/x86_64-linux/crc32c64.o: in function `crc32_iscsi_01_slver_01030015':
(.text+0x1566): multiple definition of `crc32_iscsi_01_slver'; static/x86_64-linux/crc32c64.o:(.text+0x1566): first defined here
Error: Error while linking
Fatal: There were 1 errors compiling module, stopping

mORMot2/test/build_fpc.sh compiles and links without errors.

#42 Re: mORMot Framework » Grace is an idea for mORMot - upgrade server with zero downtime » 2021-04-22 12:02:44

VPS, VDS  and Dedicated.

Dedicated is really cheap (ca. 300 EUR p.m.).
For a mongo shared cluster with 9 machines I'm using VPS-L SSD (14,99 per machine p.m.). This works stable but for production I should use VDS instead.

#44 mORMot Framework » Thx for TCrtSocket » 2021-04-18 14:26:13

danielkuettner
Replies: 1

Hi Arnaud,

I just want to say thank you for this great class. It helped me implementing a client for EPSON fiscal server!

Greetings,
Daniel

#45 Re: mORMot Framework » possible to connect to atlas shared? » 2020-11-01 16:45:44

I've made another pull request for using all connections of a cluster.

#46 Re: mORMot Framework » possible to connect to atlas shared? » 2020-11-01 13:39:04

I've made a pull request on GitHub #326 to read from Slaves:

MongoDB.Client.Connections[0].RunCommand('DatabaseName', Command, Res, [mqfSlaveOk]);

#47 Re: mORMot Framework » possible to connect to atlas shared? » 2020-10-27 10:53:38

But nevertheless I need the possibility for TMongoDatase.RunCommand to select the Databasename over a param.

Otherwise the OpenAuth should do the trick and use the admin database in background.

#49 Re: mORMot Framework » possible to connect to atlas shared? » 2020-10-27 08:21:07

Now it seems to work. After getting SEC_I_INCOMPLETE_CREDENTIALS I run InitializeSecurityContext again and get SEC_I_CONTINUE_NEEDED (as at using M0 version of Atlas).

#50 Re: mORMot Framework » possible to connect to atlas shared? » 2020-10-27 06:53:27

TLS version isn't the issue. I've turned back to TLS 1.0 in Atlas and cannot do OpenAuth

When I use free version (M0) in HandshakeLoop (1740) res has 590610 ($00090312).
With M10 res has 590624 ($00090320) -> SEC_I_INCOMPLETE_CREDENTIALS?

@Ab
Possible this link could help?
https://stackoverflow.com/questions/474 … als-unexpe

Sorry, now I've seen your comment in SynWinSock (1748)! This is what I need here.

Board footer

Powered by FluxBB