Interface based services and RowID serilization

tbo · 2021-10-19 17:14:42

Concerns mormot2

Today I changed mORMot2 for my server from version 2021-07-27 to the current version. The server has the following service function:

procedure TSectionService.GetAllLookupItems(out pmoList: TOrmSectionObjArray);
  ...
  restServer.Server.RetrieveListObjArray(pmoList, TOrmSection, _SQL, [Ord(asActive)], RawUtf8ArrayToCsv(['RowID', TOrmSection.OPN.Name]));

With version 2021-07-27 the result has looked like this:
[{"ID":1,"Name":"Main"}]

With newer versions it looks like this:
[{"RowID":1,"Name":"Main"}]

If I call the service in the client as follows:

var
  service: ISection;
  dataArr: TOrmSectionObjArray;
begin
  ...
  service.GetAllLookupItems(dataArr);

Until version 2021-07-27 was the result:
[{"ID":1,"Name":"Main"}]

After this version:
[{"ID":0,"Name":"Main"}]

The mormot version for the client software does not matter. What do I need to change in current mormot versions to get it working again?

With best regards
Thomas

Last edited by tbo (2021-10-19 17:20:52)

ab · 2021-10-19 19:36:22

What is the transmitted JSON?

tbo · 2021-10-19 20:28:35

ab wrote:

What is the transmitted JSON?

As described in the first text and determined with the following functions:
Server: WriteLn(ObjArrayToJson(pmoList, []));
Client: ShowMessage(Utf8ToString(ObjArrayToJson(dataArr, [])));

And from the server log files (mORMot2 version 2021-07-27)
{"result":[[{ID:1,Name:"Main"}]]}

And mormot version 2021-08-13 and newer versions:
{"result":[[{RowID:1,Name:"Main"}]]}

Previously "ID" was used in the json, now "RowID". It is no longer deserialized correctly in the client. The value for "ID" is always 0.

With best regards
Thomas

ab · 2021-10-20 06:28:02

I am not able to reproduce.
"RowID" is properly decoded in TOrm.RttiJsonRead.

Are you sure the client has been updated too?

tbo · 2021-10-20 10:26:00

ab wrote:

Are you sure the client has been updated too?

Preliminary remarks:

I do not use packages, only pure source code.
Each program has its own development environment (VMs). The server is developed with Delphi 10.4.2. The client with Delphi XE.
I replace the mORMot libraries as follows: Close the project; close Delphi; delete all mORMot library and DCU directories; reinstall the mORMot libraries; start Delphi; open the project.
I always make a completely new build.

The used mORMot version in the client does not matter. I have tested the following versions: 2021-07-27, 2021-08-13, 2021-10-11, 2021-10-14, 2021-10-19, 2021-10-20.
The client is under active development and I switch to the latest mORMot2 version every few weeks. The server runs unchanged for several months.

Yesterday I wanted to upgrade the server to the current mORMot2 version. With the recompiled server, the client stopped working.
I iteratively tested equal and different mORMot2 versions on server and client. The result is: when I compile the server with the version from 2021-07-27 it works normally, all later versions show the behavior described above.

Today I did it again with the current commit c33caae. The same behavior. Server and client with 2021-10-20 do not work. Server with 2021-07-27 and client with 2021-10-20 work.

With best regards
Thomas

Last edited by tbo (2021-10-20 10:26:44)

ab · 2021-10-20 14:14:29

I have added some tests for your use case.
Please see https://github.com/synopse/mORMot2/comm … b593ce5a87
They do pass with no problem.

Please check your code and provide us - in a separated download - with a https://stackoverflow.com/help/minimal- … le-example

tbo · 2021-10-20 19:46:00

Sorry, I thought you had this post in mind.

The problem is simple. With the change in the JSON result from "ID" to "RowID" after mORMot version 2021-07-27, the serialization does not work anymore.

In object TObjectWithID only the "ID" property is considered.

class procedure TObjectWithID.RttiCustomSetParser(Rtti: TRttiCustom);
begin
  Rtti.Props.Add(TypeInfo(TID), PtrInt(@TObjectWithID(nil).fID), 'ID', {first=}true);
end;

If you add the following to the above function just for testing purposes:

Rtti.Props.Add(TypeInfo(TID), PtrInt(@TObjectWithID(nil).fID), 'RowID', {first=}true);

The JSON in the client will look like this: [{"ID":1,"RowID":1,"Name":"Main"}]
In the JSON result, "ID" and "RowID" have the correct result.

With best regards
Thomas

ab · 2021-10-20 21:38:48

As I wrote above:
"RowID" is properly decoded in TOrm.RttiJsonRead.

Both ID and RowID are accepted.
The TRttiCustom.Props list is not used in ORM unserialization.

My guess is that you are mixing TOrm and not TOrm classes, and it breaks the expectations.
With one my commits from yesterday, plain TObjecWithID descendants should be able to decode also "RowID" initial field - did you include https://github.com/synopse/mORMot2/comm … 5b9999298d in your tests ?

tbo · 2021-10-21 17:53:38

Thanks for putting so much effort into this.

ab wrote:

As I wrote above:
"RowID" is properly decoded in TOrm.RttiJsonRead.
Both ID and RowID are accepted.
The TRttiCustom.Props list is not used in ORM unserialization.

mORMot2, Commit: 1e4b132 (2021-10-21)
JSON from server: {"result":[[{RowID:1,Name:"Main"}]]}

In function _JL_RttiCustomProps() "RowID" is not found for the object TObjectWithID.
Detailed course:

procedure TInterfacedObjectFake.FakeCallInternalProcess(var ctxt: TFakeCallContext);
  procedure FakeCallSetJsonToStack(ctxt, pointer(OutputJson));
  -> OutputJson = [[{RowID:1,Name:"Main"}]]
     procedure FromJson() -> TRttiJsonLoad(ArgRtti.JsonLoad)(V, ctxt)
        procedure _JL_DynArray(Data: PAnsiChar; var Ctxt: TJsonParserContext) -> load(Data, Ctxt); // will call _JL_RttiCustom() for T*ObjArray
           procedure _JL_RttiObjectWithID(Data: PAnsiChar; var Ctxt: TJsonParserContext)
           -> Ctxt.Json = {RowID:1,Name:"Main"}]]
              procedure _JL_RttiCustom(Data, Ctxt); // use default serialization
              -> Ctxt.Json = {RowID:1,Name:"Main"}]]
                 procedure _JL_RttiCustomProps(Data, Ctxt);
                 -> Ctxt.Json = {RowID:1,Name:"Main"}]]
                 nxt:  propname := GetJsonPropName(Ctxt.Json, @propnamelen);
                 propname = 'RowID'
                 if prop^.NameMatch(propname, propnamelen) then
                 ==> "RowID" is not found and it jumps to the following branch
                 // we didn't find the property in its natural place -> full lookup
                 -> Ctxt.Json =  1,Name:"Main"}]]
                 ==> In this case "RowID" is simply cut off!

If not "RowID" but "ID" is in the JSON, the search is successful!

if prop^.NameMatch(propname, propnamelen) then
  if JsonLoadProp(Data, prop^, Ctxt) then
  -> Ctxt.Json = Name:"Main"}]]

I can check this if I replace "ID" with "RowID" in the function TObjectWithID.RttiCustomSetParser(), then "RowID" has the correct value and also a test with FormatUtf8('ID: % - Name: %', [obj.IDValue, obj.Name]) shows the correct result.

If no other user has a problem with it, everything is OK. I have no problem with it. I know how to solve it for my purpose.

Thank you very much for mORMot.

With best regards
Thomas

Last edited by tbo (2021-10-21 17:55:51)

ab · 2021-10-22 17:17:56

What I don't understand is that TObjectWithID indeed calls _JL_RttiObjectWithID() which is supposed to detect {RowID: and replace it with {ID: so that _JL_RttiCustom() works as expected.

Isn't it the case?

tbo · 2021-10-23 18:53:13

ab wrote:

Isn't it the case?

If I dispense all the tricky optimizations and write it in the simplest way possible, it works fine:

procedure _JL_RttiObjectWithID(Data: PAnsiChar; var Ctxt: TJsonParserContext);
...
if (p^ = '"') and ((p + 1)^ = 'R') and ((p + 2)^ = 'o') and ((p + 3)^ = 'w') and ((p + 4)^ = 'I') and ((p + 5)^ = 'D') and ((p + 6)^ = '"') then
begin
  p^ := '_';
  (p + 1)^ := '_';
  (p + 2)^ := '{';
  (p + 3)^ := '"';
  Ctxt.Json := p + 2;
end
else if (p^ = 'R') and ((p + 1)^ = 'o') and ((p + 2)^ = 'w') and ((p + 3)^ = 'I') and ((p + 4)^ = 'D') and ((p + 5)^ = ':') then
begin
  p^ := '_';
  (p + 1)^ := '_';
  (p + 2)^ := '{';
  Ctxt.Json := p + 2;
end;

With best regards
Thomas

tbo · 2021-10-23 20:19:19

If optimization, then following is already a help:

(PCardinal(P + 4)^ and $ffdf = ord('D') + ord(':') shl 8) then

The many $f... bring one completely out of concept. I think you've already seen the mistake and just waited until my poor brain is all messed up.

With best regards
Thomas

Last edited by tbo (2021-10-23 20:35:40)

ab · 2021-10-25 12:42:58

Please try with my latest commits.

I think I got the issue.

All those $df are for efficient case-insensitive match of A-Z chars.
There was a wrong $df instead of $ff for one last non A-Z char.

tbo · 2021-10-25 18:19:41

ab wrote:

There was a wrong $df instead of $ff for one last non A-Z char.

This was also the solution in my fix from post #12. "$ffdf" instead of "$dfdf". Commit 0556cea runs without problems.

With best regards
Thomas

mORMot Open Source

#1 2021-10-19 17:14:42

Interface based services and RowID serilization

#2 2021-10-19 19:36:22

Re: Interface based services and RowID serilization

#3 2021-10-19 20:28:35

Re: Interface based services and RowID serilization

#4 2021-10-20 06:28:02

Re: Interface based services and RowID serilization

#5 2021-10-20 10:26:00

Re: Interface based services and RowID serilization

#6 2021-10-20 14:14:29

Re: Interface based services and RowID serilization

#7 2021-10-20 19:46:00

Re: Interface based services and RowID serilization

#8 2021-10-20 21:38:48

Re: Interface based services and RowID serilization

#9 2021-10-21 17:53:38

Re: Interface based services and RowID serilization

#10 2021-10-22 17:17:56

Re: Interface based services and RowID serilization

#11 2021-10-23 18:53:13

Re: Interface based services and RowID serilization

#12 2021-10-23 20:19:19

Re: Interface based services and RowID serilization

#13 2021-10-25 12:42:58

Re: Interface based services and RowID serilization

#14 2021-10-25 18:19:41

Re: Interface based services and RowID serilization

Board footer