Commit from Oct 1st breaks seemingly unrelated code

sakura · 2021-10-08 06:47:19

Hi Arnaud,

I am currently stumped, as to why a commit from Oct 1st breaks some code for me.
Commit: https://github.com/synopse/mORMot2/comm … 0296d37348

In this commit you have introduced the method TJsonWriter.AddColumn and you use it in TSqlRequest.Execute:

    // directly assign column names from SQlite3 API into W
    for i := 0 to FieldCount - 1 do
      W.AddColumn(sqlite3.column_name(Request, i), i, FieldCount);

When I change that code back to

    // get col names and types
    SetLength(W.ColNames, FieldCount);
    for i := 0 to FieldCount - 1 do
      W.ColNames[i] := sqlite3.column_name(Request, i);
    W.AddColumns; // write or init field names for appropriate Json Expand

my code, using your Rtti-methods for reading/writing properties of my objects works just fine. With the new changes, that code breaks completely. I am not saying, that I might not have made a mistake in my rtti-related code, however, for now, all comes down to that change, whether my object transpose system works, or not :-|

Partial stack trace, when error occurs:

20211008 06535548 0000009233EFCFD0  ! EXCOS EAccessViolation (c0000005) [Main] 
at 51b3d7 mormot.core.unicode.pas TSynAnsiFixedWidth.AnsiBufferToUtf8 (2853)  
@ExceptionHandler mormot.core.unicode.pas TSynAnsiConvert.AnsiBufferToRawUtf8 (2560) 
mormot.core.unicode.pas TSynAnsiConvert.AnsiToUtf8 (2545) 
mormot.core.text.pas VariantToUtf8 (8761) 
mormot.core.text.pas VariantToUtf8 (8839) 
mormot.core.rtti.pas TRttiCustomProp.SetValue (6227)

A little more information, as to the settings used in AddColumn(s):
fExpand evaluates to True
twoForceJsonExtended in CustomOptions evaluates to False
---> routing to ColNames[ i ] := '"' + ColNames[ i ] + '":'; in AddColumns.

Would you have an idea, what might break here, would you like remote access to my machine, to see what's going on?

Regards,
Daniel

P.S.: Yet there is a relation - I have just executed a SQL-request and copy those results into my object.

Last edited by sakura (2021-10-08 07:00:30)

ab · 2021-10-08 11:27:20

What is the generated JSON content?
Are the ColNames[] content correct?

Please retry with my latest update.

sakura · 2021-10-08 11:37:43

ab wrote:

Please retry with my latest update.

First off: the problem persists with the latest update.

ab wrote:

What is the generated JSON content?

'{"fieldCount":4,"values":["ID","Version","CreatedAt","HostName"],"rowCount":0}'#$A

Simple table, no content, perfectly valid json, I'd say.

ab wrote:

Are the ColNames[] content correct?

Yes.

Can't really explain what's going on myself.

ab · 2021-10-08 12:18:06

What is the full stack trace when the error occurs?

sakura · 2021-10-08 12:28:08

The rest of the call stack is completely within my sources.

First, I run the query against the database, to retrieve the current version. I use the ORM-classes directly to check. This is the moment, when the problem is "initiated".

Noticing, that the database is not initialized, I want to create my standard user. I create my user using my KDD-classes for user management and use my own mapping-tools, to write the data into the user ORM-classes. Those tools use the mormot.core.rtti to map between the classes and read/write the properties. This is, where the error occurs.

So, it does not really have to do anything with either the sqlite access, nor with the ORM classes used at this point.

However, yet, when using the rtti methods from mormot.core.rtti, the problem arises, as long, as I do not change the code to the version mentioned above. Debugging into the mapping, it looks like the memory pointers got corrupted for the properties.

ab · 2021-10-08 12:45:59

So I guess it has nothing to do with TSqlRequest.Execute but with latest variant / RTTI refactoring.
Perhaps the JSON buffer is parsed twice with no temporary copy in-between. So it is unescaped + with its nested #0 which is not valid JSON any more.

Can you try to find out which exact commit is the culprit?

sakura · 2021-10-08 12:52:24

ab wrote:

Can you try to find out which exact commit is the culprit?

It is the one, mentioned in the first post. Undoing just this one commit (using git tools), I tried it this morning, and staying otherwise up to the latest commits, as of the time before I have started this thread, everything works as expected...

P.S.: I have also tried, the commits up to that mentioned - all newer fail, the older ones work just fine.

Last edited by sakura (2021-10-08 13:06:56)

ab · 2021-10-09 07:47:42

Do you use ColNames[] in your code?

sakura · 2021-10-09 09:01:45

No, I do not anywhere :-)

However, I can confirm to you, that the source of the problem arises from within the AddColumn source. When I change it to:

  len := StrLen(aColName);
  if fExpand then
  begin
    if aColIndex = 0 then // non-expanded mode doesn't use ColNames[]
      SetLength(ColNames, aColCount);
//    collen := len + 1;
// ...
//    P[len] := ':';
      ColNames[aColIndex] := '"' + UTF8String(aColName) + '":';
    end
    else
    begin
....

everything works fine.

Regards,
Daniel

ab · 2021-10-09 10:53:52

This is very weird.
I can't find the root cause, nor reproduce the issue.

Please check with my latest commit.

sakura · 2021-10-09 11:23:07

It does not change the outcome and playing around a bit, I have not found anything that could solve it, either. Still assigning the names "the classical way" works, which stumps me as to why it should work that way, but not yours. :-|

ab · 2021-10-09 12:14:21

OK - I just changed it to use FormatUtf8() which should be a safe and efficient way of constructing the ColNames[] values with a single memory allocation.

sakura · 2021-10-11 10:27:04

Hi Arnaud,

thanks for all your thoughts, as this did not change anything, I have tried to compile a minimal example. Likely, the fault lies with me :-(

See new thread please.

Thank you for all your help,
Daniel

mORMot Open Source

#1 2021-10-08 06:47:19

Commit from Oct 1st breaks seemingly unrelated code

#2 2021-10-08 11:27:20

Re: Commit from Oct 1st breaks seemingly unrelated code

#3 2021-10-08 11:37:43

Re: Commit from Oct 1st breaks seemingly unrelated code

#4 2021-10-08 12:18:06

Re: Commit from Oct 1st breaks seemingly unrelated code

#5 2021-10-08 12:28:08

Re: Commit from Oct 1st breaks seemingly unrelated code

#6 2021-10-08 12:45:59

Re: Commit from Oct 1st breaks seemingly unrelated code

#7 2021-10-08 12:52:24

Re: Commit from Oct 1st breaks seemingly unrelated code

#8 2021-10-09 07:47:42

Re: Commit from Oct 1st breaks seemingly unrelated code

#9 2021-10-09 09:01:45

Re: Commit from Oct 1st breaks seemingly unrelated code

#10 2021-10-09 10:53:52

Re: Commit from Oct 1st breaks seemingly unrelated code

#11 2021-10-09 11:23:07

Re: Commit from Oct 1st breaks seemingly unrelated code

#12 2021-10-09 12:14:21

Re: Commit from Oct 1st breaks seemingly unrelated code

#13 2021-10-11 10:27:04

Re: Commit from Oct 1st breaks seemingly unrelated code

Board footer