mORMot Open Source

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 46

Access violation after the latest commit

In a project that was working normally until the latest commit, now I got the error "Project X raised exception class $C0000005 with message 'access violation at 0x00301608: read of address 0x1c43ef44"

The error happens when I use 'DynArraySaveJson'. It seems to be here:

mormot.core.json -- line 5501

if isHumanReadable in flags then
            c.W.WriteObjectPropNameHumanReadable(pointer(p^.Name), length(p^.Name))
          else
            c.W.AddProp(pointer(p^.Name), length(p^.Name));                                               <-- The error is here

Last edited by Prometeus (2022-12-31 22:26:53)

tbo

Member

Registered: 2015-04-20

Posts: 353

Re: Access violation after the latest commit

The error happens when I use 'DynArraySaveJson'.

Delphi 11.2, mORMot2 GitHub commit 4530

Can you make a small example? I can't reproduce it with the following source code:

type
  TRec = packed record
    firstName: String;
    dateOfBirth: TDateTime;
  end;
  TRecDynArray = array of TRec;
  
var
  recs: TRecDynArray;
begin
  SetLength(recs, 2);
  recs[0].firstName := 'one';
  recs[0].dateOfBirth := Now;
  recs[1].firstName := 'two';
  recs[1].dateOfBirth := Now;
  ShowMessage(Utf8ToString(DynArraySaveJson(recs, TypeInfo(TRecDynArray))));

It works for both 32/64 bit in release/debug mode.

With best regards
Thomas

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 46

Re: Access violation after the latest commit

Hello Thomas,

Thanks for your help. However, right now I am unable to produce a sample to test the problem. I have a lack of time due to the kind of analysis I have to do with the data read through mORMot JSON routines. It is a huge set of data, and it is really time-consuming. I was talking about the concrete case that my test application was not raising an error when I was using:

tmp := DynArraySaveJson(MyData, TypeInfo(MyTypeArray), false);

I changed nothing in my code but the "access violation" error started after the latest or previous commit. Looking a little further, I realized that the error occurs at the following looping:

mormot.core.json - line 5330

repeat
        jsonsave(P, c);  <-- The error occurs here
        dec(n);
        if n = 0 then
          break;
        c.W.BlockAfterItem(c.Options);
        inc(P, s);
      until false;

I am trying to get the previous commits and compare them to see what could have changed.

Last edited by Prometeus (2022-12-31 22:26:14)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Access violation after the latest commit

How is your MyTypeArray defined?

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 46

Re: Access violation after the latest commit

It is like

MyTypeArray = array of packed record
                                               ...
                                              end;

I read a huge JSON that I receive from a REST service and load it using 'RecordLoadJson' without trouble. At some point in my test environment, I need to save the final JSON and it is when I use 'tmp := DynArraySaveJson(MyData, TypeInfo(MyTypeArray), false);'. It was working great until some days ago. Now it raises that "access violation error".

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Access violation after the latest commit

Without a reproducible example, I don't understand what is your problem.

tbo

Member

Registered: 2015-04-20

Posts: 353

Re: Access violation after the latest commit

I read a huge JSON that I receive from a REST service and load it using 'RecordLoadJson' without trouble. At some point in my test environment, I need to save the final JSON and it is when I use 'tmp := DynArraySaveJson(MyData, TypeInfo(MyTypeArray), false);'. It was working great until some days ago. Now it raises that "access violation error".

If I understand you correctly, here's what you do:

type
  TRecDynArray = array of packed record
    firstName: String;
    dateOfBirth: TDateTime;
  end;

var
  json: RawJson;
  recs: TRecDynArray;
begin
  SetLength(recs, 2);
  recs[0].firstName := 'one';
  recs[0].dateOfBirth := Now;
  recs[1].firstName := 'two';
  recs[1].dateOfBirth := Now;
  json := DynArraySaveJson(recs, TypeInfo(TRecDynArray), False);

  SetLength(recs, 0);
  if RecordLoadJson(recs, json, TypeInfo(TRecDynArray)) then
    ShowMessage(Utf8ToString(DynArraySaveJson(recs, TypeInfo(TRecDynArray))));

In this scenario, function RecordLoadJson throws the following exception:

ExceptionMessage="RecordLoadJson: 01021385 is not a record"
ExceptionName="EJsonException"
ExceptionDisplayName="EJsonException"
ExceptionAddress=75C9E062

With DynArrayLoadJson instead of RecordLoadJson it would work.

With best regards
Thomas

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 46

Re: Access violation after the latest commit

Thomas, thanks again for your input.

I am still investigating what has changed, as I told you in the first message, nothing was changed in the code from my side. The error that I got is "access violation", typical of accessing an area of memory not previously reserved, but now the error is been raised with the same data that some days ago was working ok. If I found something new, I'll post it here.

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 46

Re: Access violation after the latest commit

I hope everybody has a Happy New Year!

Let's do a simple chronology of what has happened so far, using Delphi 11.2:

1) My test environment has been working well since March/2022, in this case, using only three mORMot objects/functions: 'TSimpleHttpClient', 'RecordLoadJson', and 'DynArraySaveJson';

2) After the commit right before my first message (or the previous one) on this thread, without changing anything in my code, I began to get an "access violation" error according to what was described above;

3) After the latest commit I got today, the "access violation" is gone. I thought something at 'mormot.core.json' could be the cause of the access violation, but I checked all the latest changes on that unit and found nothing relevant that could lead to that error;

4) Now when I close my test application I get another "access violation" error, now at 'mormot.core.base', precisely on this procedure, line 7086:

procedure ObjArrayClear(var aObjArray);
var
  a: TObjectDynArray absolute aObjArray;
begin
  if a = nil then
    exit;
  // release all owned TObject instances
  RawObjectsClear(pointer(aObjArray), PDALen(PAnsiChar(a) - _DALEN)^ + _DAOFF);      <--- The "access violation" is here
  // release the dynamic array itself
  a := nil;
end;

5) I debugged all my 'finalization' sections (I don't have any 'OnClose/OnDestroy' events in my test application) and all then are clear, no error at all;

6) I personally don't use any function/procedure of 'mormot.core.base'. It is listed in my uses clause just because I use 'RawUtf8' variables and that type is defined there;

7) It looks like some side effect is taking place in my case, but I don't think it is easily reproducible in a 'test case'. As I said, I only use those three 'object/functions' from mORMot in this test application. I really don't know why that function is been called at the end of the application, but it has anything to do with my code.

8) For sure there is a hidden "bug" somewhere in mORMot, and if I have a clue where it is I'll post it here.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Access violation after the latest commit

We run the regression tests with FastMM4 in FullDebugMode, and also on FPC with heaptrc mode, to ensure there is no memory leak or dangling pointers in the framework code.
Is likely to be a dangling pointer error in your code somewhere.

Without a program to reproduce the error, or at least without sending to me as private message the code you use, I don't know what to say.
First run FastMM4 in FullDebugMode on your program. You may find some hints there.

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

Re: Access violation after the latest commit

Such king of bugs often occurs when  variable on stack is of unexpected size (see for example this topic - https://synopse.info/forum/viewtopic.ph … 798#p28798 and fix - https://github.com/synopse/mORMot/pull/299/files)
If so, it can disappear while compiling in debug/release/different optimization level or just changing a unit order. If so, it is a symptom of "dangling pointers" or incorrect var size.

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 46

Re: Access violation after the latest commit

Thanks, ab and mpv, I'll try to follow your tips and see what I can find. Mpv, some of the symptoms you described in the post mentioned I can see too. Those 'ghost' AVs are not easy to find, but I'll keep chasing them.

tbo

Member

Registered: 2015-04-20

Posts: 353

Re: Access violation after the latest commit

I wish everyone a happy new year.

Sorry for my ignorance. But I don't know how to load an array of records with function RecordLoadJson. I don't remember being able to call this function other than with TypeInfo.Kind is Record. I can trigger an access violation with the following source code:

type
  TRec = packed record
    firstName: String;
    dateOfBirth: TDateTime;
  end;
  TRecDynArray = array of TRec;
  
var
  json: RawJson;
  recs: TRecDynArray;
begin
  json := '{"firstName":"one","dateOfBirth":"2023-01-01T20:34:27"}';
  if RecordLoadJson(recs, json, TypeInfo(TRec)) then
    ShowMessage(Utf8ToString(DynArraySaveJson(recs, TypeInfo(TRecDynArray))));

In this scenario, function DynArraySaveJson throws the following access violation:

ExceptionMessage="Access violation at address 0092A122. Read of address 00000004"
ExceptionName="EAccessViolation"
ExceptionDisplayName="$C0000005"
ExceptionAddress=0092A122
FileName="mormot.core.json.pas"
LineNumber=6748

Are you sure that no definition error has slipped into your example?

With best regards
Thomas

Last edited by tbo (2023-01-01 20:48:31)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Access violation after the latest commit

RecordLoadJson() is to load a record, and you load it into a TRecDynArray, which is a dynamic array.
So it won't work for sure.
RecordLoadJson() fills the recs variable (which is a pointer) with the fields of the TRec record. So you fill memory with garbage, which trigger the AV.

Something correct may be:

SetLength(recs, 1);
 if RecordLoadJson(recs[0], json, TypeInfo(TRec)) then
    ShowMessage(Utf8ToString(DynArraySaveJson(recs, TypeInfo(TRecDynArray))));

tbo

Member

Registered: 2015-04-20

Posts: 353

Re: Access violation after the latest commit

RecordLoadJson() is to load a record, and you load it into a TRecDynArray, which is a dynamic array.
So it won't work for sure.

Yes, I know that. I'm just speculating a bit on how the error could possibly arise. I include errors on purpose to trigger an access violation similar to the example, or at least try to.

This source code also brings an access violation in DynArraySaveJson as above:

var
  json: RawJson;
  recs: TRec;
begin
  json := '{"firstName":"one","dateOfBirth":"2023-01-01T20:34:27"}';
  if RecordLoadJson(recs, json, TypeInfo(TRec)) then
    ShowMessage(Utf8ToString(DynArraySaveJson(recs, TypeInfo(TRecDynArray))));

With best regards
Thomas

Last edited by tbo (2023-01-01 21:38:27)

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 46

Re: Access violation after the latest commit

I wish everyone a happy new year.
Sorry for my ignorance. But I don't know how to load an array of records with function RecordLoadJson. I don't remember being able to call this function other than with TypeInfo.Kind is Record. I can trigger an access violation with the following source code:

Thomas, since I discovered the outstanding mORMot framework almost three years ago, I realized I should start using it deeply in my projects, but it requires some education to begin playing with it. For the project in question, and as I am still learning mORMot, I created a testing environment application where I also learn mORMot fundamentals, and then next I use what is mature in the real projects. I was never happy with the fact that when serializing/unserializing JSON I was not using the right counterpart function, as I had tested the right pair or functions (like 'DynArrayLoadJson / DynArraySaveJson', 'RecordLoadJson / RecordSaveJson') and it never worked as I was expecting, probably due something I was doing wrong. I never felt well when I was mixing those previous functions to read/write JSON. The main JSON I have to deal with I defined like this:

TMyArray = array of packed record
                           ...
                           end;

TMyEntity = packed record
                     MyArray : TMyArray;
                   end; 

The "combination" that worked, in this case, was something like this:

When reading:

var TempRec : TMyEntity;

RecordLoadJson(TempRec, my_json TypeInfo(TMyEntity));

When writing:

var tmp : RawJson;

tmp := DynArraySaveJson(MyEntity.MyArray, TypeInfo(TMyArray), false);

I never make that work if I had used the pair "DynArrayLoadJson / DynArraySaveJson". This has been working since March/2022. No "access violation" until some weeks ago, then AV for the writing part, and after the last commit it came back to normal (ALL that without any change in my code), but now I got an AV in a unit that I don't use at all.

You said that you don't know how to load an array of records with 'RecordLoadJson'. The way I did above allow me to do it, although that would sound weird because 'RecordLoadJson' is used to load a 'TMyEntity' (that is a record of an array of record), I guess the whole bunch of JSON records are treated as a big JSON, but I am not sure. I would like to simply use 'DynArrayLoadJson / DynArraySaveJson' for that, but I never knew how to do it.

Last edited by Prometeus (2023-01-02 01:37:48)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,660

Website

Re: Access violation after the latest commit

DynArrayLoadJson / DynArraySaveJson work with no problem on my side.
Please provide us with some code to reproduce the problem.

The code you gave us above was incorrect.
We showed you why. Please go back to my answer and try to understand it. I am not sure that you got the root cause of your problem. https://synopse.info/forum/viewtopic.ph … 795#p38795
Don't speculate - try to understand.

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 46

Re: Access violation after the latest commit

DynArrayLoadJson / DynArraySaveJson work with no problem on my side.
Please provide us with some code to reproduce the problem.

When I begin to play with mORMot 'DynArrayLoadJson' was not working properly, but today I could make it work. I refactored the type used from:

TMyArray = array of packed record
                    ...
                    end;

TMyEntity = packed record
                     MyArray : TMyArray;
                   end; 

To:

TMyData  = packed record
                  ...
                  end;
TMyArray = array of TMyData; 

  Now, I can read/write the JSON data with "DynArrayLoadJson / DynArraySaveJson" the same way that my "wrong" code did, but I am still getting AV when closing the test application. I am still checking what is causing that. Thanks!

  PS: The 'AV' occurs only inside Delphi. I still need to try FastMM4 like you suggested

Last edited by Prometeus (2023-01-03 03:12:58)