JSONToObject example - nested objects

Serrg · 2013-12-10 02:26:58

I am trying to parse the following JSON string:

{"transactions":[
{
 "TRTYPE":"INCOME",
 "TRDATE":"2013-12-09 02:30:04",
 "TRAA":"1.23",
 "TRCAT1":{"TITYPE":"C1","TIID":"1","TICID":"","TIDSC30":"description1","TIORDER":"0","TIDEL":"false"},
 "TRCAT2":{"TITYPE":"C2","TIID":"2","TICID":"","TIDSC30":"description2","TIORDER":"0","TIDEL":"false"},
 "TRCAT3":{"TITYPE":"C3","TIID":"3","TICID":"","TIDSC30":"description3","TIORDER":"0","TIDEL":"false"},
 "TRRMK":"Remark",
 "TRACID":{"TITYPE":"AC","TIID":"4","TICID":"","TIDSC30":"account1","TIORDER":"0","TIDEL":"false"}
},
{
  "TRTYPE":"INCOME",
...
}
]}

I've declared the following classes:

type
  TTitleRecord = class(TCollectionItem)
  private
    fTITYPE: RawUTF8;
    fTIID: RawUTF8;
    fTICID: RawUTF8;
    fTIDSC30: RawUTF8;
    fTIORDER: RawUTF8;
    fTIDEL: RawUTF8;
  published
    property TITYPE: RawUTF8 read fTITYPE write fTITYPE;
    property TIID: RawUTF8 read fTIID write fTIID;
    property TICID: RawUTF8 read fTICID write fTICID;
    property TIDSC30: RawUTF8 read fTIDSC30 write fTIDSC30;
    property TIORDER: RawUTF8 read fTIORDER write fTIORDER;
    property TIDEL: RawUTF8 read fTIDEL write fTIDEL;
  end;

  TTransactionRecord = class(TCollectionItem)
  private
    fTRAA: RawUTF8;
    fTRTYPE: RawUTF8;
    fTRTYPEID: integer;
    fTRCAT1: TTitleRecord;
    fTRCAT2: TTitleRecord;
    fTRCAT3: TTitleRecord;
    fTRDATE: RawUTF8;
    fTRRMK: RawUTF8;
    fTRACID: TTitleRecord;
  published
    property TRAA: RawUTF8 read fTRAA write fTRAA;
    property TRTYPE: RawUTF8 read fTRTYPE write fTRTYPE;
    property TRTYPEID: integer read fTRTYPEID write fTRTYPEID;
    property TRCAT1: TTitleRecord read fTRCAT1 write fTRCAT1;
    property TRCAT2: TTitleRecord read fTRCAT2 write fTRCAT2;
    property TRCAT3: TTitleRecord read fTRCAT3 write fTRCAT3;
    property TRDATE: RawUTF8 read fTRDATE write fTRDATE;
    property TRRMK: RawUTF8 read fTRRMK write fTRRMK;
    property TRACID: TTitleRecord read fTRACID write fTRACID;
  end;

  TTransactionRecords = class(TInterfacedCollection)
  private
    function GetCollItem(aIndex: Integer): TTransactionRecord;
  protected
    class function GetClass: TCollectionItemClass; override;
  public
    function Add: TTransactionRecord;
    property Item[aIndex: Integer]: TTransactionRecord read GetCollItem; default;
  end;

function TTransactionRecords.GetCollItem(aIndex: Integer): TTransactionRecord;
begin
  result := TTransactionRecord(GetItem(aIndex));
end;

class function TTransactionRecords.GetClass: TCollectionItemClass;
begin
  result := TTransactionRecord;
end;

function TTransactionRecords.Add: TTransactionRecord;
begin
  result := TTransactionRecord(inherited Add);
end;

When I feed TTransactionRecords object to JSONToObject function - it fails to parse it. It stumbles when it parses TRCAT1 which is TTitleRecord type. I guess I can't make TTransactionRecord a TCollectionItem if it contains other objects... Am I on the right page?

Help with proper class definitions would be greatly appreciated!

ab · 2013-12-10 13:36:56

Your JSON is a bit weird here...
You are using JSON strings everywhere, whereas JSON numbers and boolean may have been defined instead.
OK - this is not the point here...

Sounds more like a value object than a regular TSQLRecord object.

What you can do in this case is to use a record and not a class to define your structure.
With the brand new "custom serialization of records from text definition", it is very easy to do it.

First, you define your record (with any nested record, if necessary).
Your records should be defined as PACKED, otherwise you may have access violation issues:

type
  TTestCustomJSON2Title = packed record
    TITYPE,TIID,TICID,TIDSC30,TIORDER,TIDEL: RawUTF8; 
  end;
  TTestCustomJSON2 = packed record
    Transactions: array of record
      TRTYPE: RawUTF8;
      TRDATE: TDateTime;
      TRAA: RawUTF8;
      TRCAT1, TRCAT2, TRCAT3, TRACID: TTestCustomJSON2Title;
      TRRMK: RawUTF8;
    end;
  end;

Then, you write the corresponding record description, which should match the same definition than the pascal source code:

const
  __TTestCustomJSON2 = 'Transactions [TRTYPE RawUTF8 TRDATE TDateTime TRAA RawUTF8 '+
    'TRCAT1,TRCAT2,TRCAT3,TRACID{TITYPE,TIID,TICID,TIDSC30,TIORDER,TIDEL RawUTF8} '+
    'TRRMK RawUTF8]';

Now you register your type:

  TTextWriter.RegisterCustomJSONSerializerFromText(TypeInfo(TTestCustomJSON2),__TTestCustomJSON2);

... or with some options if you want more readable JSON content for instance:

  TTextWriter.RegisterCustomJSONSerializerFromText(TypeInfo(TTestCustomJSON2),
    __TTestCustomJSON2).Options := [soWriteHumanReadable];

Now you can easily parse your JSON content and access it via the record fields:

  U := '{"transactions":[{"TRTYPE":"INCOME","TRDATE":"2013-12-09 02:30:04","TRAA":"1.23",'+
   '"TRCAT1":{"TITYPE":"C1","TIID":"1","TICID":"","TIDSC30":"description1","TIORDER":"0","TIDEL":"false"},'+
   '"TRCAT2":{"TITYPE":"C2","TIID":"2","TICID":"","TIDSC30":"description2","TIORDER":"0","TIDEL":"false"},'+
   '"TRCAT3":{"TITYPE":"C3","TIID":"3","TICID":"","TIDSC30":"description3","TIORDER":"0","TIDEL":"false"},'+
   '"TRRMK":"Remark",'+
   '"TRACID":{"TITYPE":"AC","TIID":"4","TICID":"","TIDSC30":"account1","TIORDER":"0","TIDEL":"false"}}]}';
  RecordLoadJSON(Trans,@U[1],TypeInfo(TTestCustomJSON2));
  Check(length(Trans.Transactions)=1);
  Check(Trans.Transactions[0].TRTYPE='INCOME');
  Check(Trans.Transactions[0].TRACID.TIDEL='false');
  Check(Trans.Transactions[0].TRRMK='Remark');

Note that your record definition properties order may not follow the JSON incoming content. This is not a problem for RecordLoadJSON().
And you are also able to write back the record content as JSON:

  U := RecordSaveJSON(Trans,TypeInfo(TTestCustomJSON2));
  FileFromString(U,'transactions.json');

which will create the following JSON file:

{
	"Transactions": [
		
		{
			"TRTYPE": "INCOME",
			"TRDATE": "2013-12-09T02:30:04",
			"TRAA": "1.23",
			"TRCAT1": 
			{
				"TITYPE": "C1",
				"TIID": "1",
				"TICID": "",
				"TIDSC30": "description1",
				"TIORDER": "0",
				"TIDEL": "false"
			},
			"TRCAT2": 
			{
				"TITYPE": "C2",
				"TIID": "2",
				"TICID": "",
				"TIDSC30": "description2",
				"TIORDER": "0",
				"TIDEL": "false"
			},
			"TRCAT3": 
			{
				"TITYPE": "C3",
				"TIID": "3",
				"TICID": "",
				"TIDSC30": "description3",
				"TIORDER": "0",
				"TIDEL": "false"
			},
			"TRACID": 
			{
				"TITYPE": "AC",
				"TIID": "4",
				"TICID": "",
				"TIDSC30": "account1",
				"TIORDER": "0",
				"TIDEL": "false"
			},
			"TRRMK": "Remark"
		}]
}

Note also that a dynamic array of such records will also be identified and serialized/unserialized as expected.

What do you think?
IMHO this is easier than using TCollection/TCollectionItem.

See this blog article about custom JSON serialization.
http://blog.synopse.info/post/2013/12/1 … ialization

ncook · 2013-12-11 22:17:48

Hi Arnaud.
I really like the new serialisation option for records. It seems to be simple and much less error prone that the older technique.

You said in the blog post that the supported Delphi data types were (boolean byte word integer cardinal Int64 single double TDateTime TTimeLog string RawUTF8 SynUnicode WideString).
Is that list of supported data types extendible by registering additional types in some way?

If it is then I guess we can add anything else we need. That's great. But if not, there is one more type that would make this 1000% more useful: TGUID.
So if we can't register additional types, is there any chance that TGUID could be added to the list of supported types please?

Regards
Neville

ab · 2013-12-12 14:35:27

Good idea.

Now TTextWriter.RegisterCustomJSONSerializerFromText() supports JSON serialization/unserialization of TGUID values (encoded as JSON strings).
See http://synopse.info/fossil/info/767c34eae7

Serrg · 2013-12-12 20:35:48

Hi Arnaud,

You are right - JSON with nested records is received from non-Mormot server. My server side is developed with Java->hibernate->php.

I tried to use TTextWriter.RegisterCustomJSONSerializerFromText as you suggested.
It threw me an error when the parser came across the nested object. This one: "ESynException.Create('V1,V2,..V16')".

Unit SynCommons.pas
...

procedure TJSONCustomParserFromTextDefinition.Parse
begin
...
    case P^ of
    ',': if PropsMax=cardinal(high(Props)) then
          raise ESynException.Create('V1,V2,..V16') else begin
          inc(P);
          inc(PropsMax);
          continue;
        end;
    ':': inc(P);
    end;
...

Still I was able to convince my java/php developer to get rid of these trees and just send flat JSON.
So I am good. :-)

Last edited by Serrg (2013-12-12 20:37:10)

ab · 2013-12-12 21:57:49

Your version of the code is too old.

This has already been fixed.

Serrg · 2013-12-13 01:52:48

Thanks for your help Arnaud,

I just downloaded the nightly build yesterday before doing my tests. SynCommons.pas is dated Nov 29, 2013.

This is the full JSON that I need to serialize:

    JSONOut:='{"result":{"ok":"true","message":"","syncTime":"2013-12-12T20:55:47.000000"},
             "transactions":['+
             '{"TRTYPE":"INCOME","TRDATE":"2013-12-12 20:45:20","TRAA":"1.23","TRCAT1":"1","TRCAT2":"2","TRCAT3":"3","TRRMK":"Remark","TRACID":"4"},'+
             '{"TRTYPE":"EXPENSE","TRDATE":"2013-12-12 20:45:20","TRAA":"1.23","TRCAT1":"1","TRCAT2":null,"TRCAT3":null,"TRRMK":"Remark","TRACID":"5"}'+
             ']}';

I tried everything I could find in the demos/forum: JSONToObject and LoadFromJSON. Nothing worked so far.
And my java/php developer will not bend any further. :-)

Can you advise what should I use? Records or Objects?

Appreciate your help!

ab · 2013-12-13 05:56:15

As I wrote above, records is easier in your case.

See my answer above.

esmondb · 2013-12-20 09:02:04

I've just been experimenting with TTextWriter.RegisterCustomJSONSerializerFromText and using RecordLoadJSON which works great except if it is called more than once. The second time RecordLoadJSON is called it causes an AV while calling FinalizeNestedRecord.

This can be seen if you repeat lines 4035 - 4041 in SynSelfTests.pas more than once (copied below)

Thanks in advance for your help

U := '{"transactions":[{"TRTYPE":"INCOME","TRDATE":"2013-12-09 02:30:04","TRAA":"1.23",'+
   '"TRCAT1":{"TITYPE":"C1","TIID":"1","TICID":"","TIDSC30":"description1","TIORDER":"0","TIDEL":"false"},'+
   '"TRCAT2":{"TITYPE":"C2","TIID":"2","TICID":"","TIDSC30":"description2","TIORDER":"0","TIDEL":"false"},'+
   '"TRCAT3":{"TITYPE":"C3","TIID":"3","TICID":"","TIDSC30":"description3","TIORDER":"0","TIDEL":"false"},'+
   '"TRRMK":"Remark",'+
   '"TRACID":{"TITYPE":"AC","TIID":"4","TICID":"","TIDSC30":"account1","TIORDER":"0","TIDEL":"false"}}]}';
  RecordLoadJSON(Trans,@U[1],TypeInfo(TTestCustomJSON2));

ab · 2013-12-20 09:14:06

As documented, all those functions change the input string in place.

So it will work only once, unless you make a private copy of the string.

esmondb · 2013-12-20 09:21:28

I my situation I'm using a different string each time.

I modified SynSelfTests so the lines above are run twice consecutively and it gives an AV

ab · 2013-12-20 10:53:48

Oups...

There was an issue when clearing the record content at re-loading from JSON.
I have added a corresponding regression test.

Should be fixed now.
See http://synopse.info/fossil/info/6e28c20745

Thanks for your feedback.

esmondb · 2013-12-20 11:18:46

Thanks for the speedy fix

esmondb · 2013-12-21 11:23:20

btw this feature is great for parsing MQL query responses from www.freebase.com (hope the plug doesn't get pulled on this one)

ab · 2013-12-21 13:00:36

Indeed...

I did not know MQL syntax.
http://wiki.freebase.com/wiki/Mql
But it is pretty interresting, and we may implement something similar to our remote ORM (in addition to the standard SQL "where" clause).

TheBlacksheep · 2019-06-27 10:11:52

Am posting here because the posting on the webpage suggested this particular topic - apologies if this is incorrect.

I am using a customreader function (D2007) to convert a JSON string to a record based on the example reference on;

http://blog.synopse.info/post/2013/12/1 … ialization

namely the example;

class function TTestServiceOrientedArchitecture.CustomReader(P: PUTF8Char;
  var aValue; out aValid: Boolean): PUTF8Char;
var V: TSQLRestCacheEntryValue absolute aValue;
    Values: TPUtf8CharDynArray;
begin
  result := JSONDecode(P,['ID','TimeStamp','JSON'],Values);
  if result=nil then
    aValid := false else begin
    V.ID := GetInteger(Values[0]);
    V.TimeStamp := GetCardinal(Values[1]);
    V.JSON := Values[2];
    aValid := true;
  end;
end;

but am trying to update my mORMot library and JSONDecode no longer accepts a TPUtf8CharDynArray (possibly changed to TValuePUTF8CharArray) but am struggling to work out what this should look like instead.

Thanks in advance

Chris

ab · 2019-06-27 11:08:25

Just define

   Values: array[0..2] of TValuePUTF8Char;

since you need 3 parameters.

(switching from a dynamic array to a static array enhance performance and stability, since the heap is not used)

Note that if your V.JSON is a RawUTF8, you could write:

   Values[2].ToUTF8(V.JSON);

which is slightly better.

Or even, to be more consistent:

   V.ID := Values[0].ToInteger;
   V.TimeStamp := Values[1].ToCardinal;
   Values[2].ToUTF8(V.JSON);

TheBlacksheep · 2019-06-27 12:13:59

excellent - sorted!

thanks

mORMot Open Source

#1 2013-12-10 02:26:58

JSONToObject example - nested objects

#2 2013-12-10 13:36:56

Re: JSONToObject example - nested objects

#3 2013-12-11 22:17:48

Re: JSONToObject example - nested objects

#4 2013-12-12 14:35:27

Re: JSONToObject example - nested objects

#5 2013-12-12 20:35:48

Re: JSONToObject example - nested objects

#6 2013-12-12 21:57:49

Re: JSONToObject example - nested objects

#7 2013-12-13 01:52:48

Re: JSONToObject example - nested objects

#8 2013-12-13 05:56:15

Re: JSONToObject example - nested objects

#9 2013-12-20 09:02:04

Re: JSONToObject example - nested objects

#10 2013-12-20 09:14:06

Re: JSONToObject example - nested objects

#11 2013-12-20 09:21:28

Re: JSONToObject example - nested objects

#12 2013-12-20 10:53:48

Re: JSONToObject example - nested objects

#13 2013-12-20 11:18:46

Re: JSONToObject example - nested objects

#14 2013-12-21 11:23:20

Re: JSONToObject example - nested objects

#15 2013-12-21 13:00:36

Re: JSONToObject example - nested objects

#16 2019-06-27 10:11:52

Re: JSONToObject example - nested objects

#17 2019-06-27 11:08:25

Re: JSONToObject example - nested objects

#18 2019-06-27 12:13:59

Re: JSONToObject example - nested objects

Board footer