Deserializing JSON on mormot2

Prometeus · 2022-01-19 19:26:53

Hi there,

While learning more about mORMot2 I am doing some tests with JSON. I am loading a JSON and trying to deserialize it with 'DynArrayLoadJson'. You can check it here https://github.com/PrometeusRec/mormot_ … n/Test.pas

The JSON file test is there as well. However, no 'JSON' is being loaded into the array. My questions are:

1) Is the JSON content different from what 'DynArrayLoadJson' expects, or is there some error in the type of the fields?

2) On that sample 'packed record' created there is a commented field that is not present in the JSON that I need to read (the exact type shown in the '.TXT' file). The original 'JSON' file may have more fields than the 'packed record' that you will need, but the contrary is not allowed? I would like to make some processing and save the results in fields of the same 'packed record'. Is that possible?

Thanks!

tbo · 2022-01-19 20:30:02

Prometeus wrote:

1) Is the JSON content different from what 'DynArrayLoadJson' expects, or is there some error in the type of the fields?

Try this:

TBar = packed record
  t: TDateTime;
  o: Double;
  ...
end;
TBarDynArray = array of TBar;

TBars = packed record
  bars: TBarDynArray;
end;
  
var
  tmp: RawByteString;
  barsHubs: TBars;
begin
  tmp := StringFromFile('20220117222332_HTTPRESPONSE.TXT');
  if tmp <> '' then
    RecordLoadJson(barsHubs, Pointer(tmp), TypeInfo(TBars));
  ...

With best regards
Thomas

ab · 2022-01-19 20:32:40

DynArrayLoadJson() works as expected - no need of using RecordLoadJson():

tmp := StringFromFile('20220117222332_HTTPRESPONSE.TXT');
DynArrayLoadJson(BarsHubs,pointer(tmp),TypeInfo(TBarsHubs));

In the original code, @tmp was incorrect: it returns the adress where the pointer string is stored, not the pointer string itself containing the text.

tbo · 2022-01-19 20:47:25

ab wrote:

DynArrayLoadJson() works as expected - no need of using RecordLoadJson():

I have now tested it in Delphi 10.4.2, mORMot2 Version 2021-12-22. Only my version works correctly!

With best regards
Thomas

Prometeus · 2022-01-19 22:00:10

Hi Thomas and Arnaud!

I can confirm that. I am using Delphi 10.3.3. 'DynArrayLoadJson' didn't work in this case. The fix shown by Thomas worked as expected, but why was it necessary to create a 'packed record' type of the array of the fields loaded from the JSON, and not the array type itself? I didn't see that need when I read the item '10.1.3.2.4 Text-based definition' of mORMot's documentation.

Thank you!

ab · 2022-01-19 23:17:58

It is because your code is incorrect.

The JSON is an object holding an array, not an array itself.

Prometeus · 2022-01-19 23:30:29

ab wrote:

It is because your code is incorrect.
The JSON is an object holding an array, not an array itself.

Ok, thanks!

tbo · 2022-01-20 10:46:47

Prometeus wrote:

...but why was it necessary to create a 'packed record' type of the array of the fields loaded from the JSON, and not the array type itself?

Sorry, I saw your mistake and should have added the explanation right away. But after the long day at work, I just wanted to turn off the computer as soon as possible.

With best regards
Thomas

Prometeus · 2022-05-15 16:47:34

I am still learning the HUGE JSON framework of mORMot, but there are situations where I still don't know how to figure them out, like the one below:

1) I need to read JSON values that are like the one in the file 'JSON_Test.TXT' (link below). The values there are not properly an array (looking at the JSON file there is no '[',']' pair), but look like one, and neither is a sole record. Is there a mORMot function that can read it and fill an 'array of packed record' able to be read by a call like 'DynArrayLoadJson' for that situation?

2) Another question: Is there a way to read the values that precede the initial record ('Record01') of the set of records for each entity? For the specific file the values 'ABC' and 'DEF'?

https://github.com/PrometeusRec/JSON_Test

Thanks for any help!

Last edited by Prometeus (2022-05-15 16:49:15)

igors233 · 2022-05-15 17:07:07

1) That is a valid JSON, it just doesn't write objects as array but as separate objects, Record01, Record02...
You'll have to convert it to array manually, you can use TDocVariantData.InitJSON to read and fill whole content and then you'll have to go through it manually and convert it to your internal array of records.

2) Yes, if you load that json in TDocVariantData you'll have json object with two properties, ABC and DEF (both are objects as well). So you could read it as
with (there are other options), GetValueOrNull('ABC').

You can also use late binding, for example
var
v: Variant;
begin
v := _JsonFast(YourJSONContent);
WriteLn(v.ABC.Record01.ax);
end;

Prometeus · 2022-05-15 17:14:35

igors233 wrote:

1) That is a valid JSON, it just doesn't write objects as array but as separate objects, Record01, Record02...
You'll have to convert it to array manually, you can use TDocVariantData.InitJSON to read and fill whole content and then you'll have to go through it manually and convert it to your internal array of records.

Thanks for the tip, I'll check it out.

2) Yes, if you load that json in TDocVariantData you'll have json object with two properties, ABC and DEF (both are objects as well). So you could read it as
with (there are other options), GetValueOrNull('ABC').

When you know the values it is ok, but what if you don't know previously that the values will be 'ABC', 'DEF'? Is there a parse function that will give you those values, so you can try the solution you told below for late binding?

tbo · 2022-05-15 20:05:58

Prometeus wrote:

When you know the values it is ok, but what if you don't know previously that the values will be 'ABC', 'DEF'? Is there a parse function that will give you those values, so you can try the solution you told below for late binding?

The first thing I would do is format the json file to be more readable for humans, this way you can see the structure better.

FileFromString(JsonReformat(StringFromFile('_testData.json')), '_testFmtData.json');

In your case you will find the object names as follows:

var
  doc: TDocVariantData;
begin
  if not doc.InitJsonFromFile('_testData.json', JSON_OPTIONS_FAST_EXTENDED) then Exit; //=>
  WriteLn(RawUtf8ArrayToCsv(doc.Names));

In response to a question in the Delphi-PRAXiS forum, I posted function ConvertJsonToCsv(). Source code is here. The explanations are in German, but the source code should be understandable. You can see how I traverse the structure, object or array with objects, and create a CSV file from it.

With best regards
Thomas

Prometeus · 2022-05-16 04:08:21

tbo wrote:

In your case you will find the object names as follows:

var
  doc: TDocVariantData;
begin
  if not doc.InitJsonFromFile('_testData.json', JSON_OPTIONS_FAST_EXTENDED) then Exit; //=>
  WriteLn(RawUtf8ArrayToCsv(doc.Names));

Thanks, Thomas, ALL worked like a charm!

Prometeus · 2022-05-17 17:03:04

Hello there,

Using the help provided here I could convert the JSON from a REST service from the format above shown on 'JSON_Test.TXT' to an array that is better handled by my application ('JSON_Test_03.txt'). However, although now it is an array I can't load it to an array using 'DynArrayLoadJson' as shown here: https://github.com/PrometeusRec/JSON_Te … N_Test.dpr

'new_array' is empty after calling 'DynArrayLoadJson'. I guess it was supposed to load that record correctly, but for some reason, it doesn't.

In the same application, I successfully loaded other JSON arrays similar to the one above, but sometimes 'DynArrayLoadJson' won't work as expected. Is there something wrong with the JSON file above for it not working?

Last edited by Prometeus (2022-05-18 02:51:32)

ab · 2022-05-17 18:12:58

On Delphi, there is extended RTTI so I guess it should work.
Try perhaps to define a record type, then an array of this record.

If the JSON does not exactly match the array definition, try to set Tolerant=true to DynArrayLoadJson().

My first guess was that '2022-05-11T23:59:56.971655858Z' (pretty weird value) is not properly decoded by _JL_DateTime/Iso8601ToDateTimePUtf8CharVar. But I made some tests, and it was fine.
Try to debug a little bit and find out where it breaks.

Edit:
I guess your code is wrong because you use String which is UnicodeString = UTF-16 on Delphi.
Replace tmp: RawUtf8; by tmp: RawUtf8; and I guess it would work.

Prometeus · 2022-05-17 18:38:20

ab wrote:

...
Edit:
I guess your code is wrong because you use String which is UnicodeString = UTF-16 on Delphi.
Replace tmp: RawUtf8; by tmp: RawUtf8; and I guess it would work.

That's it!! A 'RawUtf8' did the thing!!

I knew that the record as it is defined here is well dealt with by 'mORMot' because it worked before for other situations like that, but I didn't realize about 'RawUtf8'!

Thank you, Ab!

Last edited by Prometeus (2022-05-18 15:32:16)

mORMot Open Source

#1 2022-01-19 19:26:53

Deserializing JSON on mormot2

#2 2022-01-19 20:30:02

Re: Deserializing JSON on mormot2

#3 2022-01-19 20:32:40

Re: Deserializing JSON on mormot2

#4 2022-01-19 20:47:25

Re: Deserializing JSON on mormot2

#5 2022-01-19 22:00:10

Re: Deserializing JSON on mormot2

#6 2022-01-19 23:17:58

Re: Deserializing JSON on mormot2

#7 2022-01-19 23:30:29

Re: Deserializing JSON on mormot2

#8 2022-01-20 10:46:47

Re: Deserializing JSON on mormot2

#9 2022-05-15 16:47:34

Re: Deserializing JSON on mormot2

#10 2022-05-15 17:07:07

Re: Deserializing JSON on mormot2

#11 2022-05-15 17:14:35

Re: Deserializing JSON on mormot2

#12 2022-05-15 20:05:58

Re: Deserializing JSON on mormot2

#13 2022-05-16 04:08:21

Re: Deserializing JSON on mormot2

#14 2022-05-17 17:03:04

Re: Deserializing JSON on mormot2

#15 2022-05-17 18:12:58

Re: Deserializing JSON on mormot2

#16 2022-05-17 18:38:20

Re: Deserializing JSON on mormot2

Board footer