mORMot Open Source

Vitaly

Member

From: UAE

Registered: 2017-01-31

Posts: 168

Website

ObjectToJSON and void objects

While making the implementation of FHIR we decided to create a big and rather complex structure of resources with classes, based on TSynAutoCreateFields.
Far not all fields of some resource are filled in practice, therefore ObjectToJSON with just [woHumanReadable] (for better display here) for example might result: https://pastebin.com/LTdPjXDm
Such JSON is rather big and mostly contains useless/empty data. So, I added several more options [woDontStore0, woDontStoreEmptyString, woDontStoreDefault], which allowed to get such a result: https://pastebin.com/DRFM6RFj
It became much shorter, but it still contains void objects. To make it fully shortened I had to create an additional function https://pastebin.com/H5eBs4ys, which allowed me to reach the goal: https://pastebin.com/HcAh8V5D

I tried to find some option (smth like woDontStoreVoidObject) but unfortunately failed. Probably I missed smth and just reinvented the wheel. And I afraid this JsonRemoveVoidObjectsAndArrays might be not so optimized...
Anyway, I'd appreciate it if anyone could share his better way of getting a super-compact ObjectToJSON result.

Last edited by Vitaly (2020-06-22 15:21:09)

pvn0

Member

From: Slovenia

Registered: 2018-02-12

Posts: 211

Re: ObjectToJSON and void objects

I guess you could write custom serializer but I think you probably don't want that, another way would be to assign the onWriteObject event which will be fired everytime a property is about to be written so you can return false and avoid the write all together. The downside is you have to figure out on your own if object has published props worth writting which shouldn't be difficult and this will also be miles faster then your current solution.

function ObjectToJSONEx(Value: TObject; Options: TTextWriterWriteObjectOptions;
  OnWriteObjectEvent: TOnTextWriterObjectProp = nil): RawUTF8;
var
  temp: TTextWriterStackBuffer;
begin
  if Value = nil then
    result := NULL_STR_VAR
  else
    with DefaultTextWriterSerializer.CreateOwnedStream(temp) do
      try
        if Assigned(OnWriteObjectEvent) then
           OnWriteObject := OnWriteObjectEvent;   /// <<<<<<<
        CustomOptions := CustomOptions + [twoForceJSONStandard];
        WriteObject(Value, Options);
        SetText(result);
      finally
        Free;
      end;
end;

Vitaly

Member

From: UAE

Registered: 2017-01-31

Posts: 168

Website

Re: ObjectToJSON and void objects

Yes, you're right. I'd like to minimize custom serializers (we'll have enough of them in that huge structure of all resources). Besides this avoiding empty objects in resulting JSON will be applied to all resource objects, so a universal decision seems to me preferable.

I followed your suggestion with OnWriteObject event. The only correction (I understand that it was just a typo in your message): to avoid standard object processing it should return True result.
Anyway, it worked!

I decided to make some speed comparison tests, here are the results on my PC for the same object: https://pastebin.com/zfm5CxRR
Of course, it all highly depends on the object and class and current PC state (time values are very small), still, OnWriteObject implementation works several times faster than JsonRemoveVoidObjectsAndArrays!
Thank you very much!

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,661

Website

Re: ObjectToJSON and void objects

Your initial remark is really accurate: it didn't make sense to write such Name: { } void properties.

Manually re-processing the JSON works, but is indeed not optimized.
And writing a custom event/serializater is too much work to my opinion, and need to be tuned each time you add a new class property.

So I tried to not serialize void class instances if woDontStore0 is set.
Please check https://synopse.info/fossil/info/02fd4270fa

Vitaly

Member

From: UAE

Registered: 2017-01-31

Posts: 168

Website

Re: ObjectToJSON and void objects

Oh, that's really nice, I tested it with a few complex objects only and so far it is working great!
Many thanks, Arnaud!

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

Re: ObjectToJSON and void objects

My serialization was written long time ago (in times when OnWriteObject not exists) so I use a function in "stored". I little bit verbose, because function can't accept parameters, so I need to define it for each property, but also works

TMyClass = 
private
  function canStore_customSettings: Boolean;  
published
  property customSettings: TSomeClass read .. write ... stored canStore_customSettings;
end;

function TMyClass.canStore_customSettings: Boolean;
begin
  Result := (customSettings.Count > 0);
end;

Last edited by mpv (2020-06-23 10:10:01)

Vitaly

Member

From: UAE

Registered: 2017-01-31

Posts: 168

Website

Re: ObjectToJSON and void objects

So I tried to not serialize void class instances if woDontStore0 is set.
Please check https://synopse.info/fossil/info/02fd4270fa

Probably we got one problem with the new IsObjectDefaultOrVoid function.
We have several temporary fields (type variant), which are rather complex by the standard - we'll deal with them a bit later.
Anyway, if this variant field was not assigned, I get an AV in ObjectToJSON with woDontStore0:

Project NextFhirParserDemo.exe raised exception class $C0000005 with message 'c0000005 ACCESS_VIOLATION'.

AV raises at TPropInfo.IsDefaultOrVoid function:

...
  tkVariant: begin
    p := GetFieldAddr(Instance);
    result := (p<>nil) and VarDataIsEmptyOrNull(p^);             <------ here
  end;
...

Am I missing something? Maybe we shouldn't use a variant type for temporarily handling some complex substructures.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,661

Website

Re: ObjectToJSON and void objects

This part of the code didn't change.

I guess you have variant published fields in nested objects.
But this code should not make any AV, unless the variant published field content is wrong - which may happen when it deals with a per-reference variant of a dandling value: i.e. a variant extracted by reference from another variant (e.g. a TDocVariant) which has been released in-between.

We need a minimum reproducible error to investigate...

Vitaly

Member

From: UAE

Registered: 2017-01-31

Posts: 168

Website

Re: ObjectToJSON and void objects

Ok, I made a quick and simple project (please, don't judge strictly), where I could reproduce the same AV: https://pastebin.com/Grkhbmff
It turned out that AV appears with such conditions (at least it seems to me so):
- the variant field is in a nested object;
- the object was formed through JSONToObject;
- initial JSON didn't contain this nested object.

btw, we've got rid of using variant fields already, therefore it shouldn't be a problem anymore for us - just trying to be responsible, because I reported about this AV problem here.

Last edited by Vitaly (2020-06-23 20:28:33)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,661

Website

Re: ObjectToJSON and void objects

Thanks to your sample code, I was able to reproduce the bug.

It should be now fixed, and I have added the associated regression tests.
Please check https://synopse.info/fossil/info/b12052a4c0

Vitaly

Member

From: UAE

Registered: 2017-01-31

Posts: 168

Website

Re: ObjectToJSON and void objects

Great, thanks