Automatic JSON serialization of record via Enhanced RTTI

avista · 2014-06-05 04:47:15

Arnaud,

Thanks for implementing this feature!

It's ironic, because I spent quite a bit of time implementing this feature in SynCommons.pas, and was about to send my changes to you when you blogged about it.

In any case, there are a few scenarios my code supports that the new built-in serialization does not:

1. Static Arrays of both simple and record types (serialization will fail with any of the declarations below with an exception)

  TMyRecord = packed record
    Name: string;
    Age: Integer;
  end;

  TIntArray = array[1..5] of Integer;

  TRecArray = array[1..5] of TMyRecord;

  TTestRec = record
    IntArrayInline: array[1..5] of Integer;
    IntArrayNamed: TIntArray;
    RecArrayInline: array[1..5] of TMyRecord;
    RecArrayNamed: TRecArray;
  end;

You may ask why not just use dynamic arrays? Because sometimes it's cleaner to use a static array (especially if a fixed number of items are assumed in the code) and I have existing records with static arrays that must be serialized.

2. Serialization Options

  TJSONCustomParserSerializationOption = (
    soReadIgnoreUnknownFields, soWriteHumanReadable,
    soCustomVariantCopiedByReference);

Specifically soReadIgnoreUnknownFields and soWriteHumanReadable, which are important for my application.

I had also added another option to trim the prefixes from enumerated types both when writing and reading them:

  TJSONRecordParserOption = (poTrimLeftLowerCaseEnums);

Would it be possible to implement support for static arrays and the serialization options in the new RTTI record serialization code?

On another note, I'd like to send you the modified code with a small sample application for testing, so you can see how I implemented it (maybe it could be useful, but I expect that your implementation is much better). I tried following the existing code conventions as much as possible.

Where can I post/email my modified SynCommons.pas file so you can get it?

ab · 2014-06-05 08:09:31

Nice!

You can use webcontact01 at synopse dot info

Thanks for sharing!

martin.suer · 2014-06-05 08:31:14

Hi,

ab, if you're going to extend on this topic, I also have a feature request...

Dynamic arrays and Records are serialized into binary format when they're published properties of any TSQLRecord descendant when that's stored in a db. That's fine and is working nicely. But, when time comes one might have to extend a type and then the already stored records in the db are no longer compatible to the new record type and can't be deseriealized any more to the new version of that type - which is understandable...

While e.g. TDocVariants are serialized into json in a db record field, (since those don't have an explicit structure), you can simply introduce new fields in such types. But here I don't have compile time type checking of course.

For me ( and maybe others... ? ) it would be of great help if it were possible to have a global or per TSQLRecord or per Field configuration option for record and dynarray serialization in public properties of TSQLRecord descendants.That option should force serialization in json format instead of binary format. Then deserializing such record would be possible even if the type of the record has changed. (Like it is possible by manual calling RecordLoadJSON and using the soReadIgnoreUnknownFields option). My main point here is not the serialization format but the possibility for schema evolution and I think going for JSON format would be the easiest way...
Of course I could define custom serialization formats per record but this leads to a lot of handwriting which potentially can be done automatically. And since your enhanced usage of RTTI for 2010++ versions of Delphi I started to throw out all of the record registration calls...

Also I could use TCollection/TCollectionItem descendants as a record alternative everywhere. Sometimes that makes sense, sometimes not. DynArrays/Records often make more sense. I just have the problem that when Record Types change, then I have do convert existing DB-Content before it's usable with the changed Record Type.

What do you think?

Martin

ab · 2014-06-05 08:36:26

@Martin

You are right: binary serialization is not able to let record/dynamic array definition evolve.
I do not see any possibility to allow this.
There is just not enough information in the "old" RTTI: only managed fields info.

AFAIK under Delphi 2010 and up, the serialization will be in JSON thanks to enhanced RTTI.
So you can easily let the record/dynamic array evolve, and add new fields, without any issue.

You are right, record/dynamic arrays are great for "value objects", as defined by DDD patterns.

What is your exact concern?
That dynamic arrays are not serialized as JSON in TSQLRecord published fields?

martin.suer · 2014-06-05 08:50:32

The exact concern is schema evolution.

I am using XE5 and nightly build (may be 3-5 days old) and dynamic arrays and records are still binary in db fields and not JSON. I can make them to serialize to JSON but then I have to write custom serialization code or at minimum have to register my record types ...

To circumvent that I could also use a RawJSON property in a TSQLRecord and manually serialize / deserialize with RecordSaveJSON/RecordLoadJSON but that wrapping is all extra flights which I thought I could circumvent if I just could set an option 'somewhere' to tell mormot to use json instead of binary format.
E.g. one could use the storage parameter for DynArray or Record properties and introduce a storage binary and storage json ...

and maybe that only works with newer delphi versions, the older ones could continue to use binary format...

Last edited by martin.suer (2014-06-05 08:52:42)

ab · 2014-06-05 09:02:30

Could you please create a feature request in http://synopse.info/fossil/tktnew ?

It should be named like "Store TSQLRecord dynamic arrays published properties as JSON".

martin.suer · 2014-06-05 09:06:16

will do

ab · 2014-06-06 11:49:09

I've just added static array support for record JSON serialization using enhanced RTTI.
Sounds working as expected.

I due follow another path than in your solution, just because I wanted not to use the RTTI.pas unit, which is slower than direct access.
In fact, I enhanced TJSONCustomParserCustomSimple to support tkArray kind of content.

This modification could be made compatible with text-based record serialization in the future, if needed.

See http://synopse.info/fossil/info/927b93657b

avista · 2014-06-20 19:19:04

I've added support to the code for enabling serialization options for records (e.g., soReadIgnoreUnknownFields, soWriteHumanReadable).

  TJSONCustomParserFromRTTI = class(TJSONCustomParserAbstract)
  protected
    fRecordTypeInfo: pointer;
    function AddItemFromRTTI(const PropertyName: RawUTF8;
      Info: pointer; ItemSize: integer): TJSONCustomParserRTTI;
    {$ifdef ISDELPHI2010}
    procedure FromEnhancedRTTI(Props: TJSONCustomParserRTTI; Info: pointer);
    {$endif}
  public
    /// initialize the instance
    // - you should NOT use this constructor directly, but let e.g.
    // TJSONCustomParsers.TryToGetFromRTTI() create it for you
    constructor Create(aRecordTypeInfo: pointer; aRoot: TJSONCustomParserRTTI); reintroduce;
    /// set custom serialization options for records
    class procedure SetRecordSerializationOptions(aRecordTypeInfo: pointer; aOptions: TJSONCustomParserSerializationOptions); // <--------------
    /// the low-level address of the enhanced RTTI
    property RecordTypeInfo: pointer read fRecordTypeInfo;
  end;

[...]

class procedure TJSONCustomParserFromRTTI.SetRecordSerializationOptions(
  aRecordTypeInfo: pointer; aOptions: TJSONCustomParserSerializationOptions);
var
  ndx: Integer;
begin
  if (aRecordTypeInfo=nil) or (PFieldTable(aRecordTypeInfo)^.kind<>tkRecord) then
    raise ESynException.Create('Invalid record type in TJSONCustomParserFromRTTI.SetRecordSerializationOptions');
  ndx := GlobalJSONCustomParsers.RecordSearch(aRecordTypeInfo);
  if (ndx>=0) then
    GlobalJSONCustomParsers.fParser[ndx].RecordCustomParser.Options := aOptions;
end;

Example:

procedure SetMyRecordSerializationOptions;
var
  Options: TJSONCustomParserSerializationOptions;
begin
  Options := [soReadIgnoreUnknownFields,soWriteHumanReadable];
  TJSONCustomParserFromRTTI.SetRecordSerializationOptions(TypeInfo(TCustomerRecord), Options);
end;

I've created a corresponding ticket for this here:

http://synopse.info/fossil/info/da22968223

Last edited by avista (2014-06-20 19:25:15)

ab · 2014-06-21 04:54:53

I've implemented your proposal.
With some modifications, e.g. about method name, some border case handling, and new aAddIfNotExisting: boolean parameter.

See http://synopse.info/fossil/info/4e708e600e

Thanks a lot for your feedback!

avista · 2014-06-21 20:29:54

Thanks!

petrus · 2021-06-16 12:13:12

Hello,

first of all I'd like to say that the mORMot is a great project and very useful.

Now my subject:

I was serializing a record (TTestJSON) with some "sub" records (TTestJSON2 and TTestJSON3) with RecordSaveJSON and the Option soWriteHumanReadable with RegisterCustomJSONSerializerSetOptions.
My problem was that the CustomParser with the option was only used for the record registered with RegisterCustomJSONSerializerSetOptions.
For Records, which are part of my so sad "base" record, the Option was not active.

That is due to the fact that for every found tkCustom type (record), a CustomParser is created and stored in TJSONCustomParsers.fParser. And those do not get the Options.
So if you have records inside records the options only apply to the record registered and not to the records inside the record.

This is my record:

type TaiFeedback_SAP_HU_Gen_Box = ARRAY[1..10] OF integer;
type TaxMAC_ProtectionDoorsClosed = ARRAY[0..15] OF Boolean;
type TrModule_CycleTime = ARRAY[0..10] OF REAL;
type TadwMAC_PartsInBoxCounter = ARRAY[1..20] OF DWORD;

TYPE TTestJSON3 = packed record

	bWriteSkValueToPackage			: BYTE;
  	aiFeedback_SAP_HU_Gen_Box		: TaiFeedback_SAP_HU_Gen_Box;
	strMsgText						: String; 	

end;
TYPE TTestJSON2 = packed record

	bWriteSkValueToPackage			: BYTE;
  	aiFeedback_SAP_HU_Gen_Box		: TaiFeedback_SAP_HU_Gen_Box;
  	strMsgText						: String; 	
  	sTestJSON3             : TTestJSON3;

end;


TYPE TTestJSON = packed record
	diMsgNr							: DINT;
	strMsgText						: String; 	
   	axMAC_ProtectionDoorsClosed     : TaxMAC_ProtectionDoorsClosed;	
	rModule_CycleTime				: TrModule_CycleTime;
	bWriteSkValueToPackage			: BYTE;
  	aiFeedback_SAP_HU_Gen_Box		: TaiFeedback_SAP_HU_Gen_Box;
	adwMAC_PartsInBoxCounter			: TadwMAC_PartsInBoxCounter;
	sTestJson2 : TTestJson2;
	bCrc							: BYTE;

end;

This is the implementation:

function writeJSONFromRecord(var Data; DataType : Pointer; filename : String; xHumanReadable : Boolean) : BOOLEAN;
var
   JsonStringRawUTF8 : RawUTF8;
   Options : TJSONCustomParserSerializationOptions;

begin
   result := false;

   if xHumanReadable then begin
      Options := [soWriteHumanReadable];
      if not TTextWriterWithEcho.RegisterCustomJSONSerializerSetOptions(DataType, Options, true) then
      exit;
   end;
   JsonStringRawUTF8 := RecordSaveJSON(Data, DataType);
 
end;

After that I changed some parts of the code of SynCommons and after that with an additional Boolean in RegisterCustomJSONSerializerSetOptions, it now registers all records inside the record with the Options.
My solution is probably not very good but at least it works. If somebody of the mORMot team wants the code, I would provide it
Correct me if I'm wrong but for me that was the only way.

Maybe a better useful solution would be an global Option variable for all Registered types without an manual set option?

mORMot Open Source

#1 2014-06-05 04:47:15

Automatic JSON serialization of record via Enhanced RTTI

#2 2014-06-05 08:09:31

Re: Automatic JSON serialization of record via Enhanced RTTI

#3 2014-06-05 08:31:14

Re: Automatic JSON serialization of record via Enhanced RTTI

#4 2014-06-05 08:36:26

Re: Automatic JSON serialization of record via Enhanced RTTI

#5 2014-06-05 08:50:32

Re: Automatic JSON serialization of record via Enhanced RTTI

#6 2014-06-05 09:02:30

Re: Automatic JSON serialization of record via Enhanced RTTI

#7 2014-06-05 09:06:16

Re: Automatic JSON serialization of record via Enhanced RTTI

#8 2014-06-06 11:49:09

Re: Automatic JSON serialization of record via Enhanced RTTI

#9 2014-06-20 19:19:04

Re: Automatic JSON serialization of record via Enhanced RTTI

#10 2014-06-21 04:54:53

Re: Automatic JSON serialization of record via Enhanced RTTI

#11 2014-06-21 20:29:54

Re: Automatic JSON serialization of record via Enhanced RTTI

#12 2021-06-16 12:13:12

Re: Automatic JSON serialization of record via Enhanced RTTI

Board footer