General serialization of custom types

dc · 2016-11-28 12:37:15

Hi, I have following example:

type
  TDecimal2 = Currency;
  TDecimal3 = Currency;

  TTestClass1 = class
  public
    val2: TDecimal2;
    val3: TDecimal3;
  end;

  TTestClass2 = class
  public
    field: TDecimal3;
  end;

I would like to serialize this objects with custom serializer. I know how to do it with TJSONSerializer.RegisterCustomSerializer and it all works great. However I'm not quite happy with that method because you have to either:

prepare serialization individually for different classes and repeat reader and writer code for every custom type, which is not very DRY
prepare one big God like class for serializing everything and make your code look more like spaghetti with bunch of cases and type comparations

Either way code gets hard to maintain very quickly.

What I'd like to do is rather:

register custom reader and writer for main class(es) you need or let's say TObject which covers only serialization/deserialization flow
register custom reader and writer for specific simple types like TDecimal2, TDecimal3 which would use registered custom readers/writers

I've spend some time on analyzing SynCommons.pas and mORMot.pas code but can't find a way to register readers and writers for simple types. Is it correct or do I miss something? I think such a feature would be beneficial for the framework and it's something I can contribute to. What do you think?

dc

ab · 2016-11-28 13:03:47

IMHO you should better use records instead of classes, if your purpose is to have some simple "value objects".
Then, you won't need to call create/free and so on.

In mORMot, an alternative using classes is to use TSynPersistent and TSynAutoCreateFields, and published fields (not public fields) - with lists handled as T*ObjArray.
This will make the serialization automated, and also will allocate/free all nested classes.
And you will gain for instance list storage, via T*ObjArray dynamic arrays.

What do you call "simple types"?

dc · 2016-11-28 14:01:11

I have some "data models" in my application, some of them are records but most are classes. From my perspective it's not that important, let's assume I can't change it and would like to be able to work with both.

I need to have much more control of serialization/deserialization than just looking at published. That's why I use RegisterCustomSerializer() to begin with, so I can alert the way serialization and deserialization works. My point is that having only RegisterCustomSerializer() leads to ugly code, which could be easily made better by using some register for simple types. From this perspective using TSynPersistent won't help either.

By simple data types I mean all possible native Delphi data types but records, objects and classes (i.e. integer, currency and so on). To that group I also add custom types like TDecimal2 = Currency from my example code.

Maybe a use case will show it better:

Let's say I need to serialize numbers differently based on precision. Numbers with precision up to 2 fields are supposed to always serialize with two decimal places, and numbers with precision up to 3 always with 3 decimal places.

Then in an application I would just declare two new types TDecimal2 and TDecimal3, booth as Currency, and use a custom reader and writer for serialization and deserialization. It separates my business needs from implementation details and even gives me opportunity to use parts of my code in different places. It can work with JSON serialization but also XML as well.

dc

ab · 2016-11-28 14:08:29

What you describe sounds to me like the "Data Transfer Object" (DTO) pattern, well-known in SOA.
You just define record/TSynAutoCreateFields DTO types for custom serialization, then write some simple marshallers between your business classes and those DTO.
See http://martinfowler.com/eaaCatalog/data … bject.html

Martin Fowler wrote:

Although the main reason for using a Data Transfer Object is to batch up what would be multiple remote calls into a single call, it's worth mentioning that another advantage is to encapsulate the serialization mechanism for transferring data over the wire. By encapsulating the serialization like this, the DTOs keep this logic out of the rest of the code and also provide a clear point to change serialization should you wish.

Serialization will then use DTO layout, and will be much easier to do with JSON or XML - note that there is a JSONToXML() function (and other related) to help from JSON-XML process.

dc · 2016-11-29 09:09:56

I'm familiar with DTO and problem is not in lack of design patterns in my application. Maybe I gave you too trivial examples, but it's not problem with complexity of models, it's more about scale: I have around 160 models with different serialization and deserialization needs. I can usually quite easy define a type that has some specific serialization, and use it across different models. Last time I wrote about decimals but there is also a case for TDateTime, which you may need as:

iso 8601 format yyyy-mm-dd hh:mm:ss
iso 8601 format with T instead of space yyyy-mm-ddThh:mm:ss
iso 8601 format with miliseconds yyyy-mm-dd hh:mm:ss.zzz
custom integer like format yyyymmddhhmmss
...and so on

When you use custom TDateTime types most if not all problems are solved

dc

ab · 2016-11-29 09:22:39

So what if we add a custom serialization for simple types, like:

type
  TDateTimeIso8601WithMS = type(TDateTime);
...
  TTextWriter.RegisterCustomJSONSerializerFromTextSimpleType(TypeInfo(TDateTimeIso8601WithMS),TDateTimeIso8601WithMSReader,TDateTimeIso8601WithMSWriter);

It then may be available with record serialization.

dc · 2016-11-29 09:36:42

Good but what about classes? Will it work with classes too?

ab · 2016-11-29 10:28:05

It will work only for record serialization, this is where DTO definition do make sense.

Note also that public fields of classes won't be serialized - they need to be published.

dc · 2016-11-29 10:34:28

As for DTO it doesn't help here because you still have to write custom serialization add/or deserialization code. Problem is with unnecessary repetition of the same code for different simple types in DTO. As far as visibility of fields/properties is considered it's not a problem because I use {$RTTI EXPLICIT ...} directive for my models and put custom serialization and deserialization flow in TJSONSerializer.RegisterCustomSerializer.

dc

ab · 2016-11-29 10:54:42

So what's wrong with defining custom serialization as class methods of each class, then register it for the class?
See http://synopse.info/files/html/Synopse% … ml#TITL_52
Using inheritance of such Writer/Reader class methods, you may reduce the duplicated code, so enhance DRY.

dc · 2016-11-29 11:23:43

Yes it would partly solve the problem, but inheritance is almost never a good choice, at least not in long term. And you won't achieve full DRY principle with hundreds of models, it will be maybe a little less wet

That's why I prefer to encapsulate serialization/deserialization of simple types and this way aid serialization of models that contain them. It's may be a strategy pattern or template method, but either way it's more about composition than inheritance.

dc

ab · 2016-11-29 11:47:18

So what is your exact need?
Only adding custom serialization for simple types of class properties?

dc · 2016-11-30 07:43:05

Yes, properties and fields. I have done a prototype of this idea on higher level but I think it would be better to make it a part of Synopse. If you want I should be able to prepare a repository on github with sample application for tomorrow.

Last edited by dc (2016-11-30 07:48:41)

ab · 2016-11-30 08:11:23

Some code externally stored in github is a great idea.

dc · 2016-12-01 08:56:41

OK. Here is quick and dirty prototype: https://github.com/dominikcz/custom-serialization-idea
I've tried to make it as close to my original code as possible, so there are some parts that have nothing to do with simple types we discuss here (just ignore them). However in few places I had to remove some functions and substitute them with default Delphi counterparts (to reduce dependency at the cost of performance).

Please note that only parts of writer's code is implemented by registry. Ideally all code from WriteValue() function should be implemented like that. And last but not least, there are no readers implemented.

Tell me what do you think about it

dc · 2016-12-06 15:05:09

Hi, any thoughts about it? Anyway, I'm on a short vacations now with limited net access, so won't be able to do anything more till next Monday.

ab · 2016-12-07 08:22:29

Nice seeing some good code.
Thanks for sharing.

In fact, this is how TSQLRecord serialization was implemented, using an ever more SOLID approach, based on TSQLPropInfo class hierarchy in mORMot.pas.
And you can register your own class using TSQLPropInfoRTTI.RegisterTypeInfo.
Of course, TSQLPropInfo is not only about JSON serialization, but has other features needed by the ORM (like comparison, text or variant conversion, hash computation, RTTI attributes, corresponding DB column type...).

For record/arrays and other classes JSON serialization in SynCommons.pas, you have a mixed approach: simple types directly handled in the code, and a custom registration system.
The only missing feature seems to be custom registration of simple types.

mORMot Open Source

#1 2016-11-28 12:37:15

General serialization of custom types

#2 2016-11-28 13:03:47

Re: General serialization of custom types

#3 2016-11-28 14:01:11

Re: General serialization of custom types

#4 2016-11-28 14:08:29

Re: General serialization of custom types

#5 2016-11-29 09:09:56

Re: General serialization of custom types

#6 2016-11-29 09:22:39

Re: General serialization of custom types

#7 2016-11-29 09:36:42

Re: General serialization of custom types

#8 2016-11-29 10:28:05

Re: General serialization of custom types

#9 2016-11-29 10:34:28

Re: General serialization of custom types

#10 2016-11-29 10:54:42

Re: General serialization of custom types

#11 2016-11-29 11:23:43

Re: General serialization of custom types

#12 2016-11-29 11:47:18

Re: General serialization of custom types

#13 2016-11-30 07:43:05

Re: General serialization of custom types

#14 2016-11-30 08:11:23

Re: General serialization of custom types

#15 2016-12-01 08:56:41

Re: General serialization of custom types

#16 2016-12-06 15:05:09

Re: General serialization of custom types

#17 2016-12-07 08:22:29

Re: General serialization of custom types

Board footer