Pages: 1

#1 mORMot 1 » [REQUEST] tolerant JSON parsing for database backward compatibility » 2019-09-19 02:02:32

mik
Replies: 1

suppose we have :

  TMyItem = class(TCollectionItem)
  private
    fBar: String;
    fFoo: String;
  ...
  end;

  TMyCollection= class(TCollection)
   ...
  public
    function Add: TMyItem;
    ...
  end;


  TMyClass = class(TMySQLRecord)
  public
    fMC: TMyCollection;
    .....
  end;

in the database fMC field will be saved as JSON, for example:

[{"Bar":"b1", "Foo":"f1"},{"Bar":"b2", "Foo":"f2"}]

We release application to clients with this database as version 1.

Later on, we decide to extend TMyItem and add a new field:

  TMyItem = class(TCollectionItem)
  private
    fBar: String;
    fFoo: String;
    fNew: String;
  ...
  end;

fNew field is handled by the new code (version 2). Now in the database the field will be saved as:

[{"Bar":"b1", "Foo":"f1", "New":"n1"},{"Bar":"b2", "Foo":"f2", "New":"n2"}]

Now I want to distribute this database to all clients, some of them are using application version 1, and some application version 2 (and are not willing to upgrade). Unfortunately, this will not work for "version 1". Current TJSONToObject.Parse will discover the "unknown" property "New" - this will stop the parser, and the actual content of the read record will contain only the first collection item - all other will be ignored.

The requested fix is rather simple, and safe (IMHO):

in the mormot.pas unit:

procedure TSQLPropInfoRTTIObject.SetValue(Instance: TObject; Value: PUTF8Char;
  wasString: boolean);
var valid: boolean;
    tmp: TSynTempBuffer;
begin
  tmp.Init(Value); // private copy since the buffer will be modified
  try
    PropInfo^.ClassFromJSON(Instance,tmp.buf,valid, JSONTOOBJECT_TOLERANTOPTIONS);
  finally
    tmp.Done;
  end;
end;

I've added JSONTOOBJECT_TOLERANTOPTIONS parameter to the call of ClassFromJSON()

Can this fix be propagated to production code?

Maybe a better way to handle it is to find a way to define the "Options: TJSONToObjectOptions" in such a way that user can control it and pass its user-controlled value to TSQLPropInfoRTTIObject.SetValue() (possibly to some other methods) - but maybe this is an overkill...

Please let me know your thoughts.

#2 Re: mORMot 1 » TSQLRecord: is there a way to selectively store published properties? » 2016-02-27 17:21:55

mik

Thanks, ab! I will check out both approaches. My first intuitive impression, is that mpv's suggestion may be preferrable if there are only a few properties (out of many) which have to be optionally stored - the code seems to be cleaner and easier to understand with this approach. The TDocVariant method might work better in my case, where all the numerous properties are stored optionally.

#3 Re: mORMot 1 » TSQLRecord: is there a way to selectively store published properties? » 2016-02-27 00:49:42

mik

thanks, mpv!
This looks like a possible solution. Fortunately, it appears that [woDontStoreDefault] is used by default in TJSONSerializer

#4 Re: mORMot 1 » TSQLRecord: is there a way to selectively store published properties? » 2016-02-26 20:48:14

mik

ok, thanks. This part I understand perfectly - and I see how to do this with "variant" field. I was just looking for a simpler way. My object has several dozens of published properties, and saving it via TDocVariant is more pain than using standard ObjectToJSON -> TJSONSerializer.

so I really was looking to find something similar to what edwinsn suggested or some "callback" function in TSQLRecord which would ask me to "approve" or "deny" each stored property of any processed object.

I'd like also briefly explain why this scenario makes sense to me: in my app, when I instantiate the object, it is created with the "default" values for each property. The defaults are controlled by application, and user may customize them. When user recovers the "stored" object, it is essential (in some specific cases) that he only "overrides" a subset (critical) of properties of its instantiated object, i.e. the stored object only maintains and overrides part of the the properties, leaving the rest intact (i.e. according to user-specific defaults). The content of this stored object must be dynamic in my case, i.e. the "saved subset" might be different in different saved objects (records).

#5 Re: mORMot 1 » TSQLRecord: is there a way to selectively store published properties? » 2016-02-26 17:41:36

mik

Thanks for the answers!
Sorry, I might not be knowledgeable in the entire framework, but I'm trying to learn smile
I studied the DDD chapter, but so far I don't see how I can may it work with my specific request.
DDD to ORM mapping (as far as I understand) suggests to "flatten" the structure instead of serializing object to JSON and storing it in a single field, but this is exactly what I want to avoid and I'd like to maintain the aggregate (JSON) storage in a single field.
Besides, I think that mapping between DDD and ORM is static, while in my case I want specifically "dynamic" mapping, i.e. selective subset of stored properties, individual for every record.
If I totally misunderstood the suggestion - can you please give more details or point me to the right direction and perhaps recommend some sample I can study?

#6 mORMot 1 » TSQLRecord: is there a way to selectively store published properties? » 2016-02-26 03:16:07

mik
Replies: 12

Assume that I have a record defined as:

TBaby = class
  private
    fName: RawUTF8;
    fYear: Integer;
    fSex: RawUTF8;
  published
    property Name: RawUTF8 read fName write fName;
    property Year: Integer read fYear write fYear;
    property Sex: RawUTF8 read fSex write fSex;
end;

TSQLBaby = class(TSQLRecord)
  private
    fBaby: TBaby;
  published
    property Baby: TBaby read fBaby write fBaby;
end;

Normally when TSQLBaby is stored in the database, the field "Baby" will contain JSON text, for example:

{"Name":"Jim","Year":2010,"Sex":"M"}

I'd like to be able to control which of the published fields get stored in the database on TSQLBaby.Add or TSQLBaby.Update

Is there a way to store (depending on some program logic) only subset of the published properties?
I'd like, for example, to omit "sex" for certain records and have some babies stored as:

{"Name":"Ann","Year":2014}

Can you please let me know if this is possible?
Thank you.

#7 Re: mORMot 1 » incorrect JSON serialization of TStrings or TRawUTF8List? » 2015-10-14 17:20:54

mik

thanks for the prompt reply. before posting, I downloaded the latest nightly build and recompiled - but apparently I looked at the wrong record... just double-checked  - and it works correctly! sorry about the false alarm.

#8 mORMot 1 » incorrect JSON serialization of TStrings or TRawUTF8List? » 2015-10-14 16:58:20

mik
Replies: 2

I use in TSQLRecord-based class a field defined as TRawUTF8List or TStringList.
When this field is stored in the database - the resulting JSON string contains some garbage (after the correct list representation).
for example, if the list has two values "a" and "b", it may be stored as this:

["a","b"]garbage...garbage...garbage...

It looks like the trailing \0 is missing when the JSON string is formed.

This happens in the Debug mode (i.e. when allocated memory is not filled with zeros)

Pages: 1

Board footer

Powered by FluxBB