soReadIgnoreUnknownFields

Atys · 2019-07-01 20:53:35

Hi all,

How can I set soReadIgnoreUnknownFields option for RecordLoadJSON or global?
I'm not using RegisterCustomJSONSerializerFromText nor do I want to register custom serializers, I just want to use the default one based on RTTI.

Regards

ab · 2019-07-02 16:57:25

Did you try to use TTextWriter.RegisterCustomJSONSerializerSetOptions ?

Atys · 2019-07-02 19:19:04

Yes, it returns with false as the record type is not yet registered.

ab · 2019-07-03 08:06:53

Try to call RecordSaveJson() first, to force RTTI registration.

Atys · 2019-07-03 09:17:01

Okay. I'm not sure anymore if mapping from/to records is popular or not.
But this (with the option ^), and the fact, that the parser didn't throw any feedback on type mismatch (record field vs. json),
just returns with false, make the things very complicated and fragile.

Should I move to classes any way?
Would that solve my problems? I mean that the parser either won't die on the first missing field or on type mismatch?
Or at least gives me a verbose feedback?

ab · 2019-07-03 12:24:04

In house, we use a LOT records JSON mapping on production projects.
But not using RTTI, since we want compile server side for Linux, using FPC (which is a much better compiler than Delphi for this target). So we define the information using text definitions - which allows field names customization, e.g. to match the JavaScript field names casing.

macfly · 2019-07-03 14:14:06

According to your other question (in the other thread), are you using records to omit certain JSON properties?

If so, I believe this is not necessary.

I use methods that allow you to pass the fields (TSQLFieldBits) I want to return.

var
  AFieldBits : TSQLFieldBits;
  AUser : TSqlRecord;
...
  //Returns all properties except Password
  AFieldBits := AUser .RecordProps.FieldBitsFromExcludingCSV('Password',soSelect); 
  AUser.GetAsDocVariant(True, AFieldBits );

One suggestion would be to implement this directly in your base TSQLRecord class.

TMyBaseSQLRecordClass = class(TSQLRecord)
...
//Rerturn all Fields by default
function GetPublicFields : TSQLFieldBits; virtual;

//Use GetPublicFields to returna a JSON Variant with these fields
function GetPublicFieldsAsDocVariant : variant;
end;

TSQLUser = class(TMyBaseSQLRecordClass)
...
//Rerturn filtered fields
function GetPublicFields : TSQLFieldBits; override;
end;

Last edited by macfly (2019-07-03 14:51:04)

ab · 2019-07-03 19:45:54

This is what a "published" field is for, by definition.

macfly · 2019-07-03 20:14:32

Yes, but for example if I set the Password field to publi it will not be persisted in the database, correct?

I need the Password field to be persisted but not included in JSON.

ab · 2019-07-04 08:02:21

For that, I would not mess with TSQLRecord and add some new methods.
Internally, the RTTI for ORM is handled not by TSQLRecord, but by dedicated classes, and the TSQLModel.
Your proposal would break the existing design, and won't allow use of TSQLRecord in several TSQLModel context, which we want to avoid.

Some remarks:

1. Never store the password in plain, but store it as a hash with salt (e.g. using PBKDF2_HMAC_SHA256 from SynCrypto.pas).

2. To refine the transmitted JSON, either:
2.1. Define DTOs (Data Transfer Objects) as records in your interface-based service, filled with the fields only needed for a given context
2.2. Specify the field names when you query the TSQLRecord from the ORM, to let the Password field be void

macfly · 2019-07-04 13:57:50

Sorry, but I could not understand how the above methods can break something.

I use this exclusively within published methods as a service and not in other places.

Yes, the password should be encrypted this was just an example.

I agree with the idea of creating a unique Record to get the customized data.

But, I see no reason to do this in this particular case as I already have the fields stored in Tsqlrecord as published properties.

All I need to do is omit certain fields in the result returned by the API and this works fine, without the need to create parallel DTOs.

function TMyServives.GetUsers: Variant;
var
   AFieldBits : TSQLFieldBits; 
begin
  AFieldBits  := ...//Get all fields that I want to return
  Result := Server.Retrieve('',TSQLUser,AFieldBits );
end;

    
function TMyServives.GetUser(const AID : TID): Variant;
var
   AFieldBits : TSQLFieldBits; 
begin
  AFieldBits  := ...//Get all fields that I want to return
  Result := Server.RetrieveDocVariant(TSQLUser,'ID =?',[AID],AFieldBits);
end;

ab · 2019-07-04 14:48:40

Of course, you can return some JSON using such a custom variant value, containing only the need fields.
But a cleaner way (in the long term) is to define DTOs, because they make every field explicit.

On the client side, you would need to parse the content, probably into a TDocVariant.
Or perhaps unserialize the variant JSON into a TSQLRecord? But then, there is little benefit of using a variant.
And what about the documentation? Returning a variant is more error-prone, since you could change its layout, with no automated documentation update, whereas a record DTO would maintain an accurate vision of the returned object (or array).

I am only sharing years of experience about defining APIs.
We sometimes use variant for internal APIs (because we know what we are doing - and even TSQLRecord are published internally), but external/public APIs always have explicit DTOs, for all the reasons DDD put forward (see the framework documentation for instance).

But of course, you can do what you prefer.

macfly · 2019-07-04 15:12:16

I understood your point.

To be honest, I've never used the framework in production to be consumed with a Delphi / FPC client.

So I do not have this concern in serializing from / to TSqlrecord on the client side.

mORMot Open Source

#1 2019-07-01 20:53:35

soReadIgnoreUnknownFields

#2 2019-07-02 16:57:25

Re: soReadIgnoreUnknownFields

#3 2019-07-02 19:19:04

Re: soReadIgnoreUnknownFields

#4 2019-07-03 08:06:53

Re: soReadIgnoreUnknownFields

#5 2019-07-03 09:17:01

Re: soReadIgnoreUnknownFields

#6 2019-07-03 12:24:04

Re: soReadIgnoreUnknownFields

#7 2019-07-03 14:14:06

Re: soReadIgnoreUnknownFields

#8 2019-07-03 19:45:54

Re: soReadIgnoreUnknownFields

#9 2019-07-03 20:14:32

Re: soReadIgnoreUnknownFields

#10 2019-07-04 08:02:21

Re: soReadIgnoreUnknownFields

#11 2019-07-04 13:57:50

Re: soReadIgnoreUnknownFields

#12 2019-07-04 14:48:40

Re: soReadIgnoreUnknownFields

#13 2019-07-04 15:12:16

Re: soReadIgnoreUnknownFields

Board footer