JSONToObject and TJSONSerializer customization

mpv · 2012-04-07 09:29:14

I add functionality to SQLite3Commons to allow custom implementation of read/write class to/from JSON.

Now in my code I do:

Tm3NamedCollection = class;
Tm3Val = class;
.....
initialization

SQLite3Commons.RegisterJSONRW(Tm3NamedCollection, m3NamedCollectionToJSON, m3JSONToNamedCollection);
SQLite3Commons.RegisterJSONRW(Tm3Val, m3ValToJSON, m3JSONToVal);

and then:

var
  nc: Tm3NamedCollection;
....
  res := ObjectToJSON(nc, true);

And it's read/write my type of classes via custom JSON serialization.

It's give me a possibility to use all power of your great implementation of JSON with my classes.
Is it possible for you to add this code to SQLite3Commons? How can I attach modified SQLite3Commons.pas file for review?

Last edited by mpv (2012-04-07 10:54:36)

ab · 2012-04-07 16:36:23

I've sent to you an email.

You can send some content to this email address.

mpv · 2012-04-07 17:22:16

Thanks! I sent you files by e-mail.

ab · 2012-04-12 10:09:59

Thanks to your idea and patch proposal, I've added a TJSONSerializer.RegisterCustomSerializer() method to allow JSON serialization of any class.

This is the same callback-based implementation pattern, with some diverse minor details in the implementation:
- Registration is a class procedure of TJSONSerializer;
- The callbacks are object methods, and not plain functions (for evaluated objects, it may have sense to use a context);
- Some other code choices (for better performance or integration to the existing).

But the main ideas are still there:
- You can register any class;
- The callbacks are low-level, and process outside the "{...}" JSON object layout, allowing any serialization scheme (even a class content to be serialized as a string, array or number, on purpose);
- Low overhead.

Thanks a lot for the feedback!

See http://synopse.info/fossil/info/84383103f4

ab · 2012-04-12 13:01:55

I've updated the documentation to reflect this layout.

I think it could make sense to add a similar mechanism to plain records.
By default, records are serialized as plain binary. Only a set of well identified types (like TIntegerDynArray, TRawUTF8DynArray....) are serialized as JSON.
But defining some callbacks may benefit of the whole framework, for some other record types.
I'll look into this direction.

An issue with the current implementation of records RTTI is that in order to have the RTTI, they have to contain some reference-counted type within. So it is not possible to compile even the TypeInfo(aRecord) expression for all records.
If you have any ideas...

At least, for dynamic arrays, it could make sense to handle custom serialization.
For AJAX client, having true JSON objects instead of painful binary translated to Base64 could be a big gain.

mpv · 2012-04-12 13:36:33

Cool!!! Thanks a lot!

ab · 2012-04-12 15:28:13

I've added TTextWriter.RegisterCustomJSONSerializer() method to allow JSON serialization of any dynamic array content (used by TDynArray.LoadFromJSON and TTextWriter.AddDynArrayJSON).

So dynamic arrays can now serialize valid JSON content, even for custom types.
The registration process is similar to the one you proposed for classes.

See http://synopse.info/fossil/info/36d0fcd3c7
and http://blog.synopse.info/post/2012/04/1 … ay-content blog entry (extracted from the updated documentation).

For plain records, I do not think it would be feasible, since we lack of needed RTTI here.

ab · 2012-04-13 10:10:33

I've updated the documentation, and published the corresponding part as a dedicated blog article.

See http://blog.synopse.info/post/2012/04/1 … -any-class

It shows some sample code about how to register a custom serializer, using low-level functions of mORMot to proper create or parse the JSON content.

mpv · 2012-04-28 10:52:31

@ab Please advise me how best to proceed: when I use JSONToObject and ObjectToJSON to read/write configuration files for my application (config files is not plain and it's impossible to store it in ini) it's will be good to store Enumeration property in text representation (not as integer) to be a HumanReadable. For now I define one published property prop: RawUTF8 read ReadProp write SetProp. In setProp I do conversion of RawUTF8 to FProp: enum and in ReadProp back conversion. And one public property with enum value. But count of classes increasing and I tired to do in this way It's very easy in mORMot to implement writing enum property in text format when HumanReadable=TRUE (or adding additional param like EnumIsText default false - this way is better when use JavaScript as client). Is it possible to include this feature in the main branch?
Also it will be very good for JavaScript client of mORMot.
Thanks.

ab · 2012-04-29 17:43:58

Enumeration as text within JSON serialization does indeed make sense in some context.

I'll add it ASAP.

Thanks for the feedback!

mpv · 2012-04-29 17:52:36

If you don't mind I can add it and post source to you..

mpv · 2012-05-28 09:34:59

@ab Hi!
I'm implement HumanReadable writing and reading of set and enumeration property types.

I'm use existed HumanReadable param of ObjectToJSON function - it should not break other projects that use mORMot if they use JSONToObject for reading.....

With my modification if HumanReadable=True then function ObjectToJSON write (for example)
property logLevel: TSynLogInfos read FlogLevel write SetlogLevel
as RawUTF8StringArray like
"logLevel":["sllDebug","sllTrace","sllWarning","sllError","sllCustom1","sllCustom2","sllCustom3"],

and
property logLevel: TSynLogInfo read FlogLevel write SetlogLevel
as
"logLevel": "sllDebug",

function JSONToObject read set and enum writed as integer or in new notation

I'm sent SQLite3Commons to your mail - please, review and if it's possible, include it in main branch.

I'm use this functionality to store configuration files of my application and to pass JSON to JavaScript client in debug mode.

ab · 2012-05-28 14:05:16

Thanks for the idea.

Now ObjectToJSON/JSONToObject will unserialize sets and enumerations as an array of string, if HumanReadable is set to TRUE.
See http://synopse.info/fossil/info/21220be121

Could you please test if my implementation (which should be a bit faster) is working as expected for you?

mpv · 2012-05-28 14:19:19

Thanks! All works as expected and your implementation is pretty clear

mpv · 2012-07-22 08:56:53

Hi!
We found some issue using CustomJSONSerializer. I need to define only custom writer for some class but use standard reader - it not possible in current implementation. We made fix for this problem - I send sources to your mail. Please, include it in main branch. I make corresponding ticket also for tracking.

ab · 2012-07-23 07:28:56

Should have been implemented by http://synopse.info/fossil/info/45a798d805

mpv · 2012-09-27 07:16:12

in some case I store my program configuration in JSON format. For better human understanding I add comments in config files, like this

/* 
  This is global configuration file
*/
{
   "serverPort": "888", // HTTP server port
   "serverDomainNames": "+", 
   "handleStatic": true, // handle static files. if true - set staticRoot and staticFolder
   "staticRoot": "m3", // appRoot 
   "staticFolder": /*folder for static */ "X:\\Subsystems\\Components\\Clients\\Web\\"
}

But comment is not part of JSON spec, so I write small function to remove comment from string before pass it to JSON parser.
AB, can you add this function to SynCommons ?

/// remove comments from string before pass it to JSON parser
// may be used for prepare configuration files for loading
// for example we store server configuration in file Config.json and want to put some comments in this file
// then code for loading is:
// var cfg: RawUTF8;
//  cfg := StringFromFile(ExtractFilePath(paramstr(0)) + 'Config.json');
//  removeCommentsFromJSON(PUTF8Char(cfg));
//  pLastChar := SQLite3Commons.JSONToObject(sc, PUTF8Char(cfg), configValid);
// handle 2 type of comments:
// starting from // and till end of line and /* .....  */
procedure removeCommentsFromJSON(P: PUTF8Char);
var
  nP: PUTF8Char;
begin
  if P=nil then exit;
  while P^<>#0 do begin
    case P^ of
      '"': begin // skip string
        inc(P);
        while not (P^ in [#0, '"']) do begin
          if P^ = '\' then begin  // escaped "
            inc(P); if (P^ in ['"', '\']) then inc(P);
          end else
            inc(P);
        end;
      end;
      '/': begin
         inc(P);
         case P^ of
           '/': begin // this is // comment - replace by ' '
             dec(P);
             repeat
               P^ := ' '; inc(P)
             until P^ in [#0, #10, #13];
           end;
           '*': begin // this is /* comment - replace by ' ' but keep CRLF
             dec(P); P^ := ' '; inc(P);
             repeat
               if not (P^ in [#10, #13]) then P^ := ' '; // skeep CRLF for correct line numbering (for error for example)
               inc(P);
               if (P^='*') then begin
                 nP := P + 1;
                 if (nP^<>#0) and (nP^='/') then begin
                   P^ := ' '; nP^ := ' ';
                   inc(P); inc(P);
                   break;
                 end;
               end;
             until P^ = #0;
           end;
         end;
      end;
    end;
    inc(P);
  end;
end;

Last edited by mpv (2012-09-27 08:28:54)

ab · 2012-09-27 09:37:17

Nice proposal!

I've added RemoveCommentsFromJSON() procedure and associated regression test.
See http://synopse.info/fossil/info/87c570c277

Thanks for sharing

mpv · 2012-09-27 12:24:04

Thanks!
This peace of code is funny:

if PWord(P)^=ord('*')+ord('/')shl 8 then begin
   PWord(P)^ := $2020;
   inc(P,2);
   break;
end;

I remember for future use. I see similar when you compare string for false/true, but forgot

Last edited by mpv (2012-09-27 12:24:42)

ab · 2012-09-27 15:13:56

Yes, it compiles very nicely into asm, and the CPU pipelines like it better than two individual comparisons.

mpv · 2013-10-28 20:51:30

AB, yet another proposal for simplifying serialization of Enum and Sets

if HumanReadable=True then function ObjectToJSON now write something like this:

"logLevel":["sllDebug","sllTrace","sllWarning","sllError","sllCustom1","sllCustom2","sllCustom3"]

Proposition is to serialize it to

"logLevel":["Debug","Trace","Warning","Error","Custom1","Custom2","Custom3"]

We can do it by implement simple changes:

in TJSONSerializer.WriteObject 
  for tkEnumeration instead of GetEnumNameOrd use GetEnumNameTrimed
  for tkSet - TrimLeftLowerCaseShort(AddShort(PS^)) instead of AddShort(PS^)

for JSONToObject I propose to change TEnumType.GetEnumNameValue implementation by adding condition

TEnumType.GetEnumNameValue 

  if Value^ in ['a'..'z'] then //full enum name like 'sllDebug'  
    old implementation
  else //trimmed enum name like 'Debug' 
    stat compare each NameList form end?
  end;

With such changes we GREATLY increase readability of JSON produced by mORMot and ensuring backward compatibility. Can you implement this? Or me try?

ab · 2013-10-28 22:13:38

Good idea.

See http://synopse.info/fossil/info/fbe400a8a1

mpv · 2013-10-29 20:09:27

Perfect!

mpv · 2013-11-09 21:22:27

AB, this modification http://synopse.info/fossil/info/5d975a55df ( sets including all enumerate values will be written in JSON as "*" with woHumanReadable option (and recognized as such e.g. by JSONToObject)) breaks JavaScript client logic. If server serialize Sets as array, I expect Array on client side (Browser)... May be serialize full sets to ['*'] (array with one '*' element)?

ab · 2013-11-09 22:29:14

AFAIK it already does emit ["*"] in this case.

Could you please check?

Should I make it an option?
I've added woHumanReadableFullSetsAsStar option for JSON sets serialization (default will be compatible as previous behavior).
See http://synopse.info/fossil/info/641c0ead05

mpv · 2013-11-09 22:37:24

UPS. My mistakes. I'm not actually compile code - just look on diff. Really - it produce ['*']. Sorry for mistake. And - Feature is very good!

ab · 2013-11-10 13:05:00

I just added another feature I was thinking about from a long time: woHumanReadableEnumSetAsComment option for JSON serialization (and corresponding TEnumType.GetEnumNameTrimedAll() method).
See http://synopse.info/fossil/info/64320a3a98

I've also written some dedicated regresion tests for all human readable JSON format features.

You are using a lot of JSON files for your configuration, as I discovered in your package (I'm still investigating on it and all its nice features).
So I guess this woHumanReadableEnumSetAsComment could be pretty useful.

mpv · 2013-11-10 18:31:02

Yes - feature is very good (I'll think how to use it in my project). Just one remark - comments in the JSON files is our own features. JSON itself don't support comments (unfortunately). So this features we can use just inside Syn* projects - not for produce JSON for external systems.

ab · 2013-11-10 20:51:31

Yes - but we just follow the JavaScript Comments syntax, so it could be seen as some kind of "standard".

Even if it is not JSON compliant, it is JavaScript compliant.

mpv · 2013-12-26 11:58:38

AB, I need to serialize set variable (not member of any class) in verbose form ["SetElm1", "SetElm2"] and in current realization it is impossible - TJSONSerializer.AddTypedJSON not handle woHumanReadable Options

My propose is to add overload

procedure TJSONSerializer.AddTypedJSON(aTypeInfo: pointer; var aValue; Options: TTextWriterWriteObjectOptions); 
var i: integer;
    PS: PShortString;
begin
  if not ((woFullExpand in Options) or (woHumanReadable in Options)) then
    AddTypedJSON(aTypeInfo, aValue)
  else if aTypeInfo <> nil then
    case PTypeInfo(aTypeInfo)^.Kind of
      tkEnumeration: begin
         Add('"');
         AddShort(PTypeInfo(aTypeInfo)^.EnumBaseType^.GetEnumName(aValue)^);
         Add('"');
      end;
      tkSet: begin
        Add('[');
        with PTypeInfo(aTypeInfo)^.SetEnumType^  do begin
          PS := @NameList;
          for i := MinValue to MaxValue do begin
            if GetBit(aValue,i) then begin
              Add('"');
              AddTrimLeftLowerCase(PS);
              Add('"',',');
            end;
            inc(PtrUInt(PS),ord(PS^[0])+1); // next item
          end;
        end;
        CancelLastComma;
        Add(']');
      end;
    else
      inherited AddTypedJSON(aTypeInfo, aValue);
  end else
    AddShort('null');
end;

We also can use this overload inside TJSONSerializer.WriteObject

mpv · 2013-12-26 12:00:07

Also I confused in first line of TJSONSerializer.AddTypedJSON

  if aTypeInfo=nil then

may be must be

  if aTypeInfo<>nil then

?

ab · 2013-12-29 17:11:40

Sets including all enumerate values will be written in JSON as "*" with woHumanReadable option in new new TJSONSerializer.AddTypedJSONWithOptions() method.
See http://synopse.info/fossil/info/c14222f49e

Thanks a lot for the proposal!

I've also fixed TJSONSerializer.AddTypedJSON() so that it will produce valid JSON content.
See http://synopse.info/fossil/info/d8ee23e3e9

mpv · 2014-10-28 13:10:30

AB, we found VERY dangerous issue in TTextWriter.AddNoJSONEscapeW / AddQuotedStr - see here, and provide some solution. But we wait for your POV ASAP.

ab · 2014-10-28 14:51:55

@mpv
Should be fixed now.
See http://synopse.info/fossil/info/00c06c8fba
(we found his issue potentially occurring 4 times in the SynCOMmons.pas file)

Thanks a lot for the feedback!

mpv · 2014-10-28 15:00:56

Great! Thanks. Relay - must be 4 times.

bytejuggler · 2019-09-25 13:36:23

Hi,

Yesterday I tried to use the "ObjectToJSON" function in SynCommons.pas. However I kept getting "null" as the output.

Some further searching revealed the following Stack Overflow question where someone ran into the exact same issue and conclusion:
https://stackoverflow.com/questions/488 … ormot?rq=1

I traced the code and came to the same place as the poster in SO, and it seems to me that the code is maybe not working correctly?

However, it's probably likely that I'm maybe just missing something obvious so I'm asking here first before filing a ticket.

I've written a quick test app/test suite to demonstrate the problem. It is available here:
https://github.com/ByteJuggler/JSONSeri … ithSynopse

Somewhat related post regarding JSON serialization from the synopse blog: http://blog.synopse.info/post/2012/04/1 … -any-class

Again: Am I missing something obvious or has this routine been broken somewhere along the line?

Thanks.

Walter

Edit: Never mind! I worked out what I'm missing. You basically have to also add the mORMot unit to your uses clause, to ensure that TJSONSerializer is set as the default writer instead of TTextWriter.

Last edited by bytejuggler (2019-09-25 15:19:24)

mORMot Open Source

#1 2012-04-07 09:29:14

JSONToObject and TJSONSerializer customization

#2 2012-04-07 16:36:23

Re: JSONToObject and TJSONSerializer customization

#3 2012-04-07 17:22:16

Re: JSONToObject and TJSONSerializer customization

#4 2012-04-12 10:09:59

Re: JSONToObject and TJSONSerializer customization

#5 2012-04-12 13:01:55

Re: JSONToObject and TJSONSerializer customization

#6 2012-04-12 13:36:33

Re: JSONToObject and TJSONSerializer customization

#7 2012-04-12 15:28:13

Re: JSONToObject and TJSONSerializer customization

#8 2012-04-13 10:10:33

Re: JSONToObject and TJSONSerializer customization

#9 2012-04-28 10:52:31

Re: JSONToObject and TJSONSerializer customization

#10 2012-04-29 17:43:58

Re: JSONToObject and TJSONSerializer customization

#11 2012-04-29 17:52:36

Re: JSONToObject and TJSONSerializer customization

#12 2012-05-28 09:34:59

Re: JSONToObject and TJSONSerializer customization

#13 2012-05-28 14:05:16

Re: JSONToObject and TJSONSerializer customization

#14 2012-05-28 14:19:19

Re: JSONToObject and TJSONSerializer customization

#15 2012-07-22 08:56:53

Re: JSONToObject and TJSONSerializer customization

#16 2012-07-23 07:28:56

Re: JSONToObject and TJSONSerializer customization

#17 2012-09-27 07:16:12

Re: JSONToObject and TJSONSerializer customization

#18 2012-09-27 09:37:17

Re: JSONToObject and TJSONSerializer customization

#19 2012-09-27 12:24:04

Re: JSONToObject and TJSONSerializer customization

#20 2012-09-27 15:13:56

Re: JSONToObject and TJSONSerializer customization

#21 2013-10-28 20:51:30

Re: JSONToObject and TJSONSerializer customization

#22 2013-10-28 22:13:38

Re: JSONToObject and TJSONSerializer customization

#23 2013-10-29 20:09:27

Re: JSONToObject and TJSONSerializer customization

#24 2013-11-09 21:22:27

Re: JSONToObject and TJSONSerializer customization

#25 2013-11-09 22:29:14

Re: JSONToObject and TJSONSerializer customization

#26 2013-11-09 22:37:24

Re: JSONToObject and TJSONSerializer customization

#27 2013-11-10 13:05:00

Re: JSONToObject and TJSONSerializer customization

#28 2013-11-10 18:31:02

Re: JSONToObject and TJSONSerializer customization

#29 2013-11-10 20:51:31

Re: JSONToObject and TJSONSerializer customization

#30 2013-12-26 11:58:38

Re: JSONToObject and TJSONSerializer customization

#31 2013-12-26 12:00:07

Re: JSONToObject and TJSONSerializer customization

#32 2013-12-29 17:11:40

Re: JSONToObject and TJSONSerializer customization

#33 2014-10-28 13:10:30

Re: JSONToObject and TJSONSerializer customization

#34 2014-10-28 14:51:55

Re: JSONToObject and TJSONSerializer customization

#35 2014-10-28 15:00:56

Re: JSONToObject and TJSONSerializer customization

#36 2019-09-25 13:36:23

Re: JSONToObject and TJSONSerializer customization

Board footer