TDocVariant custom variant type

ab · 2014-02-25 13:48:28

TDocVariant implements a custom variant type which can be used to store any JSON/BSON document-based content, i.e. either:
- Name/value pairs, for object-oriented documents;
- An array of values (including nested documents), for array-oriented documents;
- Any combination of the two, by nesting TDocVariant instances.

Forum thread for blog article http://blog.synopse.info/post/2014/02/2 … riant-type

Update:
Please refer to the updated part of the documentation: http://synopse.info/files/html/Synopse% … ml#TITL_80

jbroussia · 2014-02-27 15:46:57

Looks like a great alternative to SuperObject ???

Taken from your own examples:

V1 := _Obj(['name','John','year',1972]);
V2 := _Obj(['name','John','doc',_Obj(['one',1,'two',2.5])]);
// ...
writeln('name=',V2.name,' doc.one=',V2.doc.one,' doc.two=',doc.two);
// ...
if V1.Exists('year') then
  writeln(V1.year);

Is it possible to call...

if V2.doc.Exists('one') then ...

and so on ?
Also, how do you deal with names containing a dot ? Ex:

V2 := _Obj(['name','John','doc.part1',_Obj(['one',1,'two',2.5])]);
// writeln('name=',V2.name,' doc.part1.one=',V2."doc.part1".one,' doc.two=',V2."doc.part1".two); ???
// if V2."doc.part1".Exists('one') then... ???

ab · 2014-02-27 15:59:08

jbroussia wrote:

Looks like a great alternative to SuperObject ???

With some distinctive features, like late-binding and a lower memory use and fragmentation e.g. for array of values.
In fact, we found out that "variant" is just a perfect fit for value objects, much better than any class/interface combination.

jbroussia wrote:

Is it possible to call...
if V2.doc.Exists('one') then ...

Yes!

jbroussia wrote:

Also, how do you deal with names containing a dot ?

This is not directly possible via late binding, by definition.
In this case, you can either:
- Trans-type to TDocVariantData;
- Use the _() or the Value() pseudo-method of the variant.

I mean:

V2 := _Obj(['name','John','doc.part1',_Obj(['one',1,'two',2.5])]);
 writeln('name=',V2.name,' doc.part1.one=',TDocVariantData(V2)['doc.part1'].one,' doc.two=',TDocVariantData(V2)['doc.part1'].two);
 writeln('name=',V2.name,' doc.part1.one=',TDocVariantData(V2).GetValueOrRaiseException('doc.part1').one,' doc.two=',TDocVariantData(V2).GetValueOrRaiseException('doc.part1').two);
  writeln('name=',V2.name,' doc.part1.one=',V2._('doc.part1').one,' doc.two=',V2.Value('doc.part1').two);
 if V2._('doc.part1').Exists('one') then...

jbroussia · 2014-02-27 20:10:38

Formidable !
Need to try this ASAP, may replace SO in my current personal project :-)

jbroussia · 2014-03-02 10:22:24

Coming from superobject, I'm used to do things like...

var
 a, o: ISuperObject;
 // ...
begin
 a := SA([]);
 o := SO();
 // Do some stuff with o
 a[''] := o; // Add object o to array a
 // do more stuff
end;

I know I can do it with TDocVariant by transtyping:

var
 a, o: Variant;
 // ...
begin
 a := _Arr([]);
 o := _Obj([]);
 // Do some stuff with o
 TDocVariantData(a).AddItem(o); // OK
 // do more stuff
end;

but is there a syntax similar to the one of SO ? Something like:

// a := a + o;

If not, it doesn't matter, I just that I don't want to miss some easier way to write it if it exists.

ab · 2014-03-02 18:00:58

You can use either:

a.Add(o); // Pseudo-method
TDocVariantData(a).AddItem(o); // fast access
a._ := o;  // Pseudo-property

I just added the last syntax which will work for TDocVariant arrays.
See http://synopse.info/fossil/info/63e214dd4d

jbroussia · 2014-03-02 19:17:09

Great ! Thanks, for this and everything else.

berndvf · 2014-03-03 19:46:21

Firstly, thanks this is an amazing addition!

I am trying to use the TDocVariant to store settings for a new utility I am writing. What I want to do is load the JSON from a file into the TDocVariant, then add users to the Users array and save back to the file in JSON format.

However, I am unable to write any new values to the array. Adding values to the initial value is not a problem.

Running the following:

procedure Test1();
var
  LSettings : Variant;
begin
  LSettings := _ObjFast( [ 'PhoneSettings', _Obj( [ 'IP'        , '10.0.0.xxx',
                                                    'Username'  , '123',
                                                    'Password'  , '123',
                                                    'Registrar' , '10.0.0.xxx'] ),
                           'Users', _ArrFast([ _Obj( ['DisplayName', 'Test'] ) ])
                         ] );

  TDocVariantData( LSettings.Users ).AddItem( _Obj( ['DisplayName', 'Test2' ] ) );

  Writeln(  LSettings );
end;

outputs:

{"PhoneSettings":{"IP":"10.0.0.xxx","Username":"123","Password":"123","Registrar":"10.0.0.xxx"},"Users":[{"DisplayName":"Test"}]}

I've tried this in Delphi 6 and 2010, and tried with various combinations of _Obj/_Arr and _ObjFast/_ArrFast

berndvf · 2014-03-04 07:04:04

I've managed to add items to the array by doing the following:

procedure Test3();
var
  LSettings : Variant;
  LUsers : Variant;
begin
  LSettings := _Obj( [ 'PhoneSettings', _Obj( [ 'IP'        , '10.0.0.xxx',
                                                'Username'  , '123',
                                                'Password'  , '123',
                                                'Registrar' , '10.0.0.xxx'] ),
                       'Users', _ArrFast([ _ObjFast( ['DisplayName', 'Test'] ) ])
                         ], dvoValueCopiedByReference ] );

  LUsers := LSettings.Users;
  LUsers.Add( _ObjFast( [ 'DisplayName', 'Bob' ] ) );
  LSettings := _ObjFast( [ 'PhoneSettings', LSettings.PhoneSettings, 'Users', LUsers ] );
  Writeln( LSettings );
end;

Which creates the correct JSON:

{"PhoneSettings":{"IP":"10.0.0.xxx","Username":"123","Password":"123","Registrar
":"10.0.0.xxx"},"Users":[{"DisplayName":"Test"},{"DisplayName":"Bob"}]}

Is there any way to remove an item from an the array or object?

ab · 2014-03-04 17:33:16

Your remark did make a lot of sense.

Something was broken - or at least not "natural" / KISS - in our implementation.

We have just changed how it works.
Now TDocVariant instances will return their members as varByRef, both via TDocVariantData.Value[] property or via late-binding - so it will induce better performance and proper execution of any pseudo-methods (like Add) to sub properties: as a result, we introduced the function DocVariantData(aVariant)^ which is to be used instead of TDocVariantData(aVariant) to safely access the internal structure of the TDocVariant instance, even if it was retrieved as varByRef.
See http://synopse.info/fossil/info/5da7381545

As a result, you can now write code as such:

  LSettings := _ObjFast( [ 'PhoneSettings', _Obj( [ 'IP'        , '10.0.0.xxx',
                                                    'Username'  , '123',
                                                    'Password'  , '123',
                                                    'Registrar' , '10.0.0.xxx'] ),
                           'Users', _ArrFast([ _Obj( ['DisplayName', 'Test'] ) ])
                         ] );
  DocVariantData(LSettings.Users).AddItem(_Obj( ['DisplayName', 'Test2' ] ));
  o := _Obj( ['DisplayName', 'Test3' ] );
  LSettings.Users.Add(o);

It will create the corresponding JSON:

{"PhoneSettings":{"IP":"10.0.0.xxx","Username":"123","Password":"123","Registrar":"10.0.0.xxx"},"Users":[{"DisplayName":"Test"},{"DisplayName":"Test2"},{"DisplayName":"Test3"}]}

berndvf · 2014-03-04 18:58:07

Excellent, thanks!

The only thing outstanding would be to be able to remove an item from an array or object. This would allow you to truly replace an array/TList/TCollection.

ab · 2014-03-04 20:12:49

Good idea.

I've just added TDocVariantData.Delete() methods, and corresponding Delete() pseudo methods to the TDocVariant instance.
See http://synopse.info/fossil/info/cf9bf72e43

When using late-binding, the object properties or array items are retrieved as varByRef, so you can even run the pseudo-methods on any nested member:

  V := _Json('["root",{"name":"Jim","year":1972}]');
  V.Add(3.1415);
  assert(V='["root",{"name":"Jim","year":1972},3.1415]');
  V._(1).Delete('year');          // delete a property of the nested object
  assert(V='["root",{"name":"Jim"},3.1415]');
  V.Delete(1);                    // delete an item in the main array
  assert(V='["root",3.1415]');

With late-binding, this is perfectly easy to work with!

Thanks for the feedback.

berndvf · 2014-03-05 08:24:27

Awesome, thanks for the changes. Really makes this very powerful! It is a great solution especially for settings - no more trying to save arrays to complicated INI files for me Going to change some of our REST clients to use this today.

ab · 2014-03-05 08:25:12

Any feedback is welcome.

We are looking further for what you find out.

jbroussia · 2014-03-05 15:05:17

I suppose you have to be careful when deleting an item from an object, as items in objects are not ordered (contrary to items in arrays) ?
I haven't tested yet but if this is possible...

V='["root",{"name":"Jim","year":1972},3.1415]';
V._(1).Delete(0);

...then it could result in...
V='["root",{"year":1972},3.1415]' or V='["root",{"name":"Jim"},3.1415]' ?

ab · 2014-03-05 15:53:24

Items in objects are ordered in the stored order.
I mean, if the JSON output is {"name":"Jim","year":1972} then item 0 is "name":"Jim" and item 1 is "year":1972.

Ordering follows the order in TDocVariantData Names[] and Values[] dynamic array storage.

moctes · 2014-03-21 03:04:45

Hi to all,

Sorry if this is a silly question but I was wondering what would be the preferred way to return the data from three TDatasets into one TDocVariant, something like :

///Pseudocode

initialize TDocVariant variable with somevalues ( Id, Name, etc)

Loop through TDataset1

ds.first;
while not ds.eof do 
Begin
  ¿ How to add the records to the TDocVariant ?  
I know how to retrieve the fieldname/value pair from the TDataset but I'm struggling to find the way to build the JSON contents using TDocVariant
  ds.Next;
End;

Loop through Tdataset2...
Loop through Tdataset2...

So the final result would look like:

{"Id":"34356", "Name":"John Doe", "savings":[{"accountID":"345678","accName":"Some text","accBalance":"1.00"}, {"accountID":"345679","accName":"Another text","accBalance":"9.00"}], "loans": [{"accountID":"555678","accName":"Some text","accBalance":"1.00"}, {"accountID":"555679","accName":"Another text","accBalance":"9.00"}]}

Don't know it the notation is right but I hope you get the idea.

Any hints?

ab · 2014-03-21 12:41:22

Here TDocVariant is not the best option.
You do not need to fill an in-memory document - which is what TDocVariant is for - just to create some JSON content.

You should better use TJSONWriter class.

I've just added function DataSetToJSON(Data: TDataSet): RawUTF8 in SynVirtualDataSet.pas, which is a good point to start with.
Feedback is needed, since I did only basic testing.
Speed is not very high, but not due to our TJsonWriter, but to DB.pas unit itself: the TDataSet class is indeed slow, sadly by design - this is why our SynDB.pas unit is much faster than any TDataSet-based library, even the more optimized one - like FireDAC - when you want to produce some JSON.
Our TSQLDBStatement.FetchAllToJSON() method is just blazzing fast!

See http://synopse.info/fossil/info/3c2a6e082a

moctes · 2014-03-21 15:34:29

Oh man you are awesome! I was working on something more or less like that but I wanted a shortcut ;-) I'm going to download it later and I will let you know how the things went.

I know TDataset is slow but the thing is I want to interact with some delphi made services (from third party application) which have a propietary protocol and are returning TDatasets, and I'm using mORMot to act as a bridge between these services and a Mobile application.

Thank you !

jbroussia · 2014-03-29 11:14:10

Hi ab,

I'm confused with the per-value or per-reference access to TDocVariant; here is some code I tested:

var
	o, oSeasons, oSeason: Variant;
	s: string;
	i1: Integer;
begin
	s := '{"Url": "argentina", "Seasons":[{"Name": "2011/2012", "Url": "2011-2012", "Competitions": [{"Name": "Ligue1",';
	s := s + '"Url": "ligue-1"}, {"Name": "Ligue2", "Url": "ligue-2"}]}, {"Name": "2010/2011", "Url": "2010-2011",';
	s := s + ' "Competitions": [{"Name": "Ligue1", "Url": "ligue-1"}, {"Name": "Ligue2", "Url": "ligue-2"}]}]}';
	o := _Json(s);

	oSeasons := o.Seasons;
	for i1 := 0 to oSeasons._Count - 1 do begin
		oSeason := oSeasons._(i1);
		oSeason.Name := 'CHANGED !'; // (1)
		oSeason.Extra := 'blabla'; // (2)
	end;
	
	AddToLog(o); // Will simply output the content of "o" in a TMemo.
end;

But unexpectedly (for me !) the changes made in (1) and (2) in TDocVariant "oSeason" appear in TDocVariant "o" ((See below)) !? Did I miss or misunderstand something ? Isn't oSeason supposed to be "copied-by-value" by default ?

{"Url":"argentina","Seasons":[{"Name":"CHANGED !","Url":"2011-2012","Competitions":[{"Name":"Ligue1","Url":"ligue-1"},{"Name":"Ligue2","Url":"ligue-2"}],"Extra":"blabla"},{"Name":"CHANGED !","Url":"2010-2011","Competitions":[{"Name":"Ligue1","Url":"ligue-1"},{"Name":"Ligue2","Url":"ligue-2"}],"Extra":"blabla"}]}

Last edited by jbroussia (2014-03-29 11:16:03)

ab · 2014-03-29 11:20:42

As a whole value (i.e. like o2 := o), a TDocVariant is by default copied by value.

But access to a nested property (like oSeasons := o.Seasons) will be made by-reference .
Otherwise, code will be dead slow when a TDocVariant has a a lot of nested objects.

What you can write is:

...
oSeasons := o.Seasons;
_Unique(oSeasons);
...

(oups.. wait a minute this won't work... I'll fix it)

ab · 2014-03-29 12:13:18

OK.

I've added TDocVariant.NewUnique() method, and let _Unique() and TVarDocData.InitCopy() handle varByRef variants - included corresponding regression tests.
See http://synopse.info/fossil/info/5e6a52af25

And also added _Copy() and _CopyFast() functions.
See http://synopse.info/fossil/info/d8e32f6492

Thanks for the feedback!

jbroussia · 2014-03-29 20:03:02

:-) <- that is the face I'm doing now. I do it every time I come to check your work. I'm always amazed.

BTW, I'm totally fine with the default behavior as it perfectly matches my usage.

edwinsn · 2014-04-03 12:21:58

Wow! This is amazing, any possibility of rolling it into a embedded, document-based datastore by combining your bigTable, just like mongoDb, but much lightweight like sqlite, and without the 4gb limit on a 32bit machine?

ab · 2014-04-03 20:05:35

Synopse BigTable is limited to memory virtual space in its current implementation, and is by itself a document-based datastore, with its own custom variant type for late binding.
But it won't be a good candidate for TDocVariant storage.

Here, good old SQlite3 may be great, storing the TDocVariant as JSON on the server, for instance.
You can have databases bigger than 4GB with mORMot ORM and a TDocVariant published property (as regular variant), and with amazing speed.
And still full client/server at hand if needed.

moctes · 2014-04-08 23:29:48

AB

I have been pretty busy last months hence I can't reply as soon as I wish, I just want to inform you that your DataSetToJSON function worked like a charm!

Thank you

ab · 2014-04-19 06:48:13

Happy to know this!

Some other users did ask for this feature in the mean-while...
So thanks for the proposal!

warleyalex · 2015-09-03 18:34:20

Can you confirm the pseudo Exists method does not work with array?

obj1 := _Arr(['John','Mark','Luke']);
if obj1.Exists('John') then
writeln('John found in array');
// Impossible to find "John" property in an array

ab · 2015-09-04 09:34:13

In current version, Exists pseudo method calls TDocVariantData.GetValueIndex, which indeed do not work with arrays.

mORMot Open Source

#1 2014-02-25 13:48:28

TDocVariant custom variant type

#2 2014-02-27 15:46:57

Re: TDocVariant custom variant type

#3 2014-02-27 15:59:08

Re: TDocVariant custom variant type

#4 2014-02-27 20:10:38

Re: TDocVariant custom variant type

#5 2014-03-02 10:22:24

Re: TDocVariant custom variant type

#6 2014-03-02 18:00:58

Re: TDocVariant custom variant type

#7 2014-03-02 19:17:09

Re: TDocVariant custom variant type

#8 2014-03-03 19:46:21

Re: TDocVariant custom variant type

#9 2014-03-04 07:04:04

Re: TDocVariant custom variant type

#10 2014-03-04 17:33:16

Re: TDocVariant custom variant type

#11 2014-03-04 18:58:07

Re: TDocVariant custom variant type

#12 2014-03-04 20:12:49

Re: TDocVariant custom variant type

#13 2014-03-05 08:24:27

Re: TDocVariant custom variant type

#14 2014-03-05 08:25:12

Re: TDocVariant custom variant type

#15 2014-03-05 15:05:17

Re: TDocVariant custom variant type

#16 2014-03-05 15:53:24

Re: TDocVariant custom variant type

#17 2014-03-21 03:04:45

Re: TDocVariant custom variant type

#18 2014-03-21 12:41:22

Re: TDocVariant custom variant type

#19 2014-03-21 15:34:29

Re: TDocVariant custom variant type

#20 2014-03-29 11:14:10

Re: TDocVariant custom variant type

#21 2014-03-29 11:20:42

Re: TDocVariant custom variant type

#22 2014-03-29 12:13:18

Re: TDocVariant custom variant type

#23 2014-03-29 20:03:02

Re: TDocVariant custom variant type

#24 2014-04-03 12:21:58

Re: TDocVariant custom variant type

#25 2014-04-03 20:05:35

Re: TDocVariant custom variant type

#26 2014-04-08 23:29:48

Re: TDocVariant custom variant type

#27 2014-04-19 06:48:13

Re: TDocVariant custom variant type

#28 2015-09-03 18:34:20

Re: TDocVariant custom variant type

#29 2015-09-04 09:34:13

Re: TDocVariant custom variant type

Board footer