JSON serialization array of array of double

robs · 2021-01-05 22:47:13

How can I get a record with JSON data in the structure of "array of array of double" or in JSON "[[number, number]]" ?
Every try with RegisterCustomJSONSerializerFromText in Delphi 2007 (best Delphi ever;-) fails.
Is it possible without using TDocVariantData?

Sample JSON data:
{
	"type": "LineString",
	"coordinates": [[13.3568, 52.6183], [13.4167, 52.5581], [13.3467, 52.5412], [13.3192, 52.5302]]
}

type 
TGeometry = packed record
        coordinates: array of array of double; // better for me: array of TPoint2D (=record lng,lat:double end;)
	AType: RawUTF8;
end;

here my negativ tested definitions:
const 
__TGeometry ='coordinates array of array of double AType RawUTF8';
__TGeometry ='coordinates [[double]] AType RawUTF8';
__TGeometry ='coordinates array of [double] AType RawUTF8';
__TGeometry ='coordinates array of array[0..1] of double AType RawUTF8';

TTextWriter.RegisterCustomJSONSerializerFromText(TypeInfo(TGeometry), __TGeometry ).
    Options := [soReadIgnoreUnknownFields, soWriteHumanReadable]; // => exception

macfly · 2021-01-05 23:51:06

One option is to create a record to store coordinates.

type

TCoordinate = packed record
  d1: Double;
  d2: Double;
end;

TGeometry = packed record
  coordinates: array of TCoordinate;
  AType: RawUTF8;
end;

const
  __TCoordinate = 'd1 double d2 double';

  __TGeometry = 'coordinates array of TCoordinate AType RawUTF8';

Register both:

  TTextWriter.RegisterCustomJSONSerializerFromText([
    TypeInfo(TCoordinate), __TCoordinate,
    TypeInfo(TGeometry), __TGeometry
  ]);

JSON produced:

{"coordinates":[{"d1":13.3568,"d2":52.6183},{"d1":0,"d2":52.5581}],"AType":"test"}'

ab · 2021-01-06 13:57:13

Did you try

__TGeometry ='coordinates array of TDoubleDynArray AType RawUTF8';

robs · 2021-01-06 19:04:43

This fails with "Unregistered ptCustom for TJSONRecordTextDefinition.AddItem(: TDOUBLEDYNARRAY).
@macfly: I want to read the json data format [[13.3568, 52.6183], [... with RegisterCustomJSONSerializerFromText

I'm afraid I have to do it with TDocVariantData.

macfly · 2021-01-06 20:05:16

If you cannot modify the json structure, then you will have another problem with the "type" property, which is a reserved word.

Vitaly · 2021-01-06 21:44:41

macfly wrote:

If you cannot modify the json structure, then you will have another problem with the "type" property, which is a reserved word.

Excuse me if I misunderstood smth. Wouldn't &type work then?

macfly · 2021-01-06 22:33:49

Vitaly wrote:

macfly wrote:
If you cannot modify the json structure, then you will have another problem with the "type" property, which is a reserved word.
Excuse me if I misunderstood smth. Wouldn't &type work then?

It does not work for me. How does the text definition?
'type RawUtf8' shouldn't work?

Vitaly · 2021-01-06 23:48:22

I meant that Delphi reserved word "type" can be declared as "&type" property in the record

TGeometry = packed record
        coordinates: ...;
	&type: RawUTF8;
end;

I was just trying to say that reserved word shouldn't be a problem. Probably I just misunderstood something about it, don't mind.

Anyway, it seems that @robs is working with GeoJSON. I was interested in that because I'm working with it on my actual project.

ab · 2021-01-07 07:54:06

You can name it as Typ or Type_ in pascal, but name is as type in the JSON text definition.

macfly · 2021-01-07 11:34:53

Vitaly wrote:

...
I was just trying to say that reserved word shouldn't be a problem. Probably I just misunderstood something about it, don't mind.

My bad. I forgot to change "AType" in text definition, and thought that mORMot did not support reserved words in decoding.
I tested it now and it works perfectly.

Regarding the original question of the post, it seems that there is really no way to do this, since the parser only accepts "array of" simple or record types.

So "array of array of" will not work.
And "array of TDoubleDynArray" will not be accepted.

macfly · 2021-01-07 11:54:12

About of the subject of compatibility and reserved words.

Is there an option to auto escape the name of tables and columns in queries?

For example TSQLOrder will generate "Select ... From ORDER ...."
And will not be accepted without escape in table name.

ab · 2021-01-07 14:16:11

See the doc about external DB mapping.

macfly · 2021-01-07 16:16:43

I will test this.

As the database is SQLite I need to check if using as external will not modify any existing behavior.

Vitaly · 2021-01-07 18:32:48

macfly wrote:

Regarding the original question of the post, it seems that there is really no way to do this, since the parser only accepts "array of" simple or record types.

Well, something always can be done it depends on the needs. For example, such a workaround can be considered

Records definitions:

  TPoint2D = packed record
    lng, lat: double;
    needed_for_rtti: RawUTF8;
  end;

  TGeometry = packed record
    coordinates: array of TPoint2D;
    &type: RawUTF8;
  end;

Custom serialization:

TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TPoint2D), TGeometryCustom.TPoint2DReader,
  TGeometryCustom.TPoint2DWriter);

Custom writer:

class procedure TGeometryCustom.TPoint2DWriter(const aWriter: TTextWriter; const aValue);
var
  LCoord: TPoint2D absolute aValue;
begin
  aWriter.Add('[% , %]', [LCoord.lng, LCoord.lat], twJSONEscape);
end;

Custom reader:

class function TGeometryCustom.TPoint2DReader(P: PUTF8Char; var aValue; out aValid: Boolean;
  CustomVariantOptions: PDocVariantOptions): PUTF8Char;
var
  LCoord: TPoint2D absolute aValue;
  LArray: TDoubleDynArray;
begin
  aValid := false;
  result := nil;
  if (P = nil) or (P^ <> '[') then
    exit;
  inc(P);
  { LCoord.lng := GetNextItemDouble(P);
    LCoord.lat := GetNextItemDouble(P);  - doesn't work, don't know why}
  DynArrayLoadJSON(LArray, FormatUTF8('[%]', [GetNextItem(P, ']')]), TypeInfo(TDoubleDynArray));
  if Length(LArray) > 1 then
  begin
    LCoord.lng := LArray[0];
    LCoord.lat := LArray[1];
  end;
  Finalize(LArray);
  if P = nil then
    exit;
  aValid := true;
  result := P; // ',' or ']' for last item of array
end;

GetNextItemDouble didn't work as I expected for the second value - probably I misunderstood it. So I used an additional dynarray.
I haven't got deep in all that records custom parsing stuff (prefer objects) - my almost first quick experience, but at least it seems working Hope it'll help @robs

ab · 2021-01-07 18:37:23

You don't need to put a RawUTF8 within the record.

Don't register the record, but the dynamic array of record.
Then you can write TypeInfo(TPoint2DDynArray).

TPoint2D = packed record
    lng, lat: double;
  end;
TPoint2DDynArray = array of TPoint2D;

  TGeometry = packed record
    coordinates: TPoint2DDynArray;
    &type: RawUTF8;
  end;

TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TPoint2DDynArray), TGeometryCustom.TPoint2DReader,
  TGeometryCustom.TPoint2DWriter);

And you don't need custom serialization, you can use plain text serialization with the dynamic array.

Vitaly · 2021-01-07 18:52:57

ab wrote:

You don't need to put a RawUTF8 within the record.

Agree, it left there from another earlier try. needed_for_rtti can be easily removed.

ab wrote:

Don't register the record, but the dynamic array of record.
Then you can write TypeInfo(TPoint2DDynArray).

Yeah, also thought so, but somehow it didn't work for me. Maybe I have mistaken somewhere. But it doesn't make much difference as far as I understand. At least custom reader/writer will be the same, I guess.

ab wrote:

And you don't need custom serialization, you can use plain text serialization with the dynamic array.

a dynamic array of records? wouldn't "lng"/"lat" names appear then in output?

Last edited by Vitaly (2021-01-07 18:54:32)

ab · 2021-01-07 19:02:22

Yes, you define the record as text and it should work with the TypeInfo() of the dynamic array.

Vitaly · 2021-01-07 19:28:06

Sorry, but I don't understand how text-based definition (if we're talking about it) can result [[13.3568, 52.6183],...] from TGeometry/TPoint2D(lng,lat:double) records.
Anyway, it makes me feel more and more stupid I guess I should leave the discussion for saving my poor mental health

macfly · 2021-01-07 19:38:14

Vitaly wrote:

Regarding the original question of the post, it seems that there is really no way to do this, since the parser only accepts "array of" simple or record types.

Yes, I thought he could create a custom serialization, but from the way he asked the question it seemed that he only wanted based on text definition.
I believe that via custom serialization is the only option.

ab wrote:

And you don't need custom serialization, you can use plain text serialization with the dynamic array.

As I understand he cannot change the structure of JSON.

Via text definition seems that is not possible to obtain the same structure that he expects.

Last edited by macfly (2021-01-07 19:39:07)

robs · 2021-01-08 23:09:29

Vitaly, your solution works fine... and makes me more and more smarter;-)

Any idea how to parse in dependence of geometry type (geojson)?
"Point" => TPoint2D ([13.4578, 52.6037])
"LineString" => array of TPoint2D (my json example)
"Polygon" => array of array of TPoint2D "[[[13.3142, 52.5014], [13.3103, 52.6055], [13.3326, 52.5558], [13.3142, 52.5014]]]"
...
In the callback function, the data must be read (or write) in differently depending on the "type". Alternatively, the coordinates could be delivered as json-string and evaluated separately.

ab wrote:

You don't need to put a RawUTF8 within the record.

Without RawUTF8 in TPoint2D my Delphi2007 compiler fails with "E2134 Typ besitzt keine Typinformation" (missing TypeInfo)

&type works fine. JSON read and write looks fine.

  
LCoord.lng := GetNextItemDouble(P);
LCoord.lat := GetNextItemDouble(P, ']');  // this works.

Thanks to all!

Vitaly · 2021-01-09 03:00:56

robs wrote:

Vitaly, your solution works fine... and makes me more and more smarter;-)

Glad to hear that

robs wrote:

In the callback function, the data must be read (or write) in differently depending on the "type". Alternatively, the coordinates could be delivered as json-string and evaluated separately.

Yes, I also came to these two paths. I haven't found the best approach yet (that's why I was particularly interested in this thread), but for the temporary solution, I decided to parse coordinates on-demand ("alternative way", as you described). The reason is that I have to work with rather big .geojson files (up to 400 Mb) and I do not need all GeoJSON objects coordinates at once. I do not need serialization in GeoJSON format, so it also makes it a bit easier.

One more thing, which might simplify type-handling. I assume you may know this already, so it is just in case. Using enumerations, like this:

  TGeoJsonType = (gjtUnknown, gjtPoint, gjtMultiPoint, gjtLineString, gjtMultiLineString, gjtPolygon, gjtMultiPolygon,
    gjtGeometryCollection, gjtFeature, gjtFeatureCollection, gjtName, gjtLink);

And, for example:

  TGeometry = packed record
    coordinates: RawJSON;
    &type: TGeoJsonType;
  public
    function GetLineString(out AResult: TPoint2DList): Boolean;
    function GetMultiPolygon(out AResult: TMultiPolygon): Boolean;
    ....
  end;

This option will trim all textual enumerations "gjtLineString" -> "LineString":

TJSONSerializer.SetDefaultEnumTrim(True);

Enumerations will allow to simplify (and speedup) types checking: &type=gjtLineString instead of &type='LineString'. And "case &type of ..." might also make things easier in some cases.

robs wrote:

LCoord.lng := GetNextItemDouble(P);
LCoord.lat := GetNextItemDouble(P, ']');  // this works.

Noted for the future, thanks!

mORMot Open Source

#1 2021-01-05 22:47:13

JSON serialization array of array of double

#2 2021-01-05 23:51:06

Re: JSON serialization array of array of double

#3 2021-01-06 13:57:13

Re: JSON serialization array of array of double

#4 2021-01-06 19:04:43

Re: JSON serialization array of array of double

#5 2021-01-06 20:05:16

Re: JSON serialization array of array of double

#6 2021-01-06 21:44:41

Re: JSON serialization array of array of double

#7 2021-01-06 22:33:49

Re: JSON serialization array of array of double

#8 2021-01-06 23:48:22

Re: JSON serialization array of array of double

#9 2021-01-07 07:54:06

Re: JSON serialization array of array of double

#10 2021-01-07 11:34:53

Re: JSON serialization array of array of double

#11 2021-01-07 11:54:12

Re: JSON serialization array of array of double

#12 2021-01-07 14:16:11

Re: JSON serialization array of array of double

#13 2021-01-07 16:16:43

Re: JSON serialization array of array of double

#14 2021-01-07 18:32:48

Re: JSON serialization array of array of double

#15 2021-01-07 18:37:23

Re: JSON serialization array of array of double

#16 2021-01-07 18:52:57

Re: JSON serialization array of array of double

#17 2021-01-07 19:02:22

Re: JSON serialization array of array of double

#18 2021-01-07 19:28:06

Re: JSON serialization array of array of double

#19 2021-01-07 19:38:14

Re: JSON serialization array of array of double

#20 2021-01-08 23:09:29

Re: JSON serialization array of array of double

#21 2021-01-09 03:00:56

Re: JSON serialization array of array of double

Board footer