mORMot Open Source

ertank

Member

Registered: 2016-03-16

Posts: 163

TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

Hello,

I am using Delphi 10.3.3. SynCommit is 1.18.5394

I have a rather complex json and corresponding record definitions. I was happily using it as it is. This is used to communicate with a physical POS device. Recently, other part has changed something in *some* of their devices. They actually removed some of their not used json pairs. This new version of theirs is going to spread all devices in the end. Unfortunately, this process is slow. So, I cannot just convert my records to match new json string or old version running devices will fail.

I decided to use TTextWriter.RegisterCustomJSONSerializerFromText() and put in just the necessary values in both json strings and be on my way. Unfortunately, I got below exception;

Unregistered ptCustom for TJSONRecordTextDefinition.AddItem(ItemPrice: UINT64)

Until I came to this one, I did already converted some Int16, Int32 to their corresponding variable types in my constant string. It is just Int64 seems not have any alias at all.

1- Is it possible to include it in so that above exception does not raise?
2- Is there any workaround that I can use?

Thanks & regards,
Ertan

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,238

Website

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

Did you try with QWord?

The list of built-in types are https://synopse.info/files/html/api-1.1 … ERRTTITYPE

ertank

Member

Registered: 2016-03-16

Posts: 163

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

That QWord works for UInt64. Thank you.

I have one thing I would like to understand though.

I have below sub-record. If I move that record definition up in my text definition, that gets me memory corruption (access violation) after de-serialization before exiting procedure and corruption goes away if put it bottom most (last) in my text definition.

  TStLoyaltyServiceInfo = packed record
    name: Array of Byte;
    CustomerId: string;
    ServiceId: UInt16;
    u16AppId: UInt16;
    CustomerIdType: UInt16;
    TotalDiscountAmount: UInt32;
  end;

For above record I have following text definition:

stLoyaltyService[name Array of Byte CustomerId string ServiceId Word u16AppId Word CustomerIdType Word TotalDiscountAmount Cardinal]

For the record, stLoyaltyService is NULL in my de-serialized json string. So, basically it should not have any items at all. But, I see some items are created after de-serialization in Local variable view *if* it is in the middle but not at the end in my text definition.

It is possible I am missing something here?

Thanks & regards,
Ertan

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,238

Website

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

Note that stLoyaltyService[...] defines a nested ARRAY [..] of records.
You should rather write stLoyaltyService{...} to define a nested record.

But what I usually do is more something like the following:

  TTextWriter.RegisterCustomJSONSerializerFromText([
    TypeInfo(TStLoyaltyServiceInfo), 'name:TByteDynArray CustomerId:string ServiceId,u16AppId,CustomerIdType:Word TotalDiscountAmount:Cardinal',
    TypeInfo(TStLoyaltyService), 'text: string info: TStLoyaltyServiceInfo']);

So that I could reuse the TStLoyaltyServiceInfo record somewhere else, e.g. as a SOA interface-based service parameter.

ertank

Member

Registered: 2016-03-16

Posts: 163

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

Note that stLoyaltyService[...] defines a nested ARRAY [..] of records.

I realize that I did not share all necessary info. Sorry about that.

I indeed have this record as an array of records in my definition:

  TStLoyaltyServiceInfo = packed record
    name: Array[0..23] of Byte; // original was this. I changed to "Array of byte" after having access violation errors 
    CustomerId: string;
    ServiceId: UInt16;
    u16AppId: UInt16;
    CustomerIdType: UInt16;
    TotalDiscountAmount: UInt32;
  end;
  TStLoyaltyServiceInfos = Array of TStLoyaltyServiceInfo;
...
  TStTicket = packed record
...
    stLoyaltyService: TStLoyaltyServiceInfos;  // [0..MAX_LOYALITY_TRANS_NUMBER]
  end;

So, I believe my definition of "array of records" matches in my text definition. It is just changing position other than last most in the definition leads to a memory leak in my code. Maybe it is my code to blame I do not know for sure. Just trying to understand.

I also tried to use TByteDynArray and if definition is not latest, I still get access violation.

My complete text definition is here: https://pastebin.com/u27beRi9 (Where I put it at the end)
Text definition that gives access violation is here: https://pastebin.com/kACw9Y1z (Where it is defined in the middle)
And if you check both above that is what I mean by "in the middle" or "at the end" or "bottom most (last)"

Formatted json string is here: https://pastebin.com/9NxWvx4v
De-Serialized (single line in text file) json string is here: https://pastebin.com/wQsqjtVQ
Above two are same except one is formatted by online website.

Complete (hopefully I did not forget anything) record definitions are here: https://pastebin.com/uyb2uaam

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,238

Website

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

Where does this "name" field comes from? I don't see it in the JSON.

ertank

Member

Registered: 2016-03-16

Posts: 163

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

In provided test json stLoyaltyService is null. However, when I put its text definition in the middle, it is not empty but filled in with A letters I suppose. There are other problems as well. For example, UserData variable is not filled in even there is a value in the json string for it.
Please see: https://imgur.com/a/lsjTnsM

Last edited by ertank (2020-02-25 12:55:48)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,238

Website

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

There are some obvious errors, like "name TByteDynArray of Byte" in your definition. I am even dubious how it could be accepted.
The first fix is to have it properly formatted, with name:type or name1,name2,name3:type pairs which would ensure that the definition is not mistaken.

The associated record definitions are missing AFAIK. So it is difficult to find out what is wrong, and what you expect.
For instance, static arrays are serialized as Base-64 binary by the framework. So I guess this is not what you wanted.

ertank

Member

Registered: 2016-03-16

Posts: 163

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

I am going to work on a small project to re-produce (if I can re-produce) and either post project or a feedback here. That is in a few days.

Thanks.

igors233

Member

Registered: 2012-09-10

Posts: 234

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

Just an idea, use TDocVariantData to load json from devices, it will load any valid json structure no matter what fields it contains.
Then you can manually convert DocVariantData to your internal records.

ertank

Member

Registered: 2016-03-16

Posts: 163

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

I could complete a small project and I observe same problem in that stand-alone project.

Project can be downloaded from here: https://fil.email/qYgVeiEo
P.S.: Download valid for 1 month only.

FYI, I am using Synopse from GitHub. However, I am using SVN client to get sources. I see my copy is from 26th January, 2020. There is note "Merge pull request #268 from LongDirtyAnimAlf/master fix compilation of Alpine Linux."

ertank

Member

Registered: 2016-03-16

Posts: 163

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

I do really would like to hear if somebody can re-produce issue I am facing.

ertank

Member

Registered: 2016-03-16

Posts: 163

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

I gave up and follow igors233 suggestion. As I do not know how to use TDocVariantData. I did read documents but could not comprehend them fast enough. So, I went down using Variant.

However, I could failed to read information I am looking for from sub objects in incoming json. For regular record filled with data I could write something like:

var
  Amount: Integer;
  AType: Integer;
begin
  if Rec.StPayment[0].typeOfPayment = 4 then
  begin
    Amount := Rec.StPayment[0].StBankPayment.Amount;
  end;

Now, I load json into a Variant named V. Try to write a code as below:

...
V := _Json(RetJsonString);
...
var
  Amount: Integer;
  AType: Integer;
begin
  if V.StPayment[0].typeOfPayment = 4 then
  begin
    Amount := V.StPayment[0].StBankPayment.Amount;
  end;

That compiles fine, but fails at the if statement at run-time.

I appreciate some examples as to how I can extract json data directly from a variant.

Thanks & regards,
Ertan

Last edited by ertank (2020-03-05 13:53:29)

ertank

Member

Registered: 2016-03-16

Posts: 163

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

Seems I need to use array access as following:

...
V := _Json(RetJsonString);
...
var
  Amount: Integer;
  AType: Integer;
begin
  if V.StPayment._(0).typeOfPayment = 4 then
  begin
    Amount := V.StPayment._(0).StBankPayment.Amount;
  end;
end;

Problem solved.

Thanks.

igors233

Member

Registered: 2012-09-10

Posts: 234

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

Variants (as loaded by _JSON) are just convenient wrappers around TDocVariantData with late/runtime binding. There are plenty of handy functions inside TDocVariantData for accessing data, best to check their descriptions for ideas and usage examples.
One way to handle is:

const
  JSON_STR = '{"FldTest":123,"TestArray":[{"Fld1":1,"Fld2":"1"},{"Fld1":2,"Fld2":"2"}]}';
var
  Temp: TDocVariantData;
  Temp2: PDocVariantData;
  arr: TVariantDynArray;
  v: Variant;
  i: Integer;
begin
  Temp.InitJSON(JSON_STR);
  WriteLn(Temp.S['FldTest']);
  arr := Temp.A['TestArray'].Values;
  for i := 0 to High(arr) do
  begin
    Temp2 := _Safe(arr[i]);
    WriteLn(Temp2.S['Fld1']);
    WriteLn(Temp2.S['Fld2']);
  end;

  v := _JsonFast(JSON_STR);
  WriteLn(v.FldTest);
  arr := _Safe(v.TestArray).Values;
  for i := 0 to High(arr) do
  begin
    WriteLn(arr[i].Fld1);
    WriteLn(arr[i].Fld2);
  end;

macfly

Member

From: Brasil

Registered: 2016-08-20

Posts: 374

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

Seems I need to use array access as following:

Your code will generate an exception if RetJsonString is not a valid JSON or is not an array, or is an empty array.

It is recommended that you use Safe and _Safe methods to access the properties.

It would also be interesting to check if the variant is in fact an array and has the size you need.

ertank

Member

Registered: 2016-03-16

Posts: 163

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

Thanks for all suggestions.

Actually, I did take very basic precautions like try..except block in my code. I did not put it all here in my earlier post.

...
try
  V := _Json(RetJsonString);
except
  // do what needs to be done
end;
...

After reading other replies, I also changed my code to retrieve data like following:

var
  Amount: Integer;
  LStPayment: TVariantDynArray;
begin
  LStPayment := _Safe(V.stPayment).Values;
  for Idx := 0 to High(LStPayment) do
  begin
    if LStPayment[Idx].typeOfPayment = 4 then
    begin
      Amount := LStPayment[Idx].StBankPayment.Amount;
    end;
  end;
end;

Which IMHO is easier to compared to my initial version. However, I am not quite sure this code above is free from exceptions if there is no StBankPayment exists in LStPayment (even though it should exists because payment type 4 means credit card payment) I am asking just in case. Because even rarely it does happen when company does an update to device software and breaks produced json in a way.

I appreciate if someone can confirm/deny.

Regards,
Ertan

Last edited by ertank (2020-03-05 19:46:37)

macfly

Member

From: Brasil

Registered: 2016-08-20

Posts: 374

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

You must use _Safe already in variable v, because you don't know if stPayment exists.

Using "with" helps not to create intermediate variables. I prefer it this way, but this is personal ...

  with _Safe(v)^ do  //<-- Return a Fake TDocVariant if V is not a valid TDocVariant
    with _Safe(stPayment)^ do //<-- if  first _Safe have returned a Fake TDocVariant or stPayment not Exists then this _Safe return another fake variant
      if Kind = dvArray then // <-- if last _Safe have returned a  Fake TDocVariant then Kind = dvUndefined. If Kind = dvArray then looping is performed.
      begin
        for Idx  := Low(Values) to High(Values) do
        begin
          ....
        end;
      end;

Last edited by macfly (2020-03-05 20:21:16)

igors233

Member

Registered: 2012-09-10

Posts: 234

Re: TTextWriter.RegisterCustomJSONSerializerFromText - UInt64 problem

> Which IMHO is easier to compared to my initial version. However, I am not quite sure this code above is free from exceptions if
> there is no StBankPayment exists in LStPayment (even though it should exists because payment type 4 means credit card payment) I
> am asking just in case. Because even rarely it does happen when company does an update to device software and breaks produced json in a way.

Yes, that would fail if there's no Amount or stBankPayment. For this type of situations you can use GetValueByPath, GetDocVariantByPath, GetItemByProp, GetVarData etc.

for example:
_Safe(LStPayment[Idx]).GetValueByPath('StBankPayment.Amount');