mORMot Open Source

Bjoern Henke

Member

Registered: 2022-11-22

Posts: 5

mORMot2: Loosing binary data somewhere in conversion

I've some binary data stored as RawByteString and convert it to MongoJson. When converting back from JSON to RawByteString the binary data gets lost.

Example code:

var
  BV,V: Variant;
  JSON: String;
  RBS : RawByteString;
begin
  TDocVariant.New(V);
  RBS:=#1#2#3;
  BsonVariantType.FromBinary(RBS,bbtGeneric,BV);
  TDocVariantData(V).AddValue('Test',BV);
  JSON:=VariantSaveMongoJson(V,modMongoStrict);
  V:=BsonDocumentToDoc(Bson(JSON));
  BV:=TDocVariantData(V).GetValueOrNull('Test');
  if not VarIsNull(BV) 
    then Result:=BsonVariantType.ToBlob(BV,RBS);
end;

BsonVariantType.ToBlob returns false because it expects another variant type.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,655

Website

Re: mORMot2: Loosing binary data somewhere in conversion

BsonDocumentToDoc() makes a "pure" TDocVariant value into V.

So the "Test" binary is transformed by TBsonElement.ToVariant into a BsonVariantType holding a betBinary RawByteString buffer.
Normally, GetValueOrNull() should call SetVariantByValue() and BV should be back to a TBsonVariant.

Try to debug to find what is wrong.

Bjoern Henke

Member

Registered: 2022-11-22

Posts: 5

Re: mORMot2: Loosing binary data somewhere in conversion

GetValueOrNull calls SetVariantByValue which then does TSynInvokeableVariantType.CopyByValue.
The variant type is DocVariantType (274) instead of BsonVariantType (275) needed by ToBlob.

I already tried GetVarData instead of GetValueOrNull but the result remains the same.

I couldn't yet find a way to convert a DocVariantType into a BsonVariantType and I'm wondering how TRestStorageMongoDB.RetrieveBlobFields accomplishes this.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,655

Website

Re: mORMot2: Loosing binary data somewhere in conversion

TRestStorageMongoDB.RetrieveBlobFields has no trouble with it, because it does not make as many conversion as you did.
Especially, it directly fills the RawBlob fields from the BsonVariant BLOB binary.
Why on earth do you make so much conversions?

It is likely that BsonDocumentToDoc() change the type of the BLOB.
As it is expected for a TDocVariant.

Bjoern Henke

Member

Registered: 2022-11-22

Posts: 5

Re: mORMot2: Loosing binary data somewhere in conversion

The code above is just an example to reproduce the problem.

I'm basically searching for a way to convert mongo JSON back to a variant. Mainly the opposite of VariantSaveMongoJson.

The best I found so far was the combination of Bson and BsonDocumentToDoc which seemed to be working except the loss of the binary data.