mORMot Open Source

Wy do I, you, we all need inheritance? - Because I, you and the rest of us am programming using object orientation!

My program will consist eg of simulation parameters. but the main application is only aware of the fact that there are simulation parameters, but not what the specifics look like. Using a plugin system I force my modules to use the simulation parameters base class, but what is defined in the inherited class is not restricted (sure there are limits). Still, I want ONE collection of all my simulation parameters, and not a separate collection for each derived class. I believe  this fits the DDD model quite well...

If I want to separate the persistance layer from my App implementation layer, then I would want to add my parameter class as a property of the TSQLRecord. The store/retrieve routine should then be smart enough to store the entire parameter class+properties, even if it is a derived one from the base storage class. This is what I hope for that the new property mapper will bring...

In this case it would work similar to a TCollection + TCollectionItem (equivalent to table and TSQLRecord) and a TPersistent derived property in the TCollectionItem. I imaginbe something like this (untested)

type
// parameter classes

  TMyParametersbase=class(TPersistent)
  publised
    SomeBaseProperty:integer ...
  end;

  TMySpecialParameters=class(TMyParametersbase)
  publised
    MySpecialProperty:string....
  end;

// parameter presistency through mORMot

  TMyParameterRecord=class(TSQLRecord)
  published
     property Params:TMyParametersBase read FParams write SetParams;
  end;

  I can imagine some trouble exists when you have to create the MyParameterRecord instance, and the actual Params class is still unknown (could be TMyParametersBase as well as TMySpecialParameters, even when property is declared as TMyParametersBase. So... a very possible solution would be something like this:

  TMyParameterRecord=class(TSQLRecord)
  protected
     procedure WriteParams(aSQLColumn:???);
     procedure ReadParams(aSQLColumn:???);
     procedure DefineSQLProperties(aSQLInterface:???);
  public // NOT published!
     property Params:TMyParametersBase read FParams write SetParams;
  end;

procedure TMyParameterRecord.DefineSQLProperties(aSQLInterface:???);
begin
  aSQLInterface.AddProperty('Params', ftBlob, ReadParams, WriteParams , Params.HasData);
end;  

In the example above my 1st idea is to store  it as an object resource in a blob stream. Otherwise, maybe I would store my custom object as a JSON stream in the blob. This would make things a bit more standardized, though creating objects from the stream will be a bit hard I guess.

I currently use a solution similar like this to store my params and calculated results in a database. My Params are all derived from TCOmponent, and registered using RegisterClasses. The NotifyComponent procedure is overriden to put the loaded params component in the correct property/field. My TCollection is a TDataset, and my TCollectionItem is the "current" record.

BUT... in a perfect world

Being able to store classes (and derived ones) in the same table would be swell. Well, from my POV we are not actually talking about a table but much more about a named collection of objects wich could be an object itself. This collection approach would produce far more cleaner code at "my side" of the ORM, as all the handling of property mapping loading and saving is handled by the ORM. In this case I can truly forget I am using an SQL DB, and just dump my collections of objects (of different derived classes) in the ORM layer. Whether the ORM requires one or ten tables to store my objects, should not be my concern.

In the latter case having a "true" ORM it would off course be quite important that "loading" an object not only restores (base) property values into the "existing" object, but it should also be able to restore the entire object, including its actual class type and it's special properties. Thus not just the base type and base properties.

That it is required to register my classes (including the derived ones) in the ORM makes perfect sense., Otherwise the ORM would have no way to generate metadata or SQL statements. AMOF the VCL and FMX use the same approach with the RegisterClasses calls. When reading a stream of objects, you never know what the next classtype will be unless you have read the class header. That's probaly why it is in the header

As stated before: How the ORM actually handles the base and derived classes in the model in order to store and retrieve them should be the task of the ORM, and be transparent to the created base and derived classes that require storing and retrieval.

Ab,

Is it possible to have a construction like this

  TMyBaseObject=class(TPersistent);
  TMySpecialObject=class(TMyBaseObject);

  TMySQLrecord=class(TSQLRecord)
    published
      property MyObject:TMyBaseObject ...
  end;

where TMyBaseObject actually is a TMySpecialObject which is restored to the same type it was when it was save when i call TMySQLRecord.FillOne ? This would also meet my demands, even though it complicates things a bit.

At the moment I dont think we need than 2G objects :\
But it's more about the fact that ID is an integer now, and might be something different in the future (Int64 like in SQLite3). So I think it would generally be a good idea to define a special type (nothing fancy) for the ID.

About GUID's : it's not really a M$ thing, it is actually known as a UUID and used on many platforms: http://en.wikipedia.org/wiki/Globally_unique_identifier though the notation with curly braces seems to be a M$ thing. I believed it to be M$ invention too until we did some research...

About the ID being an a simple integer: IMHO an ID, especialliy a generated ID should be impossible to be negative, so it should be a UINT32 or UINT64. But from what I read in the docs, this can be a bit of a problem for JSON communication.

Targeting as int32 or int64 instead of integer ensures the same type is defined independent of the platform it is used on.

If you want the ID to be interchangablke with pointers, you'd netter use NativeUInt. and again defining your

TSQLRecordID=NativeUInt

will come in handye because you can pout it in $ifdefs if it needs to be a different type (or type identifier) on a different platform.

Hans

Int the sources I have, in SQLite3Commons.pas, about position 3592 it says

    property ID: integer read GetID write fID;

FAIK this is not an int64 so it needs to be refactored anyway. And if we have to, then I suggest also creating a separate type for it. It the type is still the same as the old one, as I suggested before, it wont break your existing code, though it would be wise to change your ID related variables to the same type.

In delphi x64 compiler integer is still defined as int32. Maybe not in lazarus, I dont know. I agree Int64 is a better choice for the ID anyway. or a TGUID. (Where did I read this before )

Hans

I see. Thx for bearing wit me :>

Ab, I have found a clean way for checking if a property is of type GUID. I have tested this with XE3, but I am pretty sure this also works in the other versions... It's based on comparing the TypeData pointer rather than the typekind.

type
  TMyCustomObject=class(TPersistent)
  private
    FMyGUID: TGUID;
  published
    property MyGUID:TGUID read FMyGUID write FMyGUID;
  end;

  TSomeCustomRecord=record
    MyInteger:integer;
  end;

procedure TForm1.Button1Click(Sender: TObject);
VAR Obj:TMyCustomObject;
    ObjTypeInfo,
    GUIDTypeInfo,
    CustomRecordTypeInfo:PTypeInfo;
    PropList:PPropList;
    PropCnt:integer;
begin
  GUIDTypeInfo:=TypeInfo(TGUID);
  CustomRecordTypeInfo:=TypeInfo(TSomeCustomRecord);
  Obj:=TMyCustomObject.Create;
  try
    ObjTypeInfo:=Obj.ClassInfo;
    PropCnt:=GetPropList(ObjTypeInfo,PropList);
    if PropList[0].PropType^.TypeData=GUIDTypeInfo^.TypeData then
      MessageDlg('It''s a match for TGUID!',mtInformation,[mbOK],0)
    else MessageDlg('It''s not a match TGUID!',mtError,[mbOK],0); // ERROR if here

    if PropList[0].PropType^.TypeData=CustomRecordTypeInfo^.TypeData then
      MessageDlg('It''s a match for TSomeCustomRecord!',mtError,[mbOK],0) // ERROR if here
    else MessageDlg('It''s not a match TSomeCustomRecord!',mtInformation,[mbOK],0);

  finally
    Obj.Free;
  end;
  // these assignments to keep the optimizer from dropping the variables prematurely
  GUIDTypeInfo:=nil;
  PropCnt:=0;
  PropList:=nil;
  ObjTypeInfo:=nil;
end;

I guess using RTTI you might check on the type name, and simply use 'TGUID' as identifier. Or maybe compare pointers for the PTypeInfo of TGUID?

I have been thinking about it a little, and for tackling similar future issues maybe you could "Open up" the type-to-field mapping using eg virtual functions or by providing some means to "Register" custom type mappings:

RegisterTypeToFieldType(aTypeInfo:PTypeinfo;aFieldType:TFieldType);

This of course also implies that there needs to be some data handling added. So why not also add pointers to data handling routines:

  TFieldToPropProc:procedure(aField:TField;aInstance:TSQLObject;aProperty:PPropInfo) of Object;
  TPropToFieldProc:procedure(aField:TField;aInstance:TSQLObject;aProperty:PPropInfo) of Object;

procedure RegisterTypeToFieldTypeMapping(aType:TTypeinfo;aFieldType:TFieldType;aReader:TFieldToPropProc;aWriter:TPropToFieldProc);

BTW it's just a suggestion, I am quite sure you can think of better prototypes then the ones i scribbled here

Thx

I love this one - great job.

That's exactly what I did - Ive added URL validation to my alias handler which ensures this problem cannot happen anymore.

Yes that's a cool way to fix it too.

But.. unfortunately my LCS server supports NamedPipe and Messaging (and later DLL mode), which would probably break my code.
It's also a nice feature to have a more or less readable alias in requests, and generally its not a bad idea to have an Alias restricted a bit like regular identifiers.
In my case, the URI part after the LCSServer/  actually is intended to be a kind of database alias. I dont mind aliases being subject to some validation checks. at all. SO I already implemented a vaslidation check allowing only valid url characters and no special characters.

http://www.blooberry.com/indexdot/html/ … coding.htm

Hans

I have managed to slaughter the beast. After googlong a lot, I found out there were same issues with the translation of the HTT_DATA_CHUNK translation because of alignment differences between MS-C++ and Delphi. Obviously, in XE3 these alignment difderences have been changed, as applying the following patch makes everything work again. Ab I guess you will take this up a bit more cleanly using the $Include file and it's defines. I have verified this is working on both XE3 and XE2 now.

Regards - Hans

{$ifdef VER240} // XE3
  HTTP_DATA_CHUNK = record
    case DataChunkType: THttpChunkType of
    hctFromMemory: (
    FromMemory: record
      pBuffer: pointer;
      BufferLength: ULONG;
      Reserved1: ULONG;
      Reserved2: ULONG;
      Reserved3: ULONG;
    end; );
    hctFromFileHandle: (
    FromFileHandle: record
      ByteRange: HTTP_BYTE_RANGE;
      FileHandle: THandle;
    end; );
    hctFromFragmentCache: (
    FromFragmentCache: record
      FragmentNameLength: word;      // in bytes not including the #0
      pFragmentName: PWideChar;
    end; );
  end;
{$else} // not XE3
  HTTP_DATA_CHUNK = record
    case DataChunkType: THttpChunkType of
    hctFromMemory: (
    FromMemory: record
      Reserved1: ULONG;
      pBuffer: pointer;
      BufferLength: ULONG;
      Reserved2: ULONG;
      Reserved3: ULONG;
    end; );
    hctFromFileHandle: (
    FromFileHandle: record
      ByteRange: HTTP_BYTE_RANGE;
      FileHandle: THandle;
    end; );
    hctFromFragmentCache: (
    FromFragmentCache: record
      FragmentNameLength: word;      // in bytes not including the #0
      pFragmentName: PWideChar;
    end; );
  end;
{$endif}
  PHTTP_DATA_CHUNK = ^HTTP_DATA_CHUNK;

yes in 64bit mode it surely breaks. But for codereadability Id's trongly suggest using <>nil or even better length() when checking (dynamic) arrays.

Back to the XE3 problem
I cannot find (significant) difference between the Resp^ structures in XE2 and XE3. Also the FRequestID and Req^.RequestID seem to be fine.

Which leaves me frowning ... WTF is missing here... Do you know what to look for (apart from the obvious) of SendHTTPResponse parameters being reported as Invaid param,eters (error 87)

Great!

I fear some regression tests regarding HTTP Fail:

   Synopse mORMot Framework Automated tests
  ------------------------------------------


1. Synopse libraries

 1.1. Low level common:
  - System copy record: 22 assertions passed  161us
  - TDynArray: 519,427 assertions passed  151.46ms
  - TDynArrayHashed: 1,200,629 assertions passed  156.41ms
  - Fast string compare: 7 assertions passed  113us
  - IdemPropName: 10 assertions passed  104us
  - Url encoding: 105 assertions passed  1.00ms
  - IsMatch: 599 assertions passed  200us
  - Soundex: 35 assertions passed  89us
  - Numerical conversions: 785,498 assertions passed  133.56ms
  - Curr64: 20,039 assertions passed  1.46ms
  - CamelCase: 5 assertions passed  78us
  - Bits: 4,614 assertions passed  104us
  - Ini files: 7,004 assertions passed  23.94ms
  - Unicode - Utf8: 61,082 assertions passed  1.29s
  - Iso8601 date and time: 24,000 assertions passed  4.70ms
  - Url decoding: 1,100 assertions passed  278us
  - TSynTable: 873 assertions passed  4.87ms
  - TSynCache: 404 assertions passed  1.00ms
  - TSynFilter: 1,005 assertions passed  2.73ms
  - TSynValidate: 677 assertions passed  758us
  - TSynLogFile: 42 assertions passed  646us
  Total failed: 0 / 2,627,177  - Low level common PASSED  1.78s

 1.2. Low level types:
  - RTTI: 34 assertions passed  437us
  - Url encoding: 200 assertions passed  662us
  - Encode decode JSON: 250,233 assertions passed  167.71ms
  Total failed: 0 / 250,467  - Low level types PASSED  171.83ms

 1.3. Big table:
  - TSynBigTable: 19,232 assertions passed  67.76ms
  - TSynBigTableString: 16,054 assertions passed  31.64ms
  - TSynBigTableMetaData: 384,060 assertions passed  1.59s
  - TSynBigTableRecord: 452,185 assertions passed  4.02s
  Total failed: 0 / 871,531  - Big table PASSED  5.71s

 1.4. Cryptographic routines:
  - Adler32: 1 assertion passed  351us
  - MD5: 1 assertion passed  352us
  - SHA1: 5 assertions passed  353us
  - SHA256: 5 assertions passed  349us
  - AES256: 6,372 assertions passed  72.21ms
  - Base64: 11,994 assertions passed  119.05ms
  Total failed: 0 / 18,378  - Cryptographic routines PASSED  194.34ms

 1.5. Compression:
  - In memory compression: 12 assertions passed  221.80ms
  - Gzip format: 19 assertions passed  408.56ms
  - Zip format: 36 assertions passed  757.35ms
  - SynLZO: 3,006 assertions passed  84.67ms
  - SynLZ: 13,016 assertions passed  282.07ms
  Total failed: 0 / 16,089  - Compression PASSED  1.75s

 1.6. Synopse PDF:
  - TPdfDocument: 4 assertions passed  6.42ms
  - TPdfDocumentGDI: 6 assertions passed  14.27ms
  Total failed: 0 / 10  - Synopse PDF PASSED  21.50ms


2. mORMot

 2.1. Basic classes:
  - TSQLRecord: 47 assertions passed  356us
  - TSQLRecordSigned: 200 assertions passed  4.90ms
  - TSQLModel: 3 assertions passed  380us
  Total failed: 0 / 250  - Basic classes PASSED  6.41ms

 2.2. File based:
  - Database direct access: 10,138 assertions passed  220.81ms
  - Virtual table direct access: 12 assertions passed  3.17ms
  - TSQLTableJSON: 19,030 assertions passed  47.97ms
  - TSQLRestClientDB: 599,030 assertions passed  3.51s
  Total failed: 0 / 628,210  - File based PASSED  3.78s

 2.3. File based WAL:
  - Database direct access: 10,138 assertions passed  224.36ms
  - Virtual table direct access: 12 assertions passed  1.42ms
  - TSQLTableJSON: 19,030 assertions passed  43.01ms
  - TSQLRestClientDB: 599,030 assertions passed  3.44s
  Total failed: 0 / 628,210  - File based WAL PASSED  3.71s

 2.4. Memory based:
  - Database direct access: 10,136 assertions passed  197.53ms
  - Virtual table direct access: 12 assertions passed  1.30ms
  - TSQLTableJSON: 19,030 assertions passed  37.69ms
  - TSQLRestClientDB: 667,323 assertions passed  4.05s
  Total failed: 0 / 696,501  - Memory based PASSED  4.29s

 2.5. Client server access:
  - TSQLite3HttpServer: 21 assertions passed  8.01ms
     using THttpApiServer
!  - TSQLite3HttpClient: 1 / 1 FAILED  26.68ms
!  - Http client keep alive: 48 / 84 FAILED  23.16ms
     first in 22.18ms,
!  - Http client multi connect: 48 / 84 FAILED  14.61ms
     first in 13.65ms,
  - Named pipe access: 3,085 assertions passed  600.30ms
     first in 60.84ms, done in 139.89ms i.e. 7148/s, aver. 139us, 33.4 MB/s
  - Local window messages: 3,084 assertions passed  65.89ms
     first in 1.34ms, done in 61.64ms i.e. 16221/s, aver. 61us, 75.8 MB/s
  - Direct in process access: 3,052 assertions passed  59.11ms
     first in 816us, done in 58.13ms i.e. 17201/s, aver. 58us, 80.4 MB/s
  Total failed: 97 / 9,411  - Client server access FAILED  802.38ms

 2.6. Service oriented architecture:
  - Weak interfaces: 56 assertions passed  382us
  - Service initialization: 127 assertions passed  2.21ms
  - Direct call: 596,163 assertions passed  38.47ms
  - Server side: 596,173 assertions passed  38.40ms
  - Client side REST: 596,175 assertions passed  546.46ms
  - Client side JSONRPC: 596,173 assertions passed  621.30ms
  - Client side synchronized REST: 596,173 assertions passed  1.27s
  - Security: 135 assertions passed  1.11ms
  - Custom record layout: 596,173 assertions passed  567.17ms
  Total failed: 0 / 3,577,348  - Service oriented architecture PASSED  3.09s

 2.7. External database:
  - External records: 1 assertion passed  612us
  - Auto adapt SQL: 168 assertions passed  10.60ms
  - Crypted database: 253,272 assertions passed  265.75ms
  - External via REST: 243,436 assertions passed  908.07ms
  - External via virtual table: 243,436 assertions passed  1.65s
  Total failed: 0 / 740,313  - External database PASSED  2.84s


Synopse framework used: 1.17
SQlite3 engine used: 3.7.12.1
Generated with: Delphi XE3 compiler

Time elapsed for all tests: 28.19s
Tests performed at 4-10-2012 16:12:32

Total assertions failed for all test suits:  97 / 10,063,895

! Some tests FAILED: please correct the code.
Done - Press ENTER to Exit

I'd be more than happy to fix it for you in XE3. What things should I consider looking for..?

BTW I can confirm no errors are reported in XE2

Hans

Just rethinking this: shouldnt there be (or is it already there) some kind of timeout on the message send/wait in the client side? Now my client "freezes", and no error is returned unti I shutdown my server.
I wouldnt want "al" my clients to freeze without an error message if my server hangs...

I confirm this fixed.
My server app is a console app, waiting for a (console) readln to finish. This made the client app wait endlessly for a response.
I had to add a "MessageDlg" call to my server main routine, obvioulsy to ensure there is a message handling loop. Now it works fine.
I must say I actually expected the TSQLRestServer to implement the message loop...? - so: Is mormot missing a message handling loop, or is it WAD and did I miss something?

SO no priority here
Maybe we could define a standard "interface query" interface which allows querying the # of interfaces, an enumerator returning the hashes of the available interfaces, and next an interface quary wich returns the interface name and  GUID, as well as (named) methods and properties. None of this would have to be SOAP, but could be provided through the regular JSON/Mormot interfaces.