mORMot Open Source

Blob: for load/save speed and compactness. I use a blob to store my DFM compoatible encoded TValue's. Decoding and recording the DFM stream with ObjBinToText and ObjTextToBin allows easy debugging and checking of the stored values.

Off course, when using JSON encoding one could just as well use a Text memo field. Primarily, I use BLOB because of it's "unlimited" field size, not because it is binary. The binary choice is because I use my DFM encoded files in binary format.

a bit late, but coming back to TValue..
Better than variants because:
1) It's way faster
2) contains natively rtti info about the data coming with the TValue
3) Very easy to extend and re-use in many classes (like the generic collections)
4) It is user intensively in the firemonkey framewwork to transport data between controls and the rest of the applications
5) It's an important part of the livebinding model. And the more I use it, the more I like it I must say.

I have just created helper classes for reading and writing arbitrary TValues, and it supports about all except pointers , procedures, interfaces methods and unknown.
Adding support for interfaces can be done, but was no issue to me.

I can send you the sources if you want to base your json serialaizer on my "dfm" serializer. I'll be out of office next week though.

Regards - Hans

Yes, I believe this is the direction in which we will have to move. One -to-many in this case means one simulation object (=1 DB "simobject" record) has several properties (=n DB "simvalue" records). These simvalue records could then be distributed along several tables, depending on the data type. I think this will speed up loading tremenduously, and it also allows for finding the simobject ID rather quickly since for most simvalues, an index can be created on the value field.

Unfortunately, this is what we actually need. So I'll have to store the values separately...

Hi Arnaud

I am trying to use your solution, but I run into the problem that TSQLRestCLient base class (I have to use this one as TSQLRestCLientURI is derived TSQLRestCLient rather than TSQLRestCLientDB) does not have a SetUser method nor a ServiceRegister method. I am now working around it using an "Is TSQLrestCLientURI" test, but It would be nice to have something like a "TSQLRestCLientURIDirect " class, derived from TSQLRestCLientURI that encapsulates this, as aparallel to the TSQLRestCLientURIDLL class.

I have been testing with it as it is, and seems that everyting works as expected upto now.

UPDATE: You may want to consider adding the Batch* routines as virtual/abstract routines to the TSQLRestCLient class. I had to create quite a big workaround for this in my unit tests :s

Regards - Hans

While many objects share the same set of attributes, yes, the attribute list may vary from object to object. SOme objects are powerlines, some are eg cars, or something completely different like a loan or maintenance scenario.Moreover we want to have "data inheritance" which alows one object to pre-define the properties of another object. This is kind-of like a template. Changing the template would (virtually) change all (data) derived objects in the live model.

I guess not

That is kind-of what I was think about. I was thinking separatye my objects (with values blob) table into a objects base table, and to have separate values tables for each 'value kind' or 'named attribute' with fixed fields for the value.

Currently I see of my model:
Advantages are: - easier selection (indexes) - faster loading (no blob decoding) - less storage (property name not stored with each value)
Disadvantages are: - rather complex data model - requires an extra 'attributes' definition table - no easy support for "one time" named values for a single value instance of a single object

Hi Arnaud

I have been rethinking this a little and so I have a question regarding this: Did you modify sqlite3.c yourself to adapt it for mormot (static) linking? or is it an unchanged sqlite.c from another distribution?

From my point of view it would make sense to have one sqlite code base, so if you would 'wrap' your changes in #ifdef MORMOT_STATIC #endif compiler directives in sqlite3.c , it would still allow for "one" distribution of the sqlite sources, and thus also allow for the DLL projects code to be provided with the mormot project.

Yes, generics will make us (me) very happy. What would be great too is to have (limited) TValue support foor communication between server and client, and maybe even in TSQLRecord as supported property type.

Hi Arnaud

At a first glance, I would opt for a construction with an array of field names rather than a string with comma separated values.

...
  with Client.MultiFieldValues(TSQLBaby,['ID','BirthDate'],
      'Name LIKE ? AND Sex = ?',[Letters+'%',ord(Sex)])
...

I like the functionality though.

I was completely confused by the XE3(?) issue of the threadvar as described in http://synopse.info/fossil/info/155b09dc1b

now I'm all back in the clear again. The Sky looks blue again

Thanks for the fix within 10 minutes!

Hi Arnaud

From my understanding, the 'client' session on the server side (or client thread?) is dropped some time after the connection with the client is lost. From the simulation thread's view this SQLRest instance, that i obtained from the servicecontext when the simulation thred was started, 'disappears' suddenly.

On other words: I want the simulation to be able to continue even when the client has closed it's connection. Culd be that other client sessions (eg a real time measureing system, or other users) feed data in to my simulation thread as well. Therefore my conclusion is that the simulation is not "owned" (= being destroyed by) the client thread that started the simulation thread, but that it should be more like a reference counted object.

I hope get what I mean.

|Hans

Cool!

Are you still working on the other tickets involving some ASM issues that the dcc64 does not like?

Regards - Hans

Well, I don't see a DLL-Hell problem as long as you regard your application folder (the one with the binaries) as 'holy'. This means that altering or exchanging files in this area by anything else than the installation application is a no-no. (except for de developer of course).
The DLL-Hell mainly existed because every DLL was dnmped in the windows system folder and expected to be sharable between all kinds of applications.

Regarding current prices of disk space and RAM, I just dont care about disk space or some more modules.

I do care about being able to distribute my application in parts (eg plugins) and this brings me back to my bpl/dll solution.
AND, as mentioned before, it is a way to get around any compiler incompatabilities regarding the .obj file format.

If I'm right, putting the DLL in the same folder as your executable will make it find 'first', independent of other SQLITE3.DLL's that may be on the system. I use this same technique to put my BPL's in the installation folder. (my plugin system requires the usage of runtime bpl's.).
This avoids any conflicts that may occur if bpl's can be found in the PATH, provided that all required DLL's/BPL's are in the executable folder.

Therefore I would not mind mormot being linked (optionally) to a SQLITE3.DLL, as these are available in both 32bit and 64bit, and does not cause any compiler conflicts regarding the .obj format. (Which I managed to get around using the method stated above)

a potential drawback would be, in case of using the DLL that one cannot easily step into the sqlite3 sources when debugging.

Hans

For intermediate I have created this helper class to get around it. Unfortunately I had to copy GetLongStrProp and GetLongStrProp from mormot.pas into my own unit, as they were not declared in the interface section.

I strongly suggest UpdateBlobFields and RetrieveBlobfields to be added to TSQLRest as virtual (abstract?) methods so they will be available on both server and client side.

type
  TSQLRestBlobHelper=class helper for TSQLRest
  public
    function UpdateBlobFields(Value: TSQLRecord): boolean;
    function RetrieveBlobFields(Value: TSQLRecord): boolean;
  end;

...

function TSQLRestBlobHelper.RetrieveBlobFields(Value: TSQLRecord): boolean;
var BlobData: TSQLRawBlob;
    i: integer;
begin
  result := false;
  if (Self=nil) or (Value=nil) or (Value.ID<=0) then
    exit;
  result := True;
  with Value.RecordProps do
  if BlobFields<>nil then
    for i := 0 to high(BlobFields) do
    begin
      Result:=Result and RetrieveBlob(TSQLRecordClass(Value.Classtype),Value.ID,BlobFields[i].Name,BlobData);
      if Result then
        SetLongStrProp(Value,BlobFields[i].PropInfo,BlobData)
      else Exit;
    end;
  result := true;
end;

function TSQLRestBlobHelper.UpdateBlobFields(Value: TSQLRecord): boolean;
var BlobData: RawByteString;
    i: integer;
begin
  result := false;
  if (Self=nil) or (Value=nil) or (Value.ID<=0) then
    exit;
  result := True;
  with Value.RecordProps do
  if BlobFields<>nil then
    for i := 0 to high(BlobFields) do
    begin
      GetLongStrProp(Value,BlobFields[i].PropInfo,BlobData);
      Result:=Result and UpdateBlob(TSQLRecordClass(Value.Classtype),Value.ID,BlobFields[i].Name,BlobData);
      if not result then
        exit;
    end;
end;

Yay! will try!

* UPDATE *

I Tried, and all my unit tests turn up green.

Thanks!

very, very likely

  if( !db->pVtabCtx || !(pTab = db->pVtabCtx->pTab) ){

I can see why the compiler complains, but

  if( !db->pVtabCtx || !(pTab == db->pVtabCtx->pTab) ){

would not fix it because pTab has not been initialized yet...

It's very likely though, that the error is generated here because the TRUE evamulation leads to the SQLITE_MISUSE (21) error

  if( !db->pVtabCtx || !(pTab = db->pVtabCtx->pTab) ){
    sqlite3Error(db, SQLITE_MISUSE, 0);
    sqlite3_mutex_leave(db->mutex);
    return SQLITE_MISUSE_BKPT;
  }

I have created a work-around that seems to do the trick. Though I have no idea where the error code 21 (some OS error) actually originates from.

function sqlite3_check(DB: TSQLite3DB; aResult: integer): integer;
VAR ErrMessage:string;
begin
  if (DB=0) or (aResult in [SQLITE_ERROR..SQLITE_ROW-1]) then // possible error codes
  case aResult of
    SQLITE_MISUSE:
    begin
       ErrMessage:=string(sqlite3_errmsg(DB));
       if AnsiSameText(ErrMessage,'not an error') then
         aResult:=0 // HH avoid the NO ERROR exception
       else raise ESQLite3Exception.Create(ErrMessage,aResult);
    end
    else raise ESQLite3Exception.Create(DB,aResult);
  end;
  result := aResult;
end;

What's your opinion on this, can you think of a better solution?

I see.

Yes we know, here it is intentionally in order to have a Test fixture. (meaning circumstances are the same each time the test is executed). Production code will presumably not look like this.

Thx.