#1 Re: mORMot Framework » dsp TMessage errors. » 2014-02-24 09:28:26

Ha Ab, X

Combining FMX with mormot is no problem. (Out app LCS runs server and client currently with FMX and Mormot), but you do lose the multiplatform functionality.
FTM I would be helpful to split Mormot into parts (packages,units) that are platform dependent and independent(Like the parts contaning windows messaging). (Currently there is no big need for this from our side).

The compiler error here is probably caused by fmx being in the "unit scope names" list of the delphi compiler settings. Just kick "fmx" out of there, and add "Winapi" amnd "System.Win" si the correct "messages.pas" is linked instead of the fmx.messages unit. Otherwise replace the "messages" in the units list with "winapi.messages", that may help too.

Regards - Hans

#2 Re: mORMot Framework » Adding some generic-based methods to mORMot » 2013-04-26 06:38:14

Blob: for load/save speed and compactness. I use a blob to store my DFM compoatible encoded TValue's. Decoding and recording the DFM stream with ObjBinToText and ObjTextToBin allows easy debugging and checking of the stored values.

Off course, when using JSON encoding one could just as well use a Text memo field. Primarily, I use BLOB because of it's "unlimited" field size, not because it is binary. The binary choice is because I use my DFM encoded files in binary format.

#4 Re: mORMot Framework » Adding some generic-based methods to mORMot » 2013-04-25 13:13:40

a bit late, but coming back to TValue..
Better than variants because:
1) It's way faster
2) contains natively rtti info about the data coming with the TValue
3) Very easy to extend and re-use in many classes (like the generic collections)
4) It is user intensively in the firemonkey framewwork to transport data between controls and the rest of the applications
5) It's an important part of the livebinding model. And the more I use it, the more I like it I must say.

I have just created helper classes for reading and writing arbitrary TValues, and it supports about all except pointers , procedures, interfaces methods and unknown.
Adding support for interfaces can be done, but was no issue to me.

I can send you the sources if you want to base your json serialaizer on my "dfm" serializer. I'll be out of office next week though.

Regards - Hans

#5 Re: mORMot Framework » SafeLoadLibrary vs LoadLibrary » 2013-04-16 11:32:41

Ah.. but why not use SafeLoadLibrary everywhere then? - Just for the sake of simplicity.

#6 mORMot Framework » SafeLoadLibrary vs LoadLibrary » 2013-04-16 07:49:54

h.hasenack
Replies: 3

Hi Arnaud

SOmetimes you use LoadLibrary, sometimes you use SafeLoadLibrary in your code to load DLL's. Is this coincidence or is there a deeper reason for this?

Just wondering.

Hans

#7 Re: mORMot Framework » How to use sicPerUser, sicPerSession, sicPerGroup in the client? » 2013-04-15 07:02:37

Did you change the sharing type on both the server and client side? Failing to do so causes an interface contract error!

I am a little confused too about the types of sharing. Maybe Arnaud can say which one I need (my guess is perUserGroup)

1) 1st client userA connects, starts a simulation interface
2) 2nd client userB connects, starts a simulation interface
3) Connection if UserA Is dropped unexpectedly (powerloss, cable out, clean logout, whatever)
4) UserA reconnects.

Upon step 2 U want user B to use the same simulation object instance as UserA
Upon step4 I want user A to reconnect to the same simulation object instance as the one that was created in step 1

#8 Re: mORMot Framework » Data sharding doesnt work for us, what else can we imagine...? » 2013-04-10 14:32:52

If you want a many/to/many relationship with a pivot Key/Value table, you can already implement this, and define some custom methods and pseudo-properties at TSQLRecord level.
With this design, you are able to benefit of the RDBMS indexes on the pivot table.

Yes, I believe this is the direction in which we will have to move. One -to-many in this case means one simulation object (=1 DB "simobject" record) has several properties (=n DB "simvalue" records). These simvalue records could then be distributed along several tables, depending on the data type. I think this will speed up loading tremenduously, and it also allows for finding the simobject ID rather quickly since for most simvalues, an index can be created on the value field.

#9 Re: mORMot Framework » Data sharding doesnt work for us, what else can we imagine...? » 2013-04-10 06:42:03

It's nice, but still won't allow indexes (DB side) to be created for the properties as they are stored encoded in a blob field.

We currently load/import data from our old flagship app which has separate tables for the user definable property values. This loading is about 20 times faster than the loading from our 'native' data model with the values tored in a blob. (alike the request above)
Thats why I think we need a separate table for the values. whether it contains more than one value per row (in different columns) like in our old app, or one value per row is still to be decided.

#10 Re: mORMot Framework » Data sharding doesnt work for us, what else can we imagine...? » 2013-04-09 13:03:49

But this method will work only if the field layout does not change from one data row to the other.

Unfortunately, this is what we actually need. So I'll have to store the values separately...

#11 Re: mORMot Framework » TSQLRestServer.ExportServer question » 2013-04-09 11:44:04

As usual you are right. Must have been a compiler glitch ... :s

#12 Re: mORMot Framework » TSQLRestServer.ExportServer question » 2013-04-09 09:10:54

Hi Arnaud

I am trying to use your solution, but I run into the problem that TSQLRestCLient base class (I have to use this one as TSQLRestCLientURI is derived TSQLRestCLient rather than TSQLRestCLientDB) does not have a SetUser method nor a ServiceRegister method. I am now working around it using an "Is TSQLrestCLientURI" test, but It would be nice to have something like a "TSQLRestCLientURIDirect " class, derived from TSQLRestCLientURI that encapsulates this, as aparallel to the TSQLRestCLientURIDLL class.

I have been testing with it as it is, and seems that everyting works as expected upto now.

UPDATE: You may want to consider adding the Batch* routines as virtual/abstract routines to the TSQLRestCLient class. I had to create quite a big workaround for this in my unit tests :s

Regards - Hans

#13 Re: mORMot Framework » ODBC vs OLEDB » 2013-04-09 08:03:19

* UPDATE: I've created a tickt for this one: http://synopse.info/fossil/tktview/d482 … d1b209bae0

Here's an error I get for ODBC (32Bit) when adding a record:

msg '[HY090] [Microsoft][SQL Server Native Client 10.0]Invalid string or buffer length (0)'

The TSQLrecord that I'm trying to add is the following:

  TSQLUnitTestSampleRecord = class(TSQLRecord)
  private
    fQuestion: RawUTF8;
    fName: RawUTF8;
    fTime: TModTime;
    FAddress: RawUTF8;
    fPostal: RawUTF8;
    FCity: RawUTF8;
    FDatum: TDatetime;
    FFloatNumber: double;
    FIntNumber: Int64;
    FCurNumber: Currency;
    FBlobData: TSQLRawBlob;
    FRaw: RawUTF8;
  published
    property ModTime: TModTime read fTime write fTime;
    property Name: RawUTF8 index 50 read fName write fName;
    property Question: RawUTF8 index 50 read fQuestion write fQuestion;
    property Address_: RawUTF8 index 50 read FAddress write fAddress;
    property PostalCode: RawUTF8 index 12 read fPostal write fPostal;
    property City: RawUTF8 index 50 read FCity write FCity;
    property Datum: TDatetime read FDatum write FDatum;
    property FloatNumber: double read FFloatNumber write FFloatNumber;
    property IntNumber: Int64 read FIntNumber write FIntNumber;
    property CurrencyNumber: Currency read FCurNumber write FCurNumber;
    property BlobData:TSQLRawBlob read FBlobData write FBlobData;
    property BigRawUtf8: RawUTF8 read FRaw write FRaw;
  end;

The call stack when the error occurs is this:

Mormot_ODBCD17.SynDBODBC.TODBCLib.HandleError(-1,3,$CF6820,False,sllNone)
Mormot_ODBCD17.SynDBODBC.TODBCStatement.ExecutePrepared
mORMot_D17.mORMotDB.TSQLRestServerStaticExternal.ExecuteFromJSON('{"ModTime":135103816522,"Name":"Hans Hasenack 0","Question":"{CCB2E30E-9A85-4907-A2EF-E3D50DE0D908}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2013-04-09T09:59:24","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0,"BigRawUtf8":""}',soInsert,0)
mORMot_D17.mORMotDB.TSQLRestServerStaticExternal.EngineAdd(TSQLUnitTestSampleRecord,'{"ModTime":135103816522,"Name":"Hans Hasenack 0","Question":"{CCB2E30E-9A85-4907-A2EF-E3D50DE0D908}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2013-04-09T09:59:24","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0,"BigRawUtf8":""}')
mORMot_D17.mORMot.TSQLRestServer.URI($18F3A4)
mORMot_D17.mORMotSQLite3.TSQLRestClientDB.InternalURI('root/UnitTestSampleRecord?session_signature=0000004C001F348D57FB5B73','POST',$18F42C {''},$18F480 {''},$18F48C {'{"ModTime":135103816522,"Name":"Hans Hasenack 0","Question":"{CCB2E30E-9A85-4907-A2EF-E3D50DE0D908}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2013-04-09T09:59:24","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0,"BigRawUtf8":""}'})
mORMot_D17.mORMot.TSQLRestClientURI.URI('root/UnitTestSampleRecord','POST',nil {''},$18F480 {''},$18F48C {'{"ModTime":135103816522,"Name":"Hans Hasenack 0","Question":"{CCB2E30E-9A85-4907-A2EF-E3D50DE0D908}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2013-04-09T09:59:24","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0,"BigRawUtf8":""}'})
mORMot_D17.mORMot.TSQLRestClientURI.EngineAdd(TSQLUnitTestSampleRecord,'{"ModTime":135103816522,"Name":"Hans Hasenack 0","Question":"{CCB2E30E-9A85-4907-A2EF-E3D50DE0D908}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2013-04-09T09:59:24","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0,"BigRawUtf8":""}')
mORMot_D17.mORMot.TSQLRestClient.Add($FE7BD8E0,True,False)
SGMormotTestSuitePkg_D17.uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.CreateModelData(False,False)
SGMormotTestSuitePkg_D17.uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.TestTimedCRUD
DUnitTestInspectorD17.TestFramework.TTestCase.Invoke((uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.TestTimedCRUD,$FE7F0380))

In 64Bit mode I get quite a different error message:

msg '[22008] [Microsoft][ODBC SQL Server Driver]Datetime field overflow (0)'

But the call stack is about the same:

Mormot_ODBCD17.SynDBODBC.TODBCLib.HandleError(-1,3,$74F2D00,False,sllNone)
Mormot_ODBCD17.SynDBODBC.TODBCStatement.ExecutePrepared
mORMot_D17.mORMotDB.TSQLRestServerStaticExternal.ExecuteFromJSON('{"ModTime":135103816613,"Name":"Hans Hasenack 0","Question":"{D3370E96-DB9F-4A82-99FF-7F3B1150C33A}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2013-04-09T10:00:51","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0,"BigRawUtf8":""}',soInsert,0)
mORMot_D17.mORMotDB.TSQLRestServerStaticExternal.EngineAdd(TSQLRecordClass($3B83470),'{"ModTime":135103816613,"Name":"Hans Hasenack 0","Question":"{D3370E96-DB9F-4A82-99FF-7F3B1150C33A}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2013-04-09T10:00:51","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0,"BigRawUtf8":""}')
mORMot_D17.mORMot.TSQLRestServer.URI($12E5B8)
mORMot_D17.mORMotSQLite3.TSQLRestClientDB.InternalURI('root/UnitTestSampleRecord?session_signature=0000004C001F35E1CB970542','POST',$12E698 {''},$12E738 {''},$12E770 {'{"ModTime":135103816613,"Name":"Hans Hasenack 0","Question":"{D3370E96-DB9F-4A82-99FF-7F3B1150C33A}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2013-04-09T10:00:51","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0,"BigRawUtf8":""}'})
mORMot_D17.mORMot.TSQLRestClientURI.URI('root/UnitTestSampleRecord','POST',nil {''},$12E738 {''},$12E770 {'{"ModTime":135103816613,"Name":"Hans Hasenack 0","Question":"{D3370E96-DB9F-4A82-99FF-7F3B1150C33A}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2013-04-09T10:00:51","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0,"BigRawUtf8":""}'})
mORMot_D17.mORMot.TSQLRestClientURI.EngineAdd(TSQLRecordClass($3B83470),'{"ModTime":135103816613,"Name":"Hans Hasenack 0","Question":"{D3370E96-DB9F-4A82-99FF-7F3B1150C33A}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2013-04-09T10:00:51","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0,"BigRawUtf8":""}')
mORMot_D17.mORMot.TSQLRestClient.Add($2FF44D0,True,False)
SGMormotTestSuitePkg_D17.uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.CreateModelData(False,False)
SGMormotTestSuitePkg_D17.uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.TestTimedCRUD
DUnitTestInspectorD17.TestFramework.TTestCase.Invoke((uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.TestTimedCRUD,$306CA60))

#14 Re: mORMot Framework » Data sharding doesnt work for us, what else can we imagine...? » 2013-04-09 07:20:37

Do the property list of each object change from one object to the other?

While many objects share the same set of attributes, yes, the attribute list may vary from object to object. SOme objects are powerlines, some are eg cars, or something completely different like a loan or maintenance scenario.Moreover we want to have "data inheritance" which alows one object to pre-define the properties of another object. This is kind-of like a template. Changing the template would (virtually) change all (data) derived objects in the live model.

That is, if a TSQLRecord could be able to have runtime-defined fields, with a field layout fixed for all records, should it do the trick for you?

I guess not sad

That is, you may be able to publish some of the fields as true SQL columns.

That is kind-of what I was think about. I was thinking separatye my objects (with values blob) table into a objects base table, and to have separate values tables for each 'value kind' or 'named attribute' with fixed fields for the value.

Currently I see of my model:
Advantages are: - easier selection (indexes) - faster loading (no blob decoding) - less storage (property name not stored with each value)
Disadvantages are: - rather complex data model - requires an extra 'attributes' definition table - no easy support for "one time" named values for a single value instance of a single object

#15 mORMot Framework » ODBC vs OLEDB » 2013-04-09 06:27:55

h.hasenack
Replies: 2

Hi Arnaud

In http://synopse.info/fossil/info/5ca10494e0 you state that OleDB is deprecated, and that ODBC is currently the M$ recommended way. I wasn't aware of that, so in that case the news gets worse sad

I get even more errors when using ODBC in combination with MSSQL. I'll see if I can come up wit a failure list that ,may shed some light on it. The MSSQL/ODBC drivers have been installed for both 64bit and 32bit. (Running win7x64)

#16 Re: mORMot Framework » SQLite3.c compilation issue » 2013-04-09 06:22:16

Hi Arnaud

I have been rethinking this a little and so I have a question regarding this: Did you modify sqlite3.c yourself to adapt it for mormot (static) linking? or is it an unchanged sqlite.c from another distribution?

From my point of view it would make sense to have one sqlite code base, so if you would 'wrap' your changes in #ifdef MORMOT_STATIC #endif compiler directives in sqlite3.c , it would still allow for "one" distribution of the sqlite sources, and thus also allow for the DLL projects code to be provided with the mormot project.

#17 mORMot Framework » Data sharding doesnt work for us, what else can we imagine...? » 2013-04-08 14:33:58

h.hasenack
Replies: 8

Hi Arnaud (and others)

We are createing a live somulation server, which has to load a myriad of objects (5M+) into a live model before the simulation can start.

Currently we have implemented this using a data sharding strategy, where the properties of our simulation objects are stored together in a blob field with a simulation object.
The thing is, that loading this involves decoding the blob into the attributes involves time consuming decoding and saving taks a lot of time as well.

The properties of these simulation objects are fully user definable, so we are not able the 'hardcode' them into our application, instead we have a "attributes definition" table that defines each attribute.

What Idea's can you come up with that would allow swift loading of objects?
* a "Live model" consists possibly of over 5M simulation objects
* Simulation objects may have anything between 1 and around 200 properties.
* Properties have a name, type and value off course.
* Types are often simple types (like boolean, integer,float, string) but may also be complex types (objects, set's etcetera)

We have some solutions in mind, but I'm not posting them yet in order to have a real "open" question.

Your idea's are very welcome.

Hans

#18 mORMot Framework » SQLite3.c compilation issue » 2013-04-08 13:24:53

h.hasenack
Replies: 3

Hi Arnaud
in respect to http://synopse.info/fossil/info/b24c274049

I will have to move my sqlite3 DLL projects (dproj/groupproj) to a separate folder.

This is not a problem.

Thanks.

#19 Re: mORMot Framework » TSQLRestServer.ExportServer question » 2013-04-02 12:26:49

Yes, , the latter solution would be fine. This allows my tests to derive from each other (using a FRestCLient:TSQLRestClientURI field) in order to test the different 'connection modes' by just overloading the 'InitRestCLient' procedure, which currently looks something like this:

procedure TLCSServerMessageTest.InitRestClient(VAR aSQLRestClient: TSQLRestClientURI;aModel:TSQLModel);
begin
  inherited;
  aSQLRestClient:=TSQLRestClientURIMessage.Create(aModel,aModel.Root,aModel.ClassName+NewGUIDString,1000);
end;
...
procedure TLCSServerNamedPipeTest.InitRestClient(VAR aSQLRestClient: TSQLRestClientURI;aModel:TSQLModel);
begin
  inherited;
  aSQLRestClient:=TSQLRestClientURINamedPipe.Create(aModel,aModel.Root);
end;
....
procedure TLCSServerEmbeddedTest.InitRestClient(VAR aSQLRestClient: TSQLRestClientURI;aModel:TSQLModel);
begin
  inherited;
  aSQLRestClient:=TSQLRestClientURIDll.Create(aModel,@Mormot.URIRequest);
end;
...

and would maybe get to looking like this

....
procedure TLCSServerEmbeddedTest.InitRestClient(VAR aSQLRestClient: TSQLRestClientURI;aModel:TSQLModel);
begin
  inherited;
  aSQLRestClient:=TSQLRestClientURIDirect.Create(aModel,MySQLRestServerInstance);
end;
...

Which allows me to resolve the MySQLRestServerInstance any way I like...

Hans

#20 Re: mORMot Framework » Adding some generic-based methods to mORMot » 2013-04-02 12:03:28

Yes, generics will make us (me) very happy. What would be great too is to have (limited) TValue support foor communication between server and client, and maybe even in TSQLRecord as supported property type.

#21 mORMot Framework » TSQLRestServer.ExportServer question » 2013-04-02 11:43:39

h.hasenack
Replies: 6

Hi Arnaud

For the sake of debugging, and maybe for eleasing too I have been experimenting a little with the embedded (DLL) server solution using runtime packages, and it appears to work just fine.
Well, this is as long as you have only one rest server in your process, it does.

In my case, my 'root' rest server allows for the dynamic creation of 'workbase' rest servers. The problem is that these 'workbase' rest servers cannot export themselves because of the GlobalURIRequestServer variable that is alreaddy occupied by the 'root' rest server.

so... I managed to create and test a runtime, embedded link to my root server, but fail to create a runtime, embedded link to my dynamically created 'workbase' rest servers.

How to I get around this the Mormot way (Currently I'm trying to work around the problem by querying an internal list of restservers)

Maybe we need a TSQLRestCLientURIDirect with constructor Create(aModel:TSQLMode;aServerInstance:TSQLRestServer) for this?


Regards,

Hans

#22 Re: mORMot Framework » Introducing TSQLTable.Step() method » 2013-03-28 07:27:11

Hi Arnaud

At a first glance, I would opt for a construction with an array of field names rather than a string with comma separated values.

...
  with Client.MultiFieldValues(TSQLBaby,['ID','BirthDate'],
      'Name LIKE ? AND Sex = ?',[Letters+'%',ord(Sex)])
...

I like the functionality though.

#23 Re: mORMot Framework » Missing 'Admin' when reopening a SQLite3 native mormot atabase » 2013-03-27 12:37:01

Hi Arnaud

Well, the problem was that I had to omit the VirtualTableExternalRegisterAll when using the native SQLite3 engine. As we only use external databases before, it was unconditionally called.
Now that I omitted it when using sqliote itself to store data, the problem is gone and all data is stored and retrieved as expected smile

The mormot and sqlite engines appear to work very nicely in both 64bit and 32bit modes.

Regards - Hans

#24 Re: mORMot Framework » Accessing the SQLRest at server side » 2013-03-26 14:39:25

I was completely confused by the XE3(?) issue of the threadvar as described in http://synopse.info/fossil/info/155b09dc1b

now I'm all back in the clear again. The Sky looks blue again smile

Thanks for the fix within 10 minutes!

#25 Re: mORMot Framework » Accessing the SQLRest at server side » 2013-03-26 08:07:11

I have been rethinking this, and I believe I have come to a deep misunderstanding of what is going on inside the server. :s

Anyway, Sorry for bothering you with it, I'll invest some time to get a better understanding.

Hans

#26 Re: mORMot Framework » Accessing the SQLRest at server side » 2013-03-25 07:16:58

Hi Arnaud

From my understanding, the 'client' session on the server side (or client thread?) is dropped some time after the connection with the client is lost. From the simulation thread's view this SQLRest instance, that i obtained from the servicecontext when the simulation thred was started, 'disappears' suddenly.

On other words: I want the simulation to be able to continue even when the client has closed it's connection. Culd be that other client sessions (eg a real time measureing system, or other users) feed data in to my simulation thread as well. Therefore my conclusion is that the simulation is not "owned" (= being destroyed by) the client thread that started the simulation thread, but that it should be more like a reference counted object.

I hope get what I mean.

|Hans

#27 Re: mORMot Framework » Accessing the SQLRest at server side » 2013-03-22 14:39:24

Hi Arnaud

Actually the RESTServer instance at the server side might be dropped by the moromt fraework, for whatever reason.

Putting my simulation in a separate thread will make the sqlrest server unavailable through the ServiceCOntext (since its a threadvar) so, during initialization I take a pointer to the restserver because the initializeion of the simulation thread is accomplished in the restserver thread that us currently servicing a client.

But I already decided it makes more sense to make the simulation  have it's own client access to the rest server, even though it is in the server executable. The simulation actually is a bit of a rest client itself since it reads and writes data from the "main" server. This allows the simulation to continue running even when the client dropped the SQLRest connection and the TSQLRest instance is removed by the mormot service manager (or whatever you call it).

Hans

#28 mORMot Framework » Missing 'Admin' when reopening a SQLite3 native mormot atabase » 2013-03-22 13:51:14

h.hasenack
Replies: 3

Using the latest compiled SQLITE3 dll's I experience some trouble wehen reopening an existing database.
When de database is created, I can logon.
When the database is reopened (after closing/starting the server ) Then, the Admin record appers to be missing (the LogonName=? /Admin query returns no results.

A workaround is to call TSQLAuthGroup.Initializetable.

Code looks about like this:

procedure TLCSWorkbaseRestServer.FixMissingLogin;
VAR SQLUser:TSQLAuthUser;
begin
  SQLUser := TSQLAuthUser.CreateAndFillPrepare(self,'LogonName=?',['Admin']);
  try
  if not SQLUser.FillOne then
    TSQLAuthGroup.InitializeTable(self,'');
  finally
    SQLUser.Free;
  end;
end;

....

constructor TLCSWorkbaseRestServer.Create(const aAlias:UTF8String);
VAR lConnectionString:string;
    lSQLModel:TSQLModel;
    lServerName,Prot,NativeDbAlias:string;
begin
  if trim(aAlias)='' then
    raise ELCSServer.CreateFmt('%s.Create: an Alias must be proivided',[aAlias]);
  lConnectionString:=ExpandSNGMacros(AliasManager.ConnectionString[aAlias]);
  SPlitString(lConnectionString,':',Prot,NativeDBAlias);
  FPersistentConnectionProperties := LCSRestServer.CreateConnectionPropertiesForConnectionString(lConnectionString);
  if FPersistentConnectionProperties=nil then
    raise ELCSServer.CreateFmt('%s.Create: Unable to create connection properties for alias [%s] : [%s]',[ClassName,aAlias,lConnectionString]);
  FVolatileConnectionProperties := TSQLDBSQLite3ConnectionProperties.Create(cfnSQLLite3RAMFile,'','','');
  if FVolatileConnectionProperties=nil then
    raise ELCSServer.CreateFmt('%s.Create: Unable to create in-memory connection properties',[ClassName]);

  lServerName:=cDefaultLCSServerName;
  // local machine can override default server name
  lServerName:=Registry.LocalMachine.OpenAppRootKey.OpenValue(crvServerName).ReadString(lServerName);
  // current user machine can override default server name
  lServerName:=Registry.CurrentUser.OpenAppRootKey.OpenValue(crvServerName).ReadString(lServerName);
  FRootURI:=Format('%s/%s',[lServerName,Trim(aAlias)]);
  lSQLModel:=TLCSWorkbaseModel.Create(RootURI);
  lSQLModel.Owner:=self;
  RegisterModel(lSQLModel);

  if AnsiSameText(prot,cSQLiteConnectionProtocol) then
    inherited Create(lSQLModel, NativeDbAlias,True)  //*********************** Problem obly happens for this call
  else inherited Create(lSQLModel, cfnSQLLite3RAMFile,True); //******************** virtual tables appear to be not a problem,

  CreateMissingTables;
  FixMissingLogin; // *************************************Workaround call
  ServiceRegister(TLCSWorkbaseImpl,[TypeInfo(ILCSWorkbase)],sicClientDriven);
  ServiceRegister(TLCSSimulationImpl,[TypeInfo(ILCSSimulation)],sicClientDriven);
  LCSRestServer.RegisterDatabase(aAlias,self);

  ExportServerNamedPipe(RootURI);
  ExportServerMessage(RootURI);
  LCSHttpServer.AddServer(self);
end; {- TLCSWorkbaseRestServer.Create }

I suspect a bug, or maybe I just didn't get it right myself.

*UPDATE*

- It appears as if nothiong is stored at all, almost like an open transaction that is not comitted or something like that.
- Problem happens with static & dynamic32 and dynamic64 SQLite libs
- Setting breakpoints at procedure TSQLDataBase.TransactionBegin, TSQLDataBase.Rollback and TSQLDataBase.Commit reveals that in case of the SQLite3 native database, the transaction s started twice, before it is committed twice. That appears a bit odd.


Hans

#29 Re: mORMot Framework » Accessing the SQLRest at server side » 2013-03-22 12:14:25

And how about creating a REST client at server side? I could use the DLL or windows messaging based variant to 'clone' the SQLRest coudlnt I

This way the simulation data object would have it's own rest client. SO ... I can I create a "clone" like that from the active restserver when my simulation data object is instantiated?

#30 mORMot Framework » Accessing the SQLRest at server side » 2013-03-22 10:09:11

h.hasenack
Replies: 10

Hi Arnaud

On the Server side i have a background thread (actually a timer) that has to poll for changed data in a volatile (:mem:)  table.
(btw, this table is modified by both client and server - it's a data exchange container for simulation results, parameters and other data.)

The polling is initiated by the client when a 'simulation session' is started, and upon its creation, the polling object obtains a link to the 'active' sqlrest (on server side!) like this:

function TLCSSimulationDataOnRestServer.GetSQLRest: TSQLrest;
begin
  if FSQLRest=nil then
  begin
    if Assigned(ServiceContext.Factory) then
      FSQLRest:=ServiceContext.Factory.RestServer;
  end;
  Result:=FSQLRest;
  if Result=nil then
    raise ELCSSimulationDataFatal.CreateFmt('%s.GetSQLRest - Restserver not available',[ClassName]);
end;

But the problem is, that this REST server may be destroyed because the eg client dropped the connection, and this causes an nasty AV in my server. So what I need to find out is if the FSQLRest link is still valid. I do not see a 'OnDestroy' event handler or something alike that I can use to nillify my FSQLRest field.

Any ideas?

Hans

#31 mORMot Framework » Congratulations! 64bit mode appears to work in most parts. » 2013-03-20 12:20:12

h.hasenack
Replies: 1

Hi Arnaud

I have managed to compile the mormot in 64bit, as well as the SQLLite dll.
So I went on testing with our own unit tests in 64bit and 32bit mode.
It turns out that the SQLite3 native mode passes all tests (32bit and 64bit)
Oracle passes all tests (32bit and 64bit)
ODBC fails in bot 32bit and 64bit mode. (but COnnection is OK) (Connected to the same MSSQL DB as the OleDB connection)
MSSQL works in 32 bit mode, but fails in 64bit mode. (but COnnection is OK)

SOme of the generated errors:
TestTimedCRUD: ETestFailure
at  $0000000003DB62CF
Client.Add failed

TestAuthGuestReadData: ESQLite3Exception
at  $0000000003B815D7
SetUp FAILED: TOleDBStatement.Execute: wrong column "COLUMN_NAME" for "select COLUMN_NAME, DATA_TYPE, CHARACTER_MAXIMUM_LENGTH, NUMERIC_PRECISION, NUMERIC_SCALE, 0 INDEX_COUNT from INFORMATION_SCHEMA.COLUMNS where TABLE_SCHEMA = 'TEST' and TABLE_NAME = 'UNITTESTSAMPLERECORD'"

Thank you for your hard work.

Hans Hasenack
S&G Asset Management

#32 mORMot Framework » RAD Studio beta testing » 2013-03-14 07:27:50

h.hasenack
Replies: 2

Hi Arnaud

I have seen you have been patching rtl routines as you managed to write faster versions in x64. So, My suggestion would be to join the rad studio beta program and give the next release a try.
If you have a EDN account, you can join the program for free and get a full license for about 50 days. This should cover the time to adapt and test your code for the next release of rad studio.
I think many of your suggestions will be very welcome at embarcadero's. (Don't underestimate yourself)

Regards

Hans

#33 Re: mORMot Framework » SQLite3 performance: .obj vs .dll » 2013-01-21 13:58:55

ab wrote:

I think we'll switch to an external DLL for all platforms and CPUs, with the exception of the Win32 platform.

smile Nice!

ab wrote:

Therefore, I suppose we won't need to fix dcc64 issues any more.

sad I don't agree to that one. There is still some work to do to be able to compile Mormot in native 64bit. Some inline assembly is still to be altered, as well as some routines that  require a "pure pascal" alternative.

#34 Re: mORMot Framework » SQLite3 performance: .obj vs .dll » 2013-01-21 08:52:29

ab wrote:

I've created a ticket to track the request. See http://synopse.info/fossil/info/65fe490e9d

Cool!

Are you still working on the other tickets involving some ASM issues that the dcc64 does not like?

Regards - Hans

#35 Re: mORMot Framework » SQLite3 performance: .obj vs .dll » 2013-01-16 19:01:40

This would be no problem for me, especially since encryption is provided as open source,

#36 Re: mORMot Framework » SQLite3 performance: .obj vs .dll » 2013-01-16 15:15:19

Well, I don't see a DLL-Hell problem as long as you regard your application folder (the one with the binaries) as 'holy'. This means that altering or exchanging files in this area by anything else than the installation application is a no-no. (except for de developer of course).
The DLL-Hell mainly existed because every DLL was dnmped in the windows system folder and expected to be sharable between all kinds of applications.

Regarding current prices of disk space and RAM, I just dont care about disk space or some more modules.

I do care about being able to distribute my application in parts (eg plugins) and this brings me back to my bpl/dll solution.
AND, as mentioned before, it is a way to get around any compiler incompatabilities regarding the .obj file format.

#38 Re: mORMot Framework » SQLite3 performance: .obj vs .dll » 2013-01-15 08:17:42

If I'm right, putting the DLL in the same folder as your executable will make it find 'first', independent of other SQLITE3.DLL's that may be on the system. I use this same technique to put my BPL's in the installation folder. (my plugin system requires the usage of runtime bpl's.).
This avoids any conflicts that may occur if bpl's can be found in the PATH, provided that all required DLL's/BPL's are in the executable folder.

Therefore I would not mind mormot being linked (optionally) to a SQLITE3.DLL, as these are available in both 32bit and 64bit, and does not cause any compiler conflicts regarding the .obj format. (Which I managed to get around using the method stated above)

a potential drawback would be, in case of using the DLL that one cannot easily step into the sqlite3 sources when debugging.

Hans

#40 Re: mORMot Framework » Blobfields handling , server side » 2013-01-14 15:25:20

For intermediate I have created this helper class to get around it. Unfortunately I had to copy GetLongStrProp and GetLongStrProp from mormot.pas into my own unit, as they were not declared in the interface section.

I strongly suggest UpdateBlobFields and RetrieveBlobfields to be added to TSQLRest as virtual (abstract?) methods so they will be available on both server and client side.


type
  TSQLRestBlobHelper=class helper for TSQLRest
  public
    function UpdateBlobFields(Value: TSQLRecord): boolean;
    function RetrieveBlobFields(Value: TSQLRecord): boolean;
  end;

...

function TSQLRestBlobHelper.RetrieveBlobFields(Value: TSQLRecord): boolean;
var BlobData: TSQLRawBlob;
    i: integer;
begin
  result := false;
  if (Self=nil) or (Value=nil) or (Value.ID<=0) then
    exit;
  result := True;
  with Value.RecordProps do
  if BlobFields<>nil then
    for i := 0 to high(BlobFields) do
    begin
      Result:=Result and RetrieveBlob(TSQLRecordClass(Value.Classtype),Value.ID,BlobFields[i].Name,BlobData);
      if Result then
        SetLongStrProp(Value,BlobFields[i].PropInfo,BlobData)
      else Exit;
    end;
  result := true;
end;

function TSQLRestBlobHelper.UpdateBlobFields(Value: TSQLRecord): boolean;
var BlobData: RawByteString;
    i: integer;
begin
  result := false;
  if (Self=nil) or (Value=nil) or (Value.ID<=0) then
    exit;
  result := True;
  with Value.RecordProps do
  if BlobFields<>nil then
    for i := 0 to high(BlobFields) do
    begin
      GetLongStrProp(Value,BlobFields[i].PropInfo,BlobData);
      Result:=Result and UpdateBlob(TSQLRecordClass(Value.Classtype),Value.ID,BlobFields[i].Name,BlobData);
      if not result then
        exit;
    end;
end;

#41 mORMot Framework » Blobfields handling , server side » 2013-01-14 13:52:51

h.hasenack
Replies: 5

Hi Ab

I am currently trying to fill a table with data, serverside.

I get access to the "current" rest server using this method:

function TLCSSimulationImpl.GetSQLRest: TSQLRest; // getter for SQLRest property
begin
  if Assigned(ServiceContext.Factory)
    then Result:=ServiceContext.Factory.RestServer
  else Result:=nil;
  if Result=nil then
    raise ELCSSimulationFatal.CreateFmt('%s.GetSQLRest - Restserver not available',[ClassName]);
end;

And I would like to update my blob fields the same way it is accomplished using TSQLRestCLient.UpdateBlobfields, but unfortunately the UpdateBlobFields nor the RetrieveBlobFields methid is implemented in the TSQLRest base class...

  ...
    SQLRest.UpdateBlobFields(Rec);
  ...

Will this be implemented soon, or should I look for a different solution?

regards - hans

#42 Re: mORMot Framework » Error: No Error , related VirtualTableExternalRegisterAll and TSQLAuth » 2013-01-03 15:36:22

Yay! will try!

* UPDATE *

I Tried, and all my unit tests turn up green.

Thanks!

#43 Re: mORMot Framework » Error: No Error , related VirtualTableExternalRegisterAll and TSQLAuth » 2013-01-03 13:38:18

COuld be. I think it's beyond my current knowledge of SQLite. I just took a quick look at the source to see what the warning was about.

I hope you can come up with a test that demononstrates the error and fixes it smile
Typically to reproduce the error you'd have to
1) Register the TSQLAuth tables as external virtual tables
2) start a TSQLRestServer with autorization enabled

This will generate the error when the restserver actually tries to add the virtual table modules to the SQLite3 engine. (well, on XE3/Update it works like that. DOnt know about other delphi releases.)

Regards - Hans

#44 Re: mORMot Framework » Error: No Error , related VirtualTableExternalRegisterAll and TSQLAuth » 2013-01-03 12:49:17

very, very likely

  if( !db->pVtabCtx || !(pTab = db->pVtabCtx->pTab) ){

I can see why the compiler complains, but

  if( !db->pVtabCtx || !(pTab == db->pVtabCtx->pTab) ){

would not fix it because pTab has not been initialized yet...

It's very likely though, that the error is generated here because the TRUE evamulation leads to the SQLITE_MISUSE (21) error

  if( !db->pVtabCtx || !(pTab = db->pVtabCtx->pTab) ){
    sqlite3Error(db, SQLITE_MISUSE, 0);
    sqlite3_mutex_leave(db->mutex);
    return SQLITE_MISUSE_BKPT;
  }

#45 Re: mORMot Framework » Error: No Error , related VirtualTableExternalRegisterAll and TSQLAuth » 2013-01-03 11:54:14

Tried it, but no joy sad

I put in my workaround and it's OK again. I experienced this problem also with an older mormot version when I updated the sqlite3.obj files. Maybe the issue is caused there somewhere: I I also get some BCC32 compiler warnings. Here they are:

Generate 32bit OBJ files
Embarcadero C++ 6.50 for Win32 Copyright (c) 1993-2012 Embarcadero Technologies, Inc.
sqlite3.c:
Warning W8057 sqlite3.c 50241: Parameter 'pBt' is never used in function removeFromSharingList
Warning W8057 sqlite3.c 56448: Parameter 'p' is never used in function sqlite3BtreeSchemaLocked
Warning W8060 sqlite3.c 101978: Possibly incorrect assignment in function sqlite3_declare_vtab
Warning W8060 sqlite3.c 133710: Possibly incorrect assignment in function nodeAcquire
Warning W8060 sqlite3.c 135498: Possibly incorrect assignment in function SplitNode
Warning W8060 sqlite3.c 136357: Possibly incorrect assignment in function rtreeInit
Embarcadero C++ 6.50 for Win32 Copyright (c) 1993-2012 Embarcadero Technologies, Inc.
sqlite3.c:
Warning W8057 sqlite3.c 50241: Parameter 'pBt' is never used in function removeFromSharingList
Warning W8057 sqlite3.c 56448: Parameter 'p' is never used in function sqlite3BtreeSchemaLocked
Warning W8060 sqlite3.c 101978: Possibly incorrect assignment in function sqlite3_declare_vtab
Warning W8060 sqlite3.c 133710: Possibly incorrect assignment in function nodeAcquire
Warning W8060 sqlite3.c 135498: Possibly incorrect assignment in function SplitNode
Warning W8060 sqlite3.c 136357: Possibly incorrect assignment in function rtreeInit
DONE!
Press any key to continue . . .

I think "Possibly incorrect assignment in function sqlite3_declare_vtab" is definitely suspicious in this case.

#46 Re: mORMot Framework » Error: No Error , related VirtualTableExternalRegisterAll and TSQLAuth » 2013-01-03 11:23:07

I have created a work-around that seems to do the trick. Though I have no idea where the error code 21 (some OS error) actually originates from.

function sqlite3_check(DB: TSQLite3DB; aResult: integer): integer;
VAR ErrMessage:string;
begin
  if (DB=0) or (aResult in [SQLITE_ERROR..SQLITE_ROW-1]) then // possible error codes
  case aResult of
    SQLITE_MISUSE:
    begin
       ErrMessage:=string(sqlite3_errmsg(DB));
       if AnsiSameText(ErrMessage,'not an error') then
         aResult:=0 // HH avoid the NO ERROR exception
       else raise ESQLite3Exception.Create(ErrMessage,aResult);
    end
    else raise ESQLite3Exception.Create(DB,aResult);
  end;
  result := aResult;
end;

What's your opinion on this, can you think of a better solution?

#47 mORMot Framework » Error: No Error , related VirtualTableExternalRegisterAll and TSQLAuth » 2013-01-03 10:57:18

h.hasenack
Replies: 8

This is driving me nuts... it already took 2 days of my time.


Using todays mormot checkout, I keep getting this ESQLite3Exception with message "No Error". (Yesterday's version had the same problem)

The exception is raised in the following call stack:

Mormot_D17.SynSQLite3.sqlite3_check(4268688952,21)
Mormot_D17.mORMotSQLite3.TSQLVirtualTableModuleSQLite3.Create(TSQLVirtualTableExternal,$FE6213C0,$FE7BBA20)
Mormot_D17.mORMotSQLite3.TSQLVirtualTableModuleServerDB.Create(TSQLVirtualTableExternal,$FE6213C0)
Mormot_D17.mORMotSQLite3.TSQLRestServerDB.InitializeEngine
Mormot_D17.mORMotSQLite3.TSQLRestServerDB.Create($FE7CB5E0,$FE7BBA20,True)
Mormot_D17.mORMotSQLite3.TSQLRestClientDB.Create($FE7CB5E0,$FE7CB5E0,$FE7BBA20,TSQLRestServerDB,True)
Mormot_D17.mORMotSQLite3.TSQLRestClientDB.Create($FE7CB5E0,$FE7CB5E0,':memory:',TSQLRestServerDB,True,'')
SGMormotTestSuitePkg_D17.uTestMormotDBBaseCommon.TMormotDBBaseCommonTest.SetUp

In order to get the exception I have to
1) Explicitly add TSQLAuthGroup and TSQLAuthUser to my TSQLModel
2) Call VirtualTableExternalRegisterAll(MyModel,Props,FALSE)

When I use VirtualTableExternalRegisterAll(MyModel,Props,TRUE) the error does not occur, obviously because VirtualTableExternalRegister is not called for TSQLAuthGroup and TSQLAuthUser
When I leave out TSQLAuthGroup and TSQLAuthUser the exception is also not raised.

I have a similar problem with my own defined classes, but seem to be unable to pinpoint the cause. I tried debugging into sqlite3.c but that didnt work out because debug info seems to be crippled (unmatching blue bullets/stepping in debugger).

Do you have any idea how to close in on this error? It seems to happen with all engines (Oracle, SQLite3 file, SQLite3 memory and our own NexusDB)

Hans

#48 Re: mORMot Framework » Having trouble with TSQLAuthUser TSQLAuthGroup and creating a TSQLRest » 2013-01-03 07:21:11

ab wrote:

One is copied to the other at constructor level.

Each layer has its own model, so you should better create the model once, including both TSQLAuth* tables, and it will be cloned to both "sides".

I see.

ab wrote:

By the way, executing such SQL statements directly in the database will by pass the whole ORM layer, and could make the cache inconsistent.

Yes we know, here it is intentionally in order to have a Test fixture. (meaning circumstances are the same each time the test is executed). Production code will presumably not look like this.

Thx.

#49 Re: mORMot Framework » Having trouble with TSQLAuthUser TSQLAuthGroup and creating a TSQLRest » 2013-01-02 16:48:32

mmmh when providing the model as server model, everything appears to work as expected. WHy are there actually 2 parameters for server and client model? SHould onbe default to the other maybe?

Regards - Hans

#50 mORMot Framework » Having trouble with TSQLAuthUser TSQLAuthGroup and creating a TSQLRest » 2013-01-02 16:44:43

h.hasenack
Replies: 3

I am creating a restclientdb instance like this:

procedure TMormotDBBaseCommonTest.SetUp;
VAR lModel:TSQLModel;
begin
  inherited;
  InitDBProps(FProps);
  FConn := Props.NewConnection;
  FQuery := Conn.NewStatement;

  lModel := TSQLModel.Create([]);
  PrepModel(lModel);
  VirtualTableExternalRegisterAll(lModel, Props);
  FClient := TSQLRestClientDB.Create(lModel, nil, cSQLiteDBInMemory, TSQLRestServerDB,True); // FAILS HERE with COde 21 "Not an Error
  Client.Model.Owner := Client;
  Client.Server.CreateMissingTables;
  Check(Client.SetUser('Admin', 'synopse'),'Unable to set Admin user'); // Need full acccess rights to add users

  // just to ensure the table is empty prior to our tests
  Client.DB.Execute('DELETE FROM '+lModel.Tables[0].SQLTableName+';');
  Client.DB.Execute('SELECT COUNT (*) FROM '+lModel.Tables[0].SQLTableName+';');
end;

When I add TSQLAuthUser and TSQLAuthGroup to the model (in the PrepModel call), then the line creating thre FCLient failes, with a fine error message 'Not an error' (described in earlier topic how I got there)
On the other hand, If I leave them out, my FCLient gets initialized allright, but unit test code fails because I cannot add any users to my table for some reason. Check code below:

procedure TMormotDBaseAuthTest.SetUp;
var
  lUser: TSQLAuthUser;
  lGroupID: PtrInt;
begin
  inherited Setup;
  Check(Client.Server.DB.ExecuteNoException('DELETE FROM AuthUser WHERE LogonName<>''Admin'';'),'Clear AuthUser table failed'); // drop all records except Admin
  // Add a readonly user and a normal user
  lUser := TSQLAuthUser.Create;
  try
    lUser.LogonName := cGuestUser;
    lUser.PasswordPlain := cGuestPw;
    lGroupID := Client.MainFieldID(TSQLAuthGroup, 'Guest');
    lUser.GroupRights := TSQLAuthGroup(lGroupID);
    Client.Add(lUser, true); // FAILS HERE "Class Missing in Model"

    lUser.LogonName := cNormalUser;
    lUser.PasswordPlain := cGuestPw;
    lGroupID := Client.MainFieldID(TSQLAuthGroup, 'User');
    lUser.GroupRights := TSQLAuthGroup(lGroupID);
    Client.Add(lUser, true);
  finally
    lUser.Free;
  end;

end;

(TMormotDBaseAuthTest is derived from TMormotDBBaseCommonTest)

DId I get the order of inititialization wrong? Hints? Tips?

Regards - Hans

Board footer

Powered by FluxBB