You are not logged in.
So... I got http and named pipes working just fine.
Now Im trying message based C/S communication.
This is how the client is instantiated:
procedure TLCSServerMessageTest.InitRestClient(VAR aSQLRestClient: TSQLRestClientURI);
begin
inherited;
aSQLRestClient:=TSQLRestClientURIMessage.Create(FModel,cLCSServerName,ClassName,1000);
end;And this is how the server is "published" from the constructor. cLCSServerName is the same const on both sides. Obvously I am missing something, but I seem to be unable to find out what.
Win7x64, DelphiXE2
constructor TLCSRestServer.Create;
begin
FRestServersByURI := TObjectDictionary<RawUTF8,TSQLRestServerDB>.Create([]);
FRestServersByAlias := TObjectDictionary<RawUTF8,TSQLRestServerDB>.Create([]);
FServerModel := TLCSServerRootModel.Create(cLCSServerName);
FServerConnProp := CreateConnectionPropertiesForConnectionString(ExpandSNGMacros(cfpLCSServerBaseDBPath));
VirtualTableExternalRegisterAll(FServerModel, FServerConnProp);
inherited Create(FServerModel, ExpandSNGMacros(cfpLCSServerBaseSQLiteDBPath), True);
CreateMissingTables;
ServiceRegister(TLCSServer,[TypeInfo(ILCSServer)],sicShared);
ServiceRegister(TLCSAliasManager,[TypeInfo(ILCSAliasManager)],sicShared);
ServiceRegister(TLCSClientServerSession,[TypeInfo(ILCSClientServerSession)],sicClientDriven);
ExportServerNamedPipe(cLCSServerName);
ExportServerMessage(cLCSServerName);
end; {- TLCSRestServer.Create }I am currently creating a unit test for the TSQLRestClientURIMessage and found that the window handle created does not get destroyed/closed.
this causes in the unit test as creatring a new named window handle fails because of the pre-existing named window handle.
I suggest following change in SYnCOmmons.pas:
A patch is required to ensure the client window handle is remembered even when the server window is not found:
constructor TSQLRestClientURIMessage.Create(aModel: TSQLModel;
const ServerWindowName: string; ClientWindow: HWND; TimeOutMS: cardinal);
begin
inherited Create(aModel);
fClientWindow := ClientWindow; // HHPatch moved to top
fServerWindow := FindWindow(pointer(ServerWindowName),nil);
if fServerWindow=0 then
raise ECommunicationException.CreateFmt('No "%s" window available - server may be down',
[ServerWindowName]);
// fClientWindow := ClientWindow; // HHPatch Moved to top
fTimeOutMS := TimeOutMS;
end;I like the idea of sub-domains very much. Thats what we (Bas and I) came up with too as a solution.
It also allows for separation in 'server root' tables (ef the autorization tables) and other tables (my per-project tables)
Thanks!
Hi Ab
What I want is a single server instance that allows access to multiple (project) databases. The main reasons for separate (project) databases is taht we want to keep data of different customers physically separated, and also because one may DB be in NexusDB and the other in OracleDB or MS-SQL.
So far we have come to a point where we need to add an URL for each (new) project database created. Unfortunately adding new urls to the http server requires administrative rights which will probably not be available for our server. SO I get the feeling I am on the wrong track here.
So I was thinking of way to access multiple DB's through one url. One client connection will require only one database to be opened, but each client may very well open a different database.
Do you have any tips regarding this?
Regards - Hans
Imagine my client starting lengthty process on the server.
And the server (multithreadedly) generates progress information to a point where the processing has finished.
What I want basically are 2 things:
1) display progress that has been made during processing, including warnings and errors.
2) Auto update charts/forms based on a 'new data arrived' notification from the server
While I could do this using a sleep/polling loop on the client side, this feels "bad" to me. So I obviously need some kind of notification system (messages, callbacks?) to get around this.
Taking a quick look at the docs did not point me in the right direction, so... I do I get around this?
Hans
>>If you have some code to propose, I'd merge it with pleasure to the framework trunk.
Will do, once I really get the hang of it.
Regards
I try to work with class and property attributes in XE via new RTTI (TRttiContext e.t.c.) classes and got some problems:
1) An error EInsufficientRtti occurs - not for all cases new RTTI is present
2) VERY slow performance of new RTTI. Very slow.
3) It not work in FPCSo in my opinion using new RTTI is early now.
1) This one is new to me
Why/Where does it happen?
2) This should be fixable by some serious profiling and optimization sessions. If it's slow at generating/verifiying the DB metadata: that's not such a big deal. If it's slow at fetching/retrieving normal data then it definitely IS a big deal.
3) I assume using compiler directives could fix this. As stated before: I would prefer using the new RTTI, but defenitely NOT leave out the "old way index 50", not even in the same delphi version. If an app uses the new rtti to define attributes, however, this should precede the "index 50" definition. If it does NOT use attributes, it should simply use the old "index 50" definition.
Yes it would introduce some extra complexity in your source code. On the other hand, I think the future will drive us more into the class attributes direction. It's a cleaner way of implementing this. I am quite sure that in time it will be taken up by the (brilliant) guys that wrote the FPC. But it wont be ready not tomorrow.
Hans
Dou you have a profiler (like AQTime) to find out where it's slow? I may be of help there as I am quite experienced in optimizing code.
Hans
Hi ab
COnsidering the following class definition
TSQLWorkbaseObject = class(TSQLRecord)
private
FName: UTF8String;
FUserID: UTF8String;
published
property Name: UTF8String index 50 read FName write FName;
property UserID: UTF8String index 20 read FUserID write FUserID;
end;I really dislike the "index 50" in order to define the string length. Could you implement (well, at least for post XE releases) that it first looks at class and property attributes before taking a look at this kind of data definition using the "index" keyword?
This would seriously clean up our code from strange statements while still allowing compatability with earlier versions of Delphi.
Regards - Hans
Yes you were right. In My test I had to call CreateAndFillPrepare, and this took the most time.
I separated the timing mesasurements, CreateAndFillPrepare is now displayed as 'Query took ... ms' and the tested operation is measurured independently. This produces quite a different picture:
Process Start: C:\Users\Hans\Sources\Libsource\mORMot\UnitTest\TestOracle.exe. Base Address: $00400000. Process TestOracle.exe (5620)
Debug Output: ---CREATE TEST--- Process TestOracle.exe (5620)
Debug Output: 138 records added, avg speed 69 rps Batch=False Transaction=False Process TestOracle.exe (5620)
Debug Output: ---READ TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 234 ms Process TestOracle.exe (5620)
Debug Output: 138 records fetched, avg speed 138.000 rps Process TestOracle.exe (5620)
Debug Output: ---MODIFY TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 0 ms Process TestOracle.exe (5620)
Debug Output: 138 records modified, avg speed 402 rps Batch=False Transaction=False Process TestOracle.exe (5620)
Debug Output: ---DELETE TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 234 ms Process TestOracle.exe (5620)
Debug Output: 138 records deleted, avg speed 1.468 rps Batch=False Transaction=False Process TestOracle.exe (5620)
Debug Output: ---CREATE TEST--- Process TestOracle.exe (5620)
Debug Output: 137 records added, avg speed 69 rps Batch=False Transaction=True Process TestOracle.exe (5620)
Debug Output: ---READ TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 234 ms Process TestOracle.exe (5620)
Debug Output: 137 records fetched, avg speed 137.000 rps Process TestOracle.exe (5620)
Debug Output: ---MODIFY TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 0 ms Process TestOracle.exe (5620)
Debug Output: 137 records modified, avg speed 463 rps Batch=False Transaction=True Process TestOracle.exe (5620)
Debug Output: ---DELETE TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 234 ms Process TestOracle.exe (5620)
Debug Output: 137 records deleted, avg speed 2.175 rps Batch=False Transaction=True Process TestOracle.exe (5620)
Debug Output: ---CREATE TEST--- Process TestOracle.exe (5620)
Debug Output: 57.001 records added, avg speed 28.501 rps Batch=True Transaction=False Process TestOracle.exe (5620)
Debug Output: ---READ TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 499 ms Process TestOracle.exe (5620)
Debug Output: 57.001 records fetched, avg speed 1.838.742 rps Process TestOracle.exe (5620)
Debug Output: ---MODIFY TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 0 ms Process TestOracle.exe (5620)
Debug Output: 57.001 records modified, avg speed 12.020 rps Batch=True Transaction=False Process TestOracle.exe (5620)
Debug Output: ---DELETE TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 47.643 ms Process TestOracle.exe (5620)
Debug Output: 57.001 records deleted, avg speed 10.121 rps Batch=True Transaction=False Process TestOracle.exe (5620)
Debug Output: ---CREATE TEST--- Process TestOracle.exe (5620)
Debug Output: 52.001 records added, avg speed 26.001 rps Batch=True Transaction=True Process TestOracle.exe (5620)
Debug Output: ---READ TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 468 ms Process TestOracle.exe (5620)
Debug Output: 52.001 records fetched, avg speed 3.250.063 rps Process TestOracle.exe (5620)
Debug Output: ---MODIFY TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 0 ms Process TestOracle.exe (5620)
Debug Output: 52.001 records modified, avg speed 8.052 rps Batch=True Transaction=True Process TestOracle.exe (5620)
Debug Output: ---DELETE TEST--- Process TestOracle.exe (5620)
Debug Output: Query took 43.493 ms Process TestOracle.exe (5620)
Debug Output: 52.001 records deleted, avg speed 25.255 rps Batch=True Transaction=True Process TestOracle.exe (5620)
What is remarkable : The query run for the delete took 44 s. wheras the exactly the same query run for read and modify test took only 0..450 ms. I guess it has something to do with mormot (or maybe oracle) being busy storing the modifications of the modify test.
I did not investigate, but I'm quire sure there's a way to delete records without fetching them first. (like DELETE FROM .... WHERE ...). This would avoid the CLobFromDescriptor call as well.
Regards - Hans
Great!
Hi AB.
It looks like the TSQLDBOracleLib::ClobFromDescriptor takes all the time. (92.91%). This is ONLY for the (batched) delete action. Thgis is strange, I did not expect any BLOB/CLOB handling to be neccesary when deleting a bunch of records :s
- sorry for the format of the data , I do not have the time to make a nice table format -
Regards - Hans
Routine Name % Time Time Time with Children Shared Time Hit Count
TSQLDBOracleLib::ClobFromDescriptor 92,9069061902652 25,7616422925527 25,7942729994605 99,8734963109509 31001
TSQLDBOracleStatement::ExecutePrepared 4,24131770456711 1,17605153410627 1,17707970966844 99,9126503028027 33
TSQLDBOracleStatement::Step 1,42987056537413 0,396480902661659 0,396480902661659 100 31002
TSQLDBOracleStatement::ColumnsToJSON 0,179882267176309 0,0498785592136639 25,8898052263941 0,192657143526185 31001
TPropInfo::SetValue 0,159425287046616 0,0442061563095382 0,10459283614875 42,2649943698522 186006
TSQLDBOracleLib::Check 0,117679532662149 0,0326307069077142 0,0326307069077142 100 124004
UTF8DecodeToUnicodeString 0,0861917809202405 0,0238996423374387 0,0383495796003135 62,320480658524 155005
GetJSONField 0,0772425465674471 0,0214181586281787 0,0214181586281787 100 279050
StrLen 0,0707899124296421 0,0196289433876839 0,0196289433876839 100 217007All batch operations and transactions were "mod 1000" already in this case.
Changing FBatchSendingAbilities just made things for batchmode same as for unbatched mode. (Which was logical)
I'll run some AQTime profile and see what it brings.
Regards - Hans
This code, with uUaseBatch=True cases the mishap.
procedure TMormotDBBaseCommonTest.ModifyModelData(aUseBatch, aUseTransactions: boolean);
VAR cnt:integer;
Starttime, Endtime: cardinal;
Rec: TSQLSampleRecord;
Ida: TIntegerDynArray;
GUID:TGUID;
begin
cnt:=0;
StartTime:=GetTickCount;
if aUseBatch then
Client.BatchStart(TSQLSampleRecord);
if aUseTransactions then
Client.TransactionBegin(TSQLSampleRecord);
try
Rec := TSQLSampleRecord.CreateAndFillPrepare(Client,'');
try
while Rec.FillOne do
begin
CreateGUID(GUID);
Rec.Question:=GUIDToString(GUID);
if aUseBatch then
begin
Client.BatchUpdate(Rec);
if (cnt mod cBatchSize) = 0 then
begin
Client.BatchSend(Ida); << Causes the crash
Client.BatchStart(TSQLSampleRecord);
end;
end
else Client.Update(Rec);
if aUseTransactions and (cnt mod cCommitSize = 0) then
begin
Client.Commit;
Client.TransactionBegin(TSQLSampleRecord);
end;
inc(cnt);
end;
finally
Rec.Free;
end;
if aUseBatch then
Client.BatchSend(Ida);
if aUseTransactions then
Client.Commit;
except
if aUseBatch then
Client.BatchAbort;
if aUseTransactions then
Client.RollBack;
raise;
end;
EndTime:=GetTickCount;
OutputDebugString(PChar(Format('%.0n records modified, avg speed %.0n rps Batch=%s Transaction=%s',[1.0*cnt,1000.0*cnt/(EndTime-StartTime),BoolToStr(aUseBatch),BoolToStr(aUseTransactions)])));
end;Here's the call stack
System._LStrAsg(???,???)
SQLite3DB.TSQLRestServerStaticExternal.InternalBatchStop
SQLite3Commons.TSQLRestServer.RunBatch($ABB32D0,TSQLSampleRecord,'}','','')
SQLite3Commons.TSQLRestServer.URI('root/SampleRecord/0','POST','{"SampleRecord'#0#0'["PUT'#0#0'{"RowID":7345{"Time":135055073769,"Name":"","Question":"{DB304BC4-E286-440C-84E7-5896261F6613}","Address_":"","PostalCode":"","City":""}]}','','',$603BF9)
SQLite3.TSQLRestClientDB.InternalURI('root/SampleRecord/0','POST',$5E3548 {''},$18F5F8 {''},$18F6B8 {'{"SampleRecord'#0#0'["PUT'#0#0'{"RowID":7345{"Time":135055073769,"Name":"","Question":"{DB304BC4-E286-440C-84E7-5896261F6613}","Address_":"","PostalCode":"","City":""}]}'})
SQLite3Commons.TSQLRestClientURI.URI('root/SampleRecord/0','POST',$18F6B4 {''},nil {''},$18F6B8 {'{"SampleRecord'#0#0'["PUT'#0#0'{"RowID":7345{"Time":135055073769,"Name":"","Question":"{DB304BC4-E286-440C-84E7-5896261F6613}","Address_":"","PostalCode":"","City":""}]}'})
SQLite3Commons.TSQLRestClientURI.BatchSend(())
uTestMormotDBBaseCommon.TMormotDBBaseCommonTest.ModifyModelData(True,False)
uTestMormotDBBaseCommon.TMormotDBBaseCommonTest.TestTimedCRUDBatched
TestFramework.TTestCase.Invoke((uTestMormotDBBaseCommon.TMormotDBBaseCommonTest.TestTimedCRUDBatched,$2278588))
TestFramework.TTestCase.RunTest($22F5A30)
Here's my diagnosis:
the for j= loop goes to high(Fields) and not to high(Values) , therefor the last element in Values[] is not initialized.
Hope it helps.
Here's a short overview of benchmarks using an oracle 10g server over a LAN. Same test is performed using combinations of batch and transactions. The Create records routine stops after 2000 ms, and the resulting records are used in the next steps, using same transactional and batch mode settings.
Process Start: C:\Users\Hans\Sources\Libsource\mORMot\UnitTest\TestOracle.exe. Base Address: $00400000. Process TestOracle.exe (6012)
Debug Output: 138 records added, avg speed 69 rps Batch=False Transaction=False Process TestOracle.exe (6012)
Debug Output: 138 records fetched, avg speed 590 rps Process TestOracle.exe (6012)
Debug Output: 138 records modified, avg speed 354 rps Batch=False Transaction=False Process TestOracle.exe (6012)
Debug Output: 138 records deleted, avg speed 442 rps Batch=False Transaction=False Process TestOracle.exe (6012)
Debug Output: 139 records added, avg speed 70 rps Batch=False Transaction=True Process TestOracle.exe (6012)
Debug Output: 139 records fetched, avg speed 594 rps Process TestOracle.exe (6012)
Debug Output: 139 records modified, avg speed 446 rps Batch=False Transaction=True Process TestOracle.exe (6012)
Debug Output: 139 records deleted, avg speed 468 rps Batch=False Transaction=True Process TestOracle.exe (6012)
Debug Output: 58.001 records added, avg speed 29.001 rps Batch=True Transaction=False Process TestOracle.exe (6012)
Debug Output: 58.001 records fetched, avg speed 109.436 rps Process TestOracle.exe (6012)
Debug Output: 58.001 records modified, avg speed 12.863 rps Batch=True Transaction=False Process TestOracle.exe (6012)
Debug Output: 58.001 records deleted, avg speed 1.025 rps Batch=True Transaction=False Process TestOracle.exe (6012)
Debug Output: 55.001 records added, avg speed 27.501 rps Batch=True Transaction=True Process TestOracle.exe (6012)
Debug Output: 55.001 records fetched, avg speed 113.638 rps Process TestOracle.exe (6012)
Debug Output: 55.001 records modified, avg speed 12.638 rps Batch=True Transaction=True Process TestOracle.exe (6012)
Debug Output: 55.001 records deleted, avg speed 1.038 rps Batch=True Transaction=True Process TestOracle.exe (6012)What strikes me is the relatively poor performance on deleting records, I had expected it to be faster than modify. I did not examine yet wheter most time is spent in OCI or the mormot/sqlite3 framework.
What is your opinion? Do you want my unit test to try it yourself?
Hans
I adjusted a routine to avoid the AV. Obviously the buffer wasnt initialized properly. Around line 628 in SQLite3DB.pas
procedure TSQLRestServerStaticExternal.InternalBatchStop;
....
if fBatchMethod=mPut then
begin
j:=Decode.FieldCount;
if Values[j]=nil then // HH adjusted, ensure array is not nil to avoid AV,
SetLength(Values[j],Math.Min(fBatchCount-BatchBegin,max));
Values[j,n] := fBatchIDs[i]; // ?=ID parameter
end;
....The old lines looked like this:
....
if fBatchMethod=mPut then
Values[Decode.FieldCount,n] := fBatchIDs[i]; // ?=ID parameter
....BEFORE you put time in this, obviously I needed to sleep a little more ... I found the error: I Freed the client after filling it with data.
I apologize. - Thnx.
Hi
We are writing a nexusDB driver, and using a unit test to accomplish this. In order to perform a basic test we have first created a unit test based on sample code.
Apart from that we have created a small sample application that simply dumps and rereads some objects from the oracle database.
The code in TestOracle.rar appears to run just fine, whereas the code in our unittest.rar crashes for some reason, and it is basically the same code.
I am lost here, maybe you can pinpoint me at (probably mine) the error. (Let's see if I can add the attachments ...)
function TSQLRest.InternalListRecordsJSON(Table: TSQLRecordClass;
const WhereClause: RawUTF8): TSQLTableJSON;
VAR aSQL:RawUTF8;
begin
if (self=nil) or (Table=nil) then
result := nil else
begin
aSQL:=Table.RecordProps.SQLFromSelectWhere('*',WhereClause); << MESSES UP THE HEAP/STACK ? // separated in 2 lines for easier debugging
result := InternalListJSON(Table,aSQL); <<CRASH!
end;
// result := InternalListJSON(Table,
// Table.RecordProps.SQLFromSelectWhere('*',WhereClause));
end;the rar of the working code can be found here:
http://www.sg-assetmanagement.nl/secret … Oracle.rar
And expects itself to run from ./sqlite3/sample/testoracle
the rar of the crasjhing unit test (XE2) can be found here:
http://www.sg-assetmanagement.nl/secret … itTest.rar
and expects to compile and run from a folder 'UnitTest' wich is a brother folder of the SQLite3 folder.
We are running oracle 10g, and we have copied the OCI DLL's and support files into the executable output folders. (I believe this approach called "Oracle InstantClient")
Regards - Hans
Never mind. I found it in the Synopse.inc file.
{.$define USEPACKAGES}
{ define this if you compile the unit within a Delphi package
- it will avoid error like "[DCC Error] E2201 Need imported data reference ($G)
to access 'VarCopyProc' from unit 'SynCommons'"
- shall be set at the package options level, and left untouched by default }
Hans
As we use runtime packes for our plugin system, we require the mORMot to be compiled into a package.
I am trying to do this, but get stuck on
[DCC Error] E2201 Need imported data reference ($G) to access 'VarCopyProc' from unit 'SynCommons'
Reading help reveals I need to add
{$IMPORTEDDATA ON}
at top of the unit, butr no joy.
Maybe it is because the call to VarCopyProc is inside an assembler block?
SO ...
How do I get around this
What sources shpuld be in the mORMot package (minimal)
Obvously I'll need an sqlite package too... What sources go in there?
OT: Looking at these assembler blocks makes me feel a bit inconfident about a 64 bit version being avalable soon... What is to be expected just about when?
Regards - Hans
FOA - thanks for the quick reply.
1) Yes we will defnitely use object inheritance. But this inheritance is to be dynamic too.
2) Objects of the same class will all have the same properties, so these "dynamic" properties need to be defined somehow with the object type.
Rethinking this, I'll probably need to translate my current (RDB) metadata model into a usable Object class model that reflects all the aspects I require.
Then I can use these objects from within the ORM "normally".
If I come up with anything better I'll let you know.
Regards - Hans
Hi
I have a need for being able to use runtime defined properties.
I have a "workbase" containg all the objects converted/imported from different sources of the customer's database. (10M+ objects)
Imaging objects like transformators, power lines and switches.
SOme attributes are shared (eg manufacturer) and some attributes are not (like inner diameter, cooling oil and length).
Is it possible to use Mormot to define properties at runtime for my objects, or should I create a (SAP like) data model with an object table, attribute definition table and attribute values table.
The core problem here is that the attributes can be of different types, which are defined/known at runtime only.
IMHO The SAP solution for this is not so good: they use a single attribute value table with different columns for each value type. Apart from that, having an average of 30 attributes per object would result in a single attribute table of 300M+ records of which usually only 1 of 20 fields is filled out.
What would be the best practice to solve this with mORMmot?
Kindest regards
Hans