You are not logged in.
As you can see in the Callstack in the initial post, SQLite3Commons.TSQLRestServer.URI() is called, but before it reaches the call to AdaptSQLLimitForEngineList on line 18316 you see a call to SQLite3Commons.TSQLRestServer.LaunchCallBack($18F464,'',400) happening ...
Stepping through the code, the call to LaunchCallBack is made when executing line 18289
if (Method<>mLOCK) and
// GET ModelRoot/MethodName + parameters sent in URI
((URI.MethodIndex<0) or not LaunchCallBack(URI,Resp,result.Lo)) thenBut that one isnt called either ...
I have my own implementation in the TSQLDBNexusDBConnectionProperties class, so I don't have to change any code that is directly delivered by you. This means i have no merging to do when you publish a new version
You are right: those standards tables were not meant to be defined as external.
If they are not meant to be defined as external, I'm not really sure how you see the use of UserAuthentication with these tables when using an external database.
Even though i don't like public functions, I understand you sticking to SOLID principles ;-)
I'm trying to use the Authentication build into mormot, with the AuthGroup and AuthUser tables external in my own database ..
But calling SetUser results in an error because the LIMIT 1 addition is not corrected by a call to my own implementation of AdaptSQLLimitForEngineList() ..
See callstack below:
SQLite3DB.TSQLRestServerStaticExternal.ExecuteInlined('SELECT RowID,LogonName,DisplayName,PasswordHashHexa,GroupRights FROM AuthUser WHERE LogonName=:(''Guest''): LIMIT 1;',True)
SQLite3DB.TSQLRestServerStaticExternal.EngineList('SELECT RowID,LogonName,DisplayName,PasswordHashHexa,GroupRights FROM AuthUser WHERE LogonName=:(''Guest''): LIMIT 1;',False,nil)
SQLite3Commons.TSQLRestServer.InternalListJSON(TSQLAuthUser,'SELECT RowID,LogonName,DisplayName,PasswordHashHexa,GroupRights FROM AuthUser WHERE LogonName=:(''Guest''): LIMIT 1;')
SQLite3Commons.TSQLRest.Retrieve('LogonName=:(''Guest''):',$2643290)
SQLite3Commons.TSQLRecord.Create($2596260,'LogonName=?',(...))
SQLite3Commons.TSQLRestServer.Auth($18F464)
SQLite3Commons.TSQLRestServer.LaunchCallBack($18F464,'',400)
SQLite3Commons.TSQLRestServer.URI('root/auth?UserName=Guest&Password=2e4a799a589f4381206254b31517a86d5274bb3d39cd60f72c906d90a2bddbdc&ClientNonce=999a42861b620c0ee361d432c044b5bac72d39f6f89474f962a31fc99b3d8077','GET','','','',$99564D)
SQLite3.TSQLRestClientDB.InternalURI('root/auth?UserName=Guest&Password=2e4a799a589f4381206254b31517a86d5274bb3d39cd60f72c906d90a2bddbdc&ClientNonce=999a42861b620c0ee361d432c044b5bac72d39f6f89474f962a31fc99b3d8077','GET',$18F674 {''},$18F578 {''},$18F574 {''})
SQLite3Commons.TSQLRestClientURI.URI('root/auth?UserName=Guest&Password=2e4a799a589f4381206254b31517a86d5274bb3d39cd60f72c906d90a2bddbdc&ClientNonce=999a42861b620c0ee361d432c044b5bac72d39f6f89474f962a31fc99b3d8077','GET',$18F674 {''},nil {''},nil {''})
SQLite3Commons.TSQLRestClientURI.CallBackGet('auth',(...),'',nil,0,nil {''})
SQLite3Commons.TSQLRestClientURI.CallBackGetResult('auth',(...),nil,0)
SQLite3Commons.TSQLRestClientURI.SetUser('Guest','synopse',False)
When i try to create an External AuthGroup table in oracle, the following error occurs:
ORA-02329: column of datatype LOB cannot be unique or primary key
The create statement constructed is as follows:
SQLite3DB.TSQLRestServerStaticExternal.ExecuteDirect('CREATE TABLE AuthGroup (ID NUMBER(22,0) PRIMARY KEY, Ident NCLOB NOT NULL UNIQUE,SessionTimeout NUMBER(22,0),AccessRights NCLOB)',(...),(...),False)Why not add a generic method to TSQLModel to register all tables of the model externally to a TSQLConnectionPropertes?
procedure TSQLModel.RegisterAllTablesExternal(aProps: TSQLDBConnectionProperties);
var
i: integer;
begin
for i := Low(Tables) to High(Tables) do
VirtualTableExternalRegister(Self, Table[Tables[i].SQLTableName], aProps, Tables[i].SQLTableName);
end;aHa!
But when creating the TSQLRestServer i have set HandleUserauthentication to false ...
You mean running THttpApiServer.AddUrlAuthorize() ?
Done that
I'm trying to accomplish the following setup:
I have a "generic" server process "LCSRestServer" running, implementing a very simple interface with a TSQLRestServerDB instance. This server uses its own very simple TLCSServerModel = class(TSQLModel).
type
ILCS = interface(IInvokable)
['{31969E17-758B-4077-B0E8-77B81B082715}']
{1 open the database as specified in the connectionstring, return if this was succesfull
- aConnectionstring contains all info needed to connect using <protocol>://server/name syntax
- if aCreate is true than create the database if it doent exist
}
function OpenDatabase(aConnectionString: RawUTF8; var URI: RawUTF8; aCreate: boolean = false): boolean;
end;The client process can now tell the server to open the specified database, making it available to the client. This database is implementing a different TSQLModel ("TLCSModel"), the RestServer must be implementing a different interface "ILCSDatabase"
The implementation of the OpenDatabase method being:
function TLCSServer.OpenDatabase(aConnectionString: RawUTF8; var URI: RawUTF8; aCreate: boolean = false): boolean;
var
...
begin
//Determine database type and protocol
lConProp := CreateConnectionProperties(aConnectionString);
if lConProp <> nil then
begin
//Create separate TSQLRestSeverDB object for this database
lModel := TLCSModel.Create(URI);
lModel.RegisterVirtualTables(lConProp);
lRestServerDB := TLCSRestDB.Create(lModel, UTF8ToString(URI) + '.db3');
lRestServerDB.CreateMissingTables;
lRestServerDB.ServiceRegister(TLCSDatabase , [TypeInfo(ILCSDatabase)], sicShared);
//Register the new restServer with the http server of the main server process
LCSRestServer.RegisterDatabase(lRestServerDB);
Result := true;
end;
end;Somehow i'm not getting this whole setup working... The above method does work, but when i try to run the following method on the client side, it says it can't find LCSDatabase as a service on the server ...
FModel := TLCSModel.Create(cURI);
FClient := TSQLite3HttpClient.Create('localhost','888',FModel);
FClient.ServiceRegister([TypeInfo(ILCS)],sicShared);
FClient.Services[cURI].Get(FLcsInterface);
FLcsInterface.OpenDatabase(cNewFolder, lDBUri, True);
FClient.ServiceRegister([TypeInfo(ILCSDatabase)],sicShared); // <<==Error: "LCSDatabase" interface or REST routing not supported by server
FClient.Services[lDBUri].Get(lDbInterface);What am i doing wrong?
OK, i'll add my own implementation of CreateDatabase to my NexusDB driver.
Maybe its an idea to add some kind of "CanCreate():boolean" function to the main TSQLDBCOnnectionProperties, or add interfaces to let the code determine if its possible to create a database, independent of the actual TConnectionproperties implementation..
type
ICanCreateDatabase = interface(IInterface)
['{C6A016A5-2A6E-48F6-9A4D-310B643B53C0}']
function CanCreate: boolean;
function DatabaseExists: boolean;
function CreateDatabase: boolean;
end;Maybe I am not looking straight ... but I can't seem to find any possibility to create a new database through a TSQLDBconnection or TSQLDBConnectionproperties...
Any plans on this?
Simple question:
Is it possible to generate the mORMot PDF files (esp the SAD) by myself?
I have declared the following class:
TSQLUnitTestSampleRecord = class(TSQLRecord)
private
fQuestion: string;
fName: string;
fTime: TModTime;
FAddress: string;
fPostal: string;
FCity: string;
FDatum: TDatetime;
FFloatNumber: double;
FIntNumber: Int64;
FCurNumber: Currency;
published
property ModTime: TModTime read fTime write fTime;
property Name: string index 50 read fName write fName;
property Question: string index 50 read fQuestion write fQuestion;
property Address_: string index 50 read FAddress write fAddress;
property PostalCode: string index 12 read fPostal write fPostal;
property City: string index 50 read FCity write FCity;
property Datum: TDatetime read FDatum write FDatum;
property FloatNumber: double read FFloatNumber write FFloatNumber;
property IntNumber: Int64 read FIntNumber write FIntNumber;
property CurrencyNumber: Currency read FCurNumber write FCurNumber;
end;When i use this class in an oracle 11g environment, executing CresateMissingTables results in the following error:
ESQLDBOracle exception, with message 'Column "FLOATNUMBER": unknown type 101'
If i change the DB_FIELDS definition for Oracle to contain FLOAT in stead of BINARY_DOUBLE everything works find:
const
/// the known column data types corresponding to our TSQLDBFieldType types
// - will be used e.g. for TSQLDBConnectionProperties.SQLFieldCreate()
DB_FIELDS: array[TSQLDBDefinition] of TSQLDBFieldTypeDefinition = (
// ftNull, ftInt64, ftDouble, ftCurrency, ftDate, ftUTF8, ftBlob
(' NVARCHAR(%)',' BIGINT',' DOUBLE',' NUMERIC(19,4)',' TIMESTAMP',' CLOB',' BLOB'),
(' NVARCHAR(%)',' BIGINT',' DOUBLE',' NUMERIC(19,4)',' TIMESTAMP',' CLOB',' BLOB'),
(' NVARCHAR2(%)',' NUMBER(22,0)',' FLOAT',' NUMBER(19,4)',' DATE',' NCLOB',' BLOB'), //Oracle , changed BINARY_DOUBLE into FLOATHi Ab, thanks for your replies!
I really appreciate the fact that you are always open to suggestions from users, not afraid to acknowledge that somethings things could be made clearer or more convenient from a user point of view. Makes for a very serious user-participation feeling
Thanks
Ok, fog is clearing up a bit now
What i also would expect is a CreateFillAndPrepare that will only substitute '?' with bound parameters
Using an Oracle database, somethings strange happens when i execute the following code:
// Select a single record by Name
lName := Format(cPPName, [cResultID]);
Rec := TSQLUnitTestSampleRecord.CreateAndFillPrepare(Client,'Name=?', [lname], '');Executing this results in an error stating: Executeprepared expected 1 bound parameters, got 0
Looking at the callstack, it seems that the supplied parameter is not formatted into the SQLwhere clause by the FormatUTF8(FormatSQLWhere,ParamsSQLWhere) call
SQLite3Commons.TSQLRest.InternalListRecordsJSON(TSQLUnitTestSampleRecord,'Name=?')
SQLite3Commons.TSQLRecord.CreateAndFillPrepare($2470890,'Name=?','')
SQLite3Commons.TSQLRecord.CreateAndFillPrepare($2470890,'Name=?',(...),'')
uTestMormotDBBaseCommon.TMormotDBBaseCommonSelectTest.TestSelectSingleRecordByString
[Edit]
Extra info: If i add another parameter to the CreateAndFillPrepare, it works
Rec := TSQLUnitTestSampleRecord.CreateAndFillPrepare(Client,'Name=?', [], [lname], '');I see now that two different const arrays should be supplied to substitute the % and the ? parameters separately .. mmmmm :-(
IMHO its a very confusing construction
This test is done against a oracle database, the empty datetime returned as '0000-01-01T00:00:00' is from the Oracle driver ....
It turns out the field was empty for another reason (on creating the record we used BatchAdd(Rec, false, true) in stead of BatchAdd(Rec, True, True))
This created empty records in the oracle table, and when they were read back, the TDatetime field was empty too of course
While running our own unittests on an oracle database, we get an Integer Overflow error while deleting records using batch mode
Call stack:
SynCommons.Div100(???)
SynCommons.Iso8601ToDateTimePUTF8CharVar('00-01-01T00:00:00',17,0)
SQLite3Commons.TPropInfo.SetValue($25032E0,'0000-01-01T00:00:00')
SQLite3Commons.TSQLRecordFill.Fill($7C570038)
SQLite3Commons.TSQLRecord.FillRow(1,nil)
SQLite3Commons.TSQLRecord.FillOne
uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.DeleteModelData(True,True)
uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.TestTimedCRUDBatchedTransactionCode:
procedure TMormotDBBaseCommonCRUDTest.DeleteModelData(aUseBatch, aUseTransactions: boolean);
VAR cnt:integer;
Starttime, Endtime: cardinal;
Rec: TSQLUnitTestSampleRecord;
Ida: TIntegerDynArray;
begin
OutputDebugString('---DELETE TEST---');
cnt:=0;
if aUseBatch then
Client.BatchStart(TSQLUnitTestSampleRecord);
if aUseTransactions then
Client.TransactionBegin(TSQLUnitTestSampleRecord);
StartTime:=GetTickCount;
try
Rec := TSQLUnitTestSampleRecord.CreateAndFillPrepare(Client,'');
EndTime:=GetTickCount;
OutputDebugString(PChar(Format('Query took %.0n ms',[1.0*(EndTime-StartTime)])));
StartTime:=GetTickCount;
try
while Rec.FillOne do
begin
if aUseBatch then
begin
Client.BatchDelete(Rec.ID);
if (cnt mod cBatchSize) = 0 then
begin
Client.BatchSend(Ida);
Client.BatchStart(TSQLUnitTestSampleRecord);
end;
end
else Client.Delete(TSQLUnitTestSampleRecord,Rec.ID);
if aUseTransactions and (cnt mod cCommitSize = 0) then
begin
Client.Commit;
Client.TransactionBegin(TSQLUnitTestSampleRecord);
end;
inc(cnt);
end;
finally
Rec.Free;
end;
if aUseBatch then
Client.BatchSend(Ida);
if aUseTransactions then
Client.Commit;
except
if aUseBatch then
Client.BatchAbort;
if aUseTransactions then
Client.RollBack;
raise;
end;
EndTime:=GetTickCount;
OutputDebugString(PChar(Format('%.0n records deleted, avg speed %.0n rps Batch=%s Transaction=%s',[1.0*cnt,1000.0*cnt/max(1,(EndTime-StartTime)),BoolToStr(aUseBatch),BoolToStr(aUseTransactions)])));
end;I'm trying to create a unit test where selection of a specific record will be done on a TDatetime field.
// Create 10 records
CreateSmallModelData;
// Select a single record by Date
Rec := TSQLUnitTestSampleRecord.CreateAndFillPrepare(Client,'Datum=?', [], [DateToIso8601Text( EncodeDate(2012, 5, 4))]);
Rec.FillOne;
CheckEqualsString(Format(cPPName, [4]), Rec.Name);When i run this against an oracle database, the SQL statement that is eventually executed in the TSQLRestServerStaticExternal.ExecuteInlined method is
SELECT ID,ModTime,Name,Question,Address_,PostalCode,City,Datum FROM UnitTestSampleRecord WHERE Datum="2012-05-04'';
When i run this against the nexusDB database (for which we are coding a native driver for mORMot) then the executed method is:
SELECT ID,ModTime,Name,Question,Address_,PostalCode,City,Datum FROM UnitTestSampleRecord WHERE Datum=''2012-05-04'';
Exactly the same .. Both oracle and Nexus have their own ISO8601 implementation but i dont see any of those methods used when executing a SQL statement via
the ExecuteInline way ... When Nexus tries to execute the SQL it results in an error, because a date literal should always be preceded with the keyword "DATE"
How should we solve this?
Getting an error compiling SynDB, because the protected keyword cannot be used in a record structure.
so, this
TQueryValue = {$ifndef UNICODE}object{$else}record{$endif}
protected
/// fName should be the first property, i.e. the searched hashed value
fName: string;
fValue: Variant;
....should be something like this
TQueryValue = {$ifndef UNICODE}object{$else}record{$endif}
{$ifndef UNICODE}protected{$endif}
/// fName should be the first property, i.e. the searched hashed value
fName: string;
fValue: Variant;
....Ok, thanks for the adjustments.
Is it possible to add my own native database implementation, without having to make major alterations to your sources?
With the current setup I have to add dNexusDB to the TSQLDBDefinition. But when i do that, i also have to alter all the const arrays containing database specific implementations ..
So i was wondering if that was the way to go ..
Any news on this issue? Or any idea when you will be able to implement this?
In my current implementation of NexusDB i use the original TQuery.Params[] created when assigning the SQL statement to the TQuery.SQL.Text property
The binding is done with BindTextU and this assigns the Iso8601 string to the param value .AsString
As far as i know the TParam object does not know what kind of datatype is expected ... so i cannot make a special case when a date is given
The DATE keyword is mandatory for NexusDB, so I guess there will be another constant array referring to NexusDBIso8601ToDate methods like the already present OracleSQLIso8601ToDate
Another difference: date literals in parameters, for NexusDB they need to be formatted like DATE '2012-01-29'
How can i determine the database implementation used in i.e. TSQLRest?
Busy implementing a native NexusDB implementation, i ran into a difference of SQL implementation.
While SQLite is using LIMIT 1 to only retrieve one record with a SQL statement, NexusDB uses TOP 1
How can i best implement this?
SHould we add LIMIT 1 to the implementation specific constants as DB_FIELDS[] and DB_SERVERTIME[] ?
Edit:
The place of TOP 1 in the SELECT statement is also different form the LIMIT 1
<TOP n> should be directly behind the SELECT keyword
Ok, your first line had me convinced LOL: "Step" is not "Next"
But what confused me was the word "Next"in the first line in the documentation of the Step() method:
{/ Access the next or first row of data from the SQL Statement result
- return true on success, with data ready to be retrieved by Column*()
- return false if no more row is available (e.g. if the SQL statement
is not a SELECT but an UPDATE or INSERT command)
- if SeekFirst is TRUE, will put the cursor on the first row of results
- should raise an Exception on any error }
function Step(SeekFirst: boolean=false): boolean; virtual; abstract;I'm still not convinced ;-)
What will be the result of the following code:
var Customer: Variant;
begin
with Props.Execute( 'select * from Sales.Customer where AccountNumber like ?', ['AW000001%'],@Customer) do
assert(Copy(Customer.AccountNumber,1,8)='AW000001');
end;Or in a question: Isn't the first record the active record after running Props.Execute() ?
While trying to code a native NexusDB driver something struck me in the base code of TSQLDBStatement.
The FetchAllToJSON method starts the data-processing loop with a call to Step. If I would follow this in my NexusDB implementation (or rather if i did not override it ;-)) then the first record would not be processed, as this is the current record when entering the FetchAllToJSON method.
I would expect the data processing loop to look something like this
while Step(result=0) do
begin
ColumnsToJSON(W);
W.Add(',');
inc(result);
end;Thanks for your suggestions, they are really usefull!
The reason we are looking for this extra implementation is that the current version of our product make heavy use of NexusDB. We are in the start-up of developing a major overhaul of our application, and of course cannot ignore our installed base, even it is only necessary to convert to a better solution ;-)
We'll will keep you posted on our progress, and of course will supply you with the resulting unit
We would like to have native support in mORMot for NexusDB database servers, supporting both the embedded server possibilities of Nexusdb and the remote server possibilities.
Performance is a very big issue, so we don't want to introduce any extra layers by using OLEDB.
How do we start creating this SynDBNexusDB.pas? (or maybe somebody already has started work in this ;-)?)
What kind of already existing unit-tests can we use to determine if our implementation is complete and in working order?
I'll find out how to circumvent this issue. Certainly by setting OCI_ATTR_DATEFORMAT for the column.
Can i set this attribute by hand somehow? maybe with SQLPLUS?
Hahaha you have fixed the problem faster than i can think of a workaround,
thanks
Ok, solved this one myself ;-)
Turns out the Perftest uses a SampleRecord table, and that one already existed in our database LOL
Next error:
ORA-01861: Literal does not match format string
probably during "select max(id) from SampleRecord"
no the error occurs during the following statement:
SQLite3DB.TSQLRestServerStaticExternal.ExecuteInlined('insert into SampleRecord (ID,FirstName,LastName,Amount,BirthDate,LastChange,CreatedAt) VALUES (:(1):,:(''Namé 5''):,:(''5''):,:(0.05):,:(''1900-01-05''):,:(135052048292):,:(135052048292):)',False)I'm trying to run the new Perftest project against our own oracle 10g installation, but getting the following error:
..raised exception class ESQLDBOracle with message 'ORA-00904: "CREATEDAT": invalid identifier
Any suggestions?
Thanks for the blog-entry! It made a lot of things clearer to me.
One correction: the link at the bottom of the article to this forum thread is not correct
Again, thanks for the advise!
One of the things I don't like so much about your approach to the ORM is the mis-use of existing Delphi constructs like "index n" attribute for the maximum length of a string-property. Other ORMs solve this i.e. with official Class-attributes like TMS Aurelius:
type
[Entity]
[Table('ARTISTS')]
[Sequence('SEQ_ARTISTS')]
[Id('FId', TIdGenerator.IdentityOrSequence)]
TArtist = class
private
[Column('ID', [TColumnProp.Unique, TColumnProp.Required, TColumnProp.NoUpdate])]
FId: Integer;
FArtistName: string;
FGenre: Nullable<string>;
function GetArtistName: string;
procedure SetArtistName(const Value: string);
public
property Id: integer read FId;
[Column('ARTIST_NAME', [TColumnProp.Required], 100)]
property ArtistName: string read GetArtistName write SetArtistName;
[Column('GENRE', [], 100)]
property Genre: Nullable<string> read FGenre write FGenre;
end;IMHO this approach has a few advantages:
Every class can be persisted this way, without any specific demands on class-hierarchy
Cleaner code, much better readable because there are no standard language construct that mean something else in this ORM context
What are your thoughts on this?
Thanks for the info
Your answer does confuse me a bit. What do you mean by "Client-server ORM" as opposed to SOA mentioned later on?
I have read the SAD Doc, until page 186 and it does appeal to me ;-)
But now i have to convince my boss, when he gets back from holiday
We are not sure yet. At the moment we are testing several ORMs learning what can be realistic performances.
At the moment we have a single client windows application, who's design principles date more than 15 years back. It uses an embedded NexusDB engine which is (of course) lightning fast. There are a lot of very data-intensive calculations needed in all kinds of analyses. The dataset can be from just a few Mb to several dozens of GB. Becuase of its design all the analyses were done on a record-by-record basis in the client (Dataset.First - .Next - Until .EOF)
For the newly designed application client-server with multiple clients (windows, HTML, Scripting, ...), SOAP or REST, multiple databases (Oracle and MSSQL required, others are optional) are all MustHave requirements. This all demands a very different approach to the new design, and our way of thinking (guess we need a bit of re-education ;-)).
The application also has to have the possibility to convert external data into its own model, and this could also be a few MB or several GB that has to be converted on a regular basis...
Another important function for us developer ( not so much from a users point of view) is the possibility to update the Model and its underlying database when the model changes with new version of the application (new or dropped classes, new or changed published properties, new indeces etc). This is something we have not found in mORMot yet (while it is present i.e. in InstantObjects. Any plans on such a thing?
TNSPING has a 10 ms resolution, so it is not precise enough when you want to know the latency of multiple requests.
A good old PING does make sense in such case.
Good old Ping says: 0ms average
The latency as stated by TNSPING is 10-20 msec
Executed some more tests, directly executing 4 possible scenarios in one go, using the following code:
procedure TForm1.FormCreate(Sender: TObject);
begin
Props := TSQLDBOracleConnectionProperties.Create('OPFTEST','OPF_MORMOT','secret', 871);
Model := CreateSampleModel; // from SampleData unit
VirtualTableExternalRegister(Model,TSQLSampleRecord,Props,'SampleRecord');
Client := TSQLRestClientDB.Create(Model, nil, 'test.db3', TSQLRestServerDB);
Client.Server.CreateMissingTables;
end;
function TForm1.Addmany(aUseBatch, aUseTransactions: boolean): double;
var
i: Integer;
Starttime, Endtime: cardinal;
Rec: TSQLSampleRecord;
Ida: TIntegerDynArray;
begin
Rec := TSQLSampleRecord.Create;
Rec.Question := QuestionMemo.Text;
Rec.Address_ := 'Dit is het adres laan, 23';
Rec.postalCode := '1234AA';
Rec.City := 'Nijmegen';
if aUseBatch then
Client.BatchStart(TSQLSampleRecord);
if aUseTransactions then
Client.TransactionBegin(TSQLSampleRecord);
StartTime := GetTickCount;
try
for i := 0 to cRecordCount do
begin
Rec.Name := NameEdit.Text + IntToString(i);
if aUseBatch then
begin
Client.BatchAdd(Rec,false,true);
if (i mod 1000) = 0 then
begin
Client.BatchSend(Ida);
Client.BatchStart(TSQLSampleRecord);
end;
end
else
Client.Add(Rec,true);
end;
finally
Rec.Free;
if aUseBatch then
Client.BatchSend(Ida);
if aUseTransactions then
Client.Commit;
end;
EndTime := GetTickCount;
Result := cRecordCount /((endtime - starttime)/1000);
end;No Batch, No Transactions: 600 obj/sec
Batch, No Transactions: 706 obj/sec
Batch, Transactions: 681 obj/sec
No Batch, Transactions: 624 obj/sec
As to your other questions and suggestions: I will be very happy to provide you with the answers and results, but as we are only investigating the possibilities of mORMot (we haven't reached a conclusion yet), I cannot invest to much time in finding out how these can be realized ;-). If you provide me with the code or hints on how to do those things, I'll execute them and post the result in here.
Ok, added the Batch and Transaction parts to the code:
procedure TForm1.btWrite1MilionClick(Sender: TObject);
var
i: Integer;
Starttime, Endtime: cardinal;
Rec: TSQLSampleRecord;
Ida: TIntegerDynArray;
begin
Rec := TSQLSampleRecord.Create;
Rec.Question := QuestionMemo.Text;
Rec.Address_ := 'Dit is het adres laan, 23';
Rec.postalCode := '1234AA';
Rec.City := 'Nijmegen';
Client.BatchStart(TSQLSampleRecord);
Client.TransactionBegin(TSQLSampleRecord);
StartTime := GetTickCount;
try
for i := 0 to 10000 do
begin
Rec.Name := NameEdit.Text + IntToString(i);
Client.BatchAdd(Rec,false,true);
if (i mod 1000) = 0 then
Client.BatchSend(Ida);
end;
finally
Rec.Free;
Client.BatchSend(Ida);
Client.Commit;
end;
EndTime := GetTickCount;
OutputDebugString(PChar(Format('Done. %f Objects/sec ', [ 10.000/ ((endtime - starttime)/1000) ])));
Beep;
end;The results:
using Regular Client.Add: 625 objects per sec
Using BatchAdd: 711 objects/sec
Using BatchAdd + transactions: 653 objects/sec
We're in the process of comparing different OPFs to see if and which one will be a candidate to the next generation of our application.
I've made a test project based on sample3, that creates 100000 occurences and stores them in the database as follows:
procedure TForm1.btWriteManyClick(Sender: TObject);
var
i: Integer;
Starttime, Endtime: cardinal;
Rec: TSQLSampleRecord;
begin
Rec := TSQLSampleRecord.Create;
Rec.Question := QuestionMemo.Text;
Rec.Address_ := 'Dit is het adres laan, 23';
Rec.postalCode := '1234AA';
Rec.City := 'Nijmegen';
StartTime := GetTickCount;
try
for i := 0 to 100000 do
begin
Rec.Name := NameEdit.Text + IntToString(i);
Client.Add(Rec,true);
end;
EndTime := GetTickCount;
finally
Rec.Free;
end;
OutputDebugString(PChar(Format('Elapsed: %d ms', [ endtime - starttime ])));
Beep;
end;This way we're getting around 600 object per second.
Is there a way to speed this up?
did as you asked, and with the instant client 10.2.0.5 in the exe folder, and the TNSNAMES file there to everything is working!
Looking at the registry, i see the Oracle settings in HKLM\SOFTWARE\Wow6432Node\Oracle
so i'm pretty sure its the 32bits client i have installed
Tried it on another pc with the same Oracle client installed, and get the same results
I can connect with commandline sqlplus, resulting in the following connect message:
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining optionsThe Status returned is -1
I can connect to the database with the Enterprise Manager COnsole
I have only one client installed on my PC.
I added ORACLE_HOME environment var. cause that wasnt defined, but doesnt make a difference
I checked all loaded oracle dlls, but loaded libs and dll are from the correct clientfolder ...
Is it triggered with the sample program you sent me?
Yes, same program.
It is perfectly normal that fEnv=nil the first time it reaches this part of the code.
The EnvNlsCreate() OCI call will initialize the environment and FILL fEnv parameter with the appropriate handle value.
The access violation occurs on SynDBOracle.TSQLDBOracleLib.CodePageToCharSet line 1201
function TSQLDBOracleLib.CodePageToCharSet(env: pointer;
aCodePage: integer): integer;
var ocp: PUTF8Char;
i: integer;
begin
case aCodePage of
CP_UTF8:
result := OCI_UTF8;
CP_UTF16:
result := OCI_UTF16ID;
else begin
ocp := CODEPAGES[0].Text; // default is MS Windows Code Page 1252
for i := 0 to high(CODEPAGES) do
if aCodePage=CODEPAGES[i].Num then begin
ocp := CODEPAGES[i].Text;
break;
end;
result := NlsCharSetNameToID(env,ocp); // <==== ACCESS VIOLATION HERE!!!!
if result=0 then
result := OCI_WE8MSWIN1252;
end;
end;
end;The call stack directly after the access violation is: (notice the call to CodePageToCharSet having the first parameter NIL)
:61d41761 ; D:\oracle\product\10.2.0\client_1\bin\OraClient10.Dll
:61c28fee OraClient10.OCINlsCharSetNameToId + 0xe
:1000e8e0 oci.OCINlsCharSetNameToId + 0x18
SynDBOracle.TSQLDBOracleLib.CodePageToCharSet(nil,1252)
SynDBOracle.TSQLDBOracleConnection.Connect
SynDB.TSQLDBStatement.Prepare('select sysdate from dual',True)
SynDBOracle.TSQLDBOracleStatement.Prepare('select sysdate from dual',True)
SynDB.TSQLDBStatement.Execute('select sysdate from dual',True,(...))
SynDB.TSQLDBConnectionProperties.Execute('select sysdate from dual',(...),nil)
SynDB.TSQLDBConnection.GetServerTimeStamp
SQLite3DB.TSQLRestServerStaticExternal.Create(TSQLSampleRecord,$1FBFBA0,'D:\Sources\Research\mORMot\SQLite3\Samples\03a - Client-Server Oracle\SampleRecord.external',False)
SQLite3Commons.TSQLVirtualTable.Create($1F71350,'SampleRecord',3,$208ABE4)
SQLite3.vt_Create(33589048,$1F71350,6,$208ABD8,$20437C8,nil {#0})
SynSQLite3.sqlite3_create_module_v2(33589048,'ˆ7'#4#2'ÿÿÿÿ',$1FB74A8,$5B9520,$1F91CFC)
:0059865e sqlite3_create_module_v2 + $5BE
SynSQLite3.sqlite3_create_module_v2(33589048,nil {#0},$20437A8,$1F91CFC,$2008738)
:005989fb sqlite3_create_module_v2 + $95B
SynSQLite3.sqlite3_value_numeric_type(33103048)
:00578e6a sqlite3_value_numeric_type + $5CE6
SynSQLite3.sqlite3_result_value(33103048,0)
:00572042 sqlite3_result_value + $1E2
SynSQLite3.sqlite3_step(33103048)
:0057215b sqlite3_step + $4F
SynSQLite3.TSQLRequest.Step
SynSQLite3.TSQLRequest.Execute
SynSQLite3.TSQLRequest.Execute(33589048,'CREATE VIRTUAL TABLE SampleRecord USING External(Time INTEGER, Name TEXT COLLATE SYSTEMNOCASE, Question TEXT COLLATE SYSTEMNOCASE);')
SynSQLite3.TSQLDatabase.Execute('CREATE VIRTUAL TABLE SampleRecord USING External(Time INTEGER, Name TEXT COLLATE SYSTEMNOCASE, Question TEXT COLLATE SYSTEMNOCASE);')
SQLite3.TSQLRestServerDB.CreateMissingTables(0)
ufrmServer.TfrmServer.FormCreate($1FED0C0)
:00501c4b TCustomForm.DoCreate + $37
:0050188f TCustomForm.AfterConstruction + $17
:0050185e TCustomForm.Create + $1A6
:0050c289 TApplication.CreateForm + $79
Project03aServer.Project03aServer
:7605339a kernel32.BaseThreadInitThunk + 0x12
:77299ef2 ntdll.RtlInitializeExceptionChain + 0x63
:77299ec5 ntdll.RtlInitializeExceptionChain + 0x36Which version of the OCI do you use? Instant Client or Full Client? It needs to support OCI_UTF8 encoding, which should be always the case.
I have oracle client 10.2.0 installed, actually don't really know if its InstantClient or FullClient .. I'm not such a Oracle expert
If you need any more info, feel free to ask. I really like the framework you are setting up here, but for us to use it, Oracle is a MustHave
PS Is there a way to get notified by mail on new entriues of this forum thread? I cannot seem to find a Notify button