#51 Re: mORMot 1 » AdaptSQLLimitForEngineList() not called when using Client.SetUser » 2012-09-10 13:54:11

As you can see in the Callstack in the initial post, SQLite3Commons.TSQLRestServer.URI() is called, but before it reaches the call to AdaptSQLLimitForEngineList on line 18316 you see a call to SQLite3Commons.TSQLRestServer.LaunchCallBack($18F464,'',400) happening ...

Stepping through the code, the call to LaunchCallBack is made when executing line 18289

      if (Method<>mLOCK) and
          // GET ModelRoot/MethodName + parameters sent in URI
         ((URI.MethodIndex<0) or not LaunchCallBack(URI,Resp,result.Lo)) then

#52 Re: mORMot 1 » AdaptSQLLimitForEngineList() not called when using Client.SetUser » 2012-09-10 12:44:02

But that one isnt called either ...

I have my own implementation in the TSQLDBNexusDBConnectionProperties class, so  I don't have to change any code that is directly delivered by you. This means i have no merging to do when you publish a new version

#53 Re: mORMot 1 » Error when creating AuthGroup table in Oracle 11g » 2012-09-10 12:42:55

You are right: those standards tables were not meant to be defined as external.

If they are not meant to be defined as external, I'm not really sure how you see the use of UserAuthentication with these tables when using an external database.

#54 Re: mORMot 1 » Suggestion to add to TSQLModel » 2012-09-10 12:17:57

Even though i don't like public functions, I understand you sticking to SOLID principles ;-)

#55 mORMot 1 » AdaptSQLLimitForEngineList() not called when using Client.SetUser » 2012-09-10 12:16:23

Bascy
Replies: 5

I'm trying to use the Authentication build into mormot, with the AuthGroup and AuthUser tables external in  my own database ..
But calling SetUser results in an error because the LIMIT 1 addition is not corrected by a call to my own implementation of AdaptSQLLimitForEngineList() ..

See callstack below:

SQLite3DB.TSQLRestServerStaticExternal.ExecuteInlined('SELECT RowID,LogonName,DisplayName,PasswordHashHexa,GroupRights FROM AuthUser WHERE LogonName=:(''Guest''): LIMIT 1;',True)
SQLite3DB.TSQLRestServerStaticExternal.EngineList('SELECT RowID,LogonName,DisplayName,PasswordHashHexa,GroupRights FROM AuthUser WHERE LogonName=:(''Guest''): LIMIT 1;',False,nil)
SQLite3Commons.TSQLRestServer.InternalListJSON(TSQLAuthUser,'SELECT RowID,LogonName,DisplayName,PasswordHashHexa,GroupRights FROM AuthUser WHERE LogonName=:(''Guest''): LIMIT 1;')
SQLite3Commons.TSQLRest.Retrieve('LogonName=:(''Guest''):',$2643290)
SQLite3Commons.TSQLRecord.Create($2596260,'LogonName=?',(...))
SQLite3Commons.TSQLRestServer.Auth($18F464)
SQLite3Commons.TSQLRestServer.LaunchCallBack($18F464,'',400)
SQLite3Commons.TSQLRestServer.URI('root/auth?UserName=Guest&Password=2e4a799a589f4381206254b31517a86d5274bb3d39cd60f72c906d90a2bddbdc&ClientNonce=999a42861b620c0ee361d432c044b5bac72d39f6f89474f962a31fc99b3d8077','GET','','','',$99564D)
SQLite3.TSQLRestClientDB.InternalURI('root/auth?UserName=Guest&Password=2e4a799a589f4381206254b31517a86d5274bb3d39cd60f72c906d90a2bddbdc&ClientNonce=999a42861b620c0ee361d432c044b5bac72d39f6f89474f962a31fc99b3d8077','GET',$18F674 {''},$18F578 {''},$18F574 {''})
SQLite3Commons.TSQLRestClientURI.URI('root/auth?UserName=Guest&Password=2e4a799a589f4381206254b31517a86d5274bb3d39cd60f72c906d90a2bddbdc&ClientNonce=999a42861b620c0ee361d432c044b5bac72d39f6f89474f962a31fc99b3d8077','GET',$18F674 {''},nil {''},nil {''})
SQLite3Commons.TSQLRestClientURI.CallBackGet('auth',(...),'',nil,0,nil {''})
SQLite3Commons.TSQLRestClientURI.CallBackGetResult('auth',(...),nil,0)
SQLite3Commons.TSQLRestClientURI.SetUser('Guest','synopse',False)

#56 mORMot 1 » Error when creating AuthGroup table in Oracle 11g » 2012-09-10 11:09:35

Bascy
Replies: 4

When i try to create an External AuthGroup table in oracle, the following error occurs:

ORA-02329: column of datatype LOB cannot be unique or primary key

The create statement constructed is as follows:

SQLite3DB.TSQLRestServerStaticExternal.ExecuteDirect('CREATE TABLE AuthGroup (ID  NUMBER(22,0) PRIMARY KEY, Ident NCLOB NOT NULL UNIQUE,SessionTimeout NUMBER(22,0),AccessRights NCLOB)',(...),(...),False)

#57 mORMot 1 » Suggestion to add to TSQLModel » 2012-09-10 10:05:30

Bascy
Replies: 2

Why not add a generic method to TSQLModel to register all tables of the model externally to a TSQLConnectionPropertes?

procedure TSQLModel.RegisterAllTablesExternal(aProps: TSQLDBConnectionProperties);
var
  i: integer;
begin
  for i := Low(Tables) to High(Tables) do
    VirtualTableExternalRegister(Self, Table[Tables[i].SQLTableName], aProps, Tables[i].SQLTableName);
end;

#59 Re: mORMot 1 » Opening a remote database through Client-server services » 2012-09-07 10:07:45

But when creating the TSQLRestServer i have set HandleUserauthentication to false ...

#60 Re: mORMot 1 » Opening a remote database through Client-server services » 2012-09-07 09:35:51

You mean running  THttpApiServer.AddUrlAuthorize() ?
Done that

#61 mORMot 1 » Opening a remote database through Client-server services » 2012-09-07 08:37:34

Bascy
Replies: 6

I'm trying to accomplish the following setup:

I have a "generic" server process "LCSRestServer" running, implementing a very simple interface with a TSQLRestServerDB instance. This server uses its own very simple TLCSServerModel = class(TSQLModel).

type
  ILCS = interface(IInvokable)
    ['{31969E17-758B-4077-B0E8-77B81B082715}']
    {1 open the database as specified in the connectionstring, return if this was succesfull
        - aConnectionstring contains all info needed to connect using <protocol>://server/name syntax
        - if aCreate is true than create the database if it doent exist
    }
    function OpenDatabase(aConnectionString: RawUTF8; var URI: RawUTF8; aCreate: boolean = false): boolean;
  end;

The client process can now tell the server to open the specified database, making it available to the client. This database is implementing a different TSQLModel ("TLCSModel"), the RestServer must be implementing a different interface "ILCSDatabase"

The implementation of the OpenDatabase method being:

function TLCSServer.OpenDatabase(aConnectionString: RawUTF8; var URI: RawUTF8; aCreate: boolean = false): boolean;
var
  ...
begin
    //Determine database type and protocol
    lConProp := CreateConnectionProperties(aConnectionString);

    if lConProp <> nil then
    begin
      //Create separate TSQLRestSeverDB object for this database

      lModel := TLCSModel.Create(URI);
      lModel.RegisterVirtualTables(lConProp);

      lRestServerDB := TLCSRestDB.Create(lModel, UTF8ToString(URI) + '.db3');
      lRestServerDB.CreateMissingTables;

      lRestServerDB.ServiceRegister(TLCSDatabase , [TypeInfo(ILCSDatabase)], sicShared);

      //Register the new restServer with the http server of the main server process
      LCSRestServer.RegisterDatabase(lRestServerDB);

      Result := true;
    end;
  end;

Somehow i'm not getting this whole setup working... The above method does work, but when i try to run the following method on the client side, it says it can't find LCSDatabase as a service on the server ...

  FModel := TLCSModel.Create(cURI);
  FClient := TSQLite3HttpClient.Create('localhost','888',FModel);
  FClient.ServiceRegister([TypeInfo(ILCS)],sicShared);

  FClient.Services[cURI].Get(FLcsInterface);

  FLcsInterface.OpenDatabase(cNewFolder, lDBUri, True);

  FClient.ServiceRegister([TypeInfo(ILCSDatabase)],sicShared);  // <<==Error:  "LCSDatabase" interface or REST routing not supported by server
  FClient.Services[lDBUri].Get(lDbInterface);

What am i doing wrong?

#62 Re: mORMot 1 » Creating a database » 2012-09-07 06:48:18

OK, i'll add my own implementation of CreateDatabase to my NexusDB driver.

Maybe its an idea to add some kind of "CanCreate():boolean" function to the main TSQLDBCOnnectionProperties, or add interfaces to let the code determine if its possible to create a database, independent of the actual TConnectionproperties implementation..

type
  ICanCreateDatabase = interface(IInterface)
  ['{C6A016A5-2A6E-48F6-9A4D-310B643B53C0}']
    function CanCreate: boolean;
    function DatabaseExists: boolean;
    function CreateDatabase: boolean;
  end;

#63 mORMot 1 » Creating a database » 2012-09-05 14:00:34

Bascy
Replies: 3

Maybe I am not looking straight ... but I can't seem to find any possibility to create a new database through a TSQLDBconnection or TSQLDBConnectionproperties...
Any plans on this?

#64 mORMot 1 » Generating mORMot PDF documentation » 2012-09-05 07:05:34

Bascy
Replies: 2

Simple question:

Is it possible to generate the mORMot PDF files (esp the SAD) by myself?

#65 mORMot 1 » "Unknown type 101" error when declaring a double property with Oracle » 2012-08-31 13:47:39

Bascy
Replies: 1

I have declared the following class:

  TSQLUnitTestSampleRecord = class(TSQLRecord)
  private
    fQuestion: string;
    fName: string;
    fTime: TModTime;
    FAddress: string;
    fPostal: string;
    FCity: string;
    FDatum: TDatetime;
    FFloatNumber: double;
    FIntNumber: Int64;
    FCurNumber: Currency;
  published
    property ModTime: TModTime read fTime write fTime;
    property Name: string index 50 read fName write fName;
    property Question: string index 50 read fQuestion write fQuestion;
    property Address_: string index 50 read FAddress write fAddress;
    property PostalCode: string index 12 read fPostal write fPostal;
    property City: string index 50 read FCity write FCity;
    property Datum: TDatetime read FDatum write FDatum;
    property FloatNumber: double read FFloatNumber write FFloatNumber;
    property IntNumber: Int64 read FIntNumber write FIntNumber;
    property CurrencyNumber: Currency read FCurNumber write FCurNumber;
  end;

When i use this class in an oracle 11g environment, executing CresateMissingTables results in the following error:

ESQLDBOracle exception, with message 'Column "FLOATNUMBER": unknown type 101'

If i change the DB_FIELDS definition for Oracle to contain FLOAT in stead of BINARY_DOUBLE everything works find:

const
  /// the known column data types corresponding to our TSQLDBFieldType types
  // - will be used e.g. for TSQLDBConnectionProperties.SQLFieldCreate()
  DB_FIELDS: array[TSQLDBDefinition] of TSQLDBFieldTypeDefinition = (
  // ftNull, ftInt64, ftDouble, ftCurrency, ftDate, ftUTF8, ftBlob
  (' NVARCHAR(%)',' BIGINT',' DOUBLE',' NUMERIC(19,4)',' TIMESTAMP',' CLOB',' BLOB'),
  (' NVARCHAR(%)',' BIGINT',' DOUBLE',' NUMERIC(19,4)',' TIMESTAMP',' CLOB',' BLOB'),
  (' NVARCHAR2(%)',' NUMBER(22,0)',' FLOAT',' NUMBER(19,4)',' DATE',' NCLOB',' BLOB'),  //Oracle , changed BINARY_DOUBLE into FLOAT

#66 Re: mORMot 1 » Using '?' parameters in CreateAndFillPrepare() going wrong » 2012-08-31 06:20:28

Hi Ab, thanks for your replies!
I really appreciate the fact that you are always open to suggestions from users, not afraid to acknowledge that somethings things could be made clearer or more convenient from a user point of view. Makes for a very serious user-participation feeling

Thanks

#67 Re: mORMot 1 » Using '?' parameters in CreateAndFillPrepare() going wrong » 2012-08-30 15:10:04

Ok, fog is clearing up a bit now
What i also would expect is a CreateFillAndPrepare that will only substitute '?' with bound parameters

#68 mORMot 1 » Using '?' parameters in CreateAndFillPrepare() going wrong » 2012-08-30 09:48:41

Bascy
Replies: 5

Using an Oracle database, somethings strange happens when i execute the following code:

  // Select a single record by Name
  lName := Format(cPPName, [cResultID]);
  Rec := TSQLUnitTestSampleRecord.CreateAndFillPrepare(Client,'Name=?', [lname], '');

Executing this results in an error stating: Executeprepared expected 1 bound parameters, got 0

Looking at the callstack, it seems that the supplied parameter is not formatted into the SQLwhere clause by the FormatUTF8(FormatSQLWhere,ParamsSQLWhere) call

SQLite3Commons.TSQLRest.InternalListRecordsJSON(TSQLUnitTestSampleRecord,'Name=?')
SQLite3Commons.TSQLRecord.CreateAndFillPrepare($2470890,'Name=?','')
SQLite3Commons.TSQLRecord.CreateAndFillPrepare($2470890,'Name=?',(...),'')
uTestMormotDBBaseCommon.TMormotDBBaseCommonSelectTest.TestSelectSingleRecordByString

[Edit]
Extra info: If i add another parameter to the CreateAndFillPrepare, it works

  Rec := TSQLUnitTestSampleRecord.CreateAndFillPrepare(Client,'Name=?', [], [lname], '');

I see now that two different const arrays should be supplied to substitute the % and the ? parameters separately .. mmmmm :-(
IMHO its a very confusing construction

#69 Re: mORMot 1 » Integer overflow in Iso8601ToDateTime » 2012-08-28 09:59:30

This test is done against a oracle database, the empty datetime returned as '0000-01-01T00:00:00' is from the Oracle driver ....

It turns out the field was empty for another reason (on creating the record we used BatchAdd(Rec, false, true) in stead of BatchAdd(Rec, True, True))
This created empty records in the oracle table, and when they were read back, the TDatetime field was empty too of course

#70 mORMot 1 » Integer overflow in Iso8601ToDateTime » 2012-08-27 12:47:41

Bascy
Replies: 3

While running our own unittests on an oracle database, we get an Integer Overflow error while deleting records using batch mode

Call stack:

SynCommons.Div100(???)
SynCommons.Iso8601ToDateTimePUTF8CharVar('00-01-01T00:00:00',17,0)
SQLite3Commons.TPropInfo.SetValue($25032E0,'0000-01-01T00:00:00')
SQLite3Commons.TSQLRecordFill.Fill($7C570038)
SQLite3Commons.TSQLRecord.FillRow(1,nil)
SQLite3Commons.TSQLRecord.FillOne
uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.DeleteModelData(True,True)
uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.TestTimedCRUDBatchedTransaction

Code:

procedure TMormotDBBaseCommonCRUDTest.DeleteModelData(aUseBatch, aUseTransactions: boolean);
VAR cnt:integer;
  Starttime, Endtime: cardinal;
  Rec: TSQLUnitTestSampleRecord;
  Ida: TIntegerDynArray;
begin
  OutputDebugString('---DELETE TEST---');
  cnt:=0;
  if aUseBatch then
    Client.BatchStart(TSQLUnitTestSampleRecord);
  if aUseTransactions then
    Client.TransactionBegin(TSQLUnitTestSampleRecord);

  StartTime:=GetTickCount;
  try
    Rec := TSQLUnitTestSampleRecord.CreateAndFillPrepare(Client,'');
    EndTime:=GetTickCount;
    OutputDebugString(PChar(Format('Query took %.0n ms',[1.0*(EndTime-StartTime)])));
    StartTime:=GetTickCount;
    try
      while Rec.FillOne do
      begin
        if aUseBatch then
        begin
          Client.BatchDelete(Rec.ID);
          if (cnt mod cBatchSize) = 0 then
          begin
            Client.BatchSend(Ida);
            Client.BatchStart(TSQLUnitTestSampleRecord);
          end;
        end
        else Client.Delete(TSQLUnitTestSampleRecord,Rec.ID);

        if aUseTransactions and (cnt mod cCommitSize = 0) then
        begin
          Client.Commit;
          Client.TransactionBegin(TSQLUnitTestSampleRecord);
        end;
        inc(cnt);
      end;
    finally
      Rec.Free;
    end;
    if aUseBatch then
      Client.BatchSend(Ida);
    if aUseTransactions then
      Client.Commit;
  except
    if aUseBatch then
      Client.BatchAbort;
    if aUseTransactions then
      Client.RollBack;
    raise;
  end;
  EndTime:=GetTickCount;
  OutputDebugString(PChar(Format('%.0n records deleted, avg speed %.0n rps Batch=%s Transaction=%s',[1.0*cnt,1000.0*cnt/max(1,(EndTime-StartTime)),BoolToStr(aUseBatch),BoolToStr(aUseTransactions)])));
end;

#71 mORMot 1 » Query selection on a date » 2012-08-27 09:50:06

Bascy
Replies: 2

I'm trying to create a unit test where selection of a specific record will be done on a TDatetime field.

  // Create 10 records
  CreateSmallModelData;

  // Select a single record by Date
  Rec := TSQLUnitTestSampleRecord.CreateAndFillPrepare(Client,'Datum=?', [], [DateToIso8601Text( EncodeDate(2012, 5, 4))]);
  Rec.FillOne;
  CheckEqualsString(Format(cPPName, [4]), Rec.Name);

When i run this against an oracle database, the SQL statement that is eventually executed in the TSQLRestServerStaticExternal.ExecuteInlined method is

SELECT ID,ModTime,Name,Question,Address_,PostalCode,City,Datum FROM UnitTestSampleRecord WHERE Datum="2012-05-04'';

When i run this against the nexusDB database (for which we are coding a native driver for mORMot) then the executed method is:

SELECT ID,ModTime,Name,Question,Address_,PostalCode,City,Datum FROM UnitTestSampleRecord WHERE Datum=''2012-05-04'';

Exactly the same .. Both oracle and Nexus have their own ISO8601 implementation but i dont see any of those methods used when executing a SQL statement via
the ExecuteInline way ... When Nexus tries to execute the SQL it results in an error, because a date literal should always be preceded with the keyword "DATE"


How should we solve this?

#72 mORMot 1 » Cannot compile 2012-08-21 version » 2012-08-22 07:40:16

Bascy
Replies: 1

Getting an error compiling SynDB, because the protected keyword cannot be used in a record structure.

so, this

  TQueryValue = {$ifndef UNICODE}object{$else}record{$endif}
  protected
    /// fName should be the first property, i.e. the searched hashed value
    fName: string;
    fValue: Variant;
   ....

should be something like this

  TQueryValue = {$ifndef UNICODE}object{$else}record{$endif}
  {$ifndef UNICODE}protected{$endif}
    /// fName should be the first property, i.e. the searched hashed value
    fName: string;
    fValue: Variant;
    ....

#73 Re: mORMot 1 » Alternatives for the LIMIT 1 additive » 2012-08-22 07:34:23

Ok, thanks for the adjustments.

Is it possible to add my own native database implementation, without having to make major alterations to your sources?
With the current setup I have to add dNexusDB to the TSQLDBDefinition. But when i do that, i also have to alter all the const arrays containing database specific implementations ..
So i was wondering if that was the way to go ..

#74 Re: mORMot 1 » Alternatives for the LIMIT 1 additive » 2012-08-21 08:58:14

Any news on this issue? Or any idea when you will be able to implement this?

#75 Re: mORMot 1 » Alternatives for the LIMIT 1 additive » 2012-08-16 12:06:17

In my current implementation of NexusDB i use the original TQuery.Params[] created when assigning the SQL statement to the TQuery.SQL.Text property
The binding is done with BindTextU and this assigns the Iso8601 string to the param value .AsString
As far as i know the TParam object does not know what kind of datatype is expected ... so i cannot make a special case when a date is given

#76 Re: mORMot 1 » Alternatives for the LIMIT 1 additive » 2012-08-16 11:32:51

The DATE keyword is mandatory for NexusDB, so I guess there will be another constant array referring to NexusDBIso8601ToDate methods like the already present OracleSQLIso8601ToDate

#77 Re: mORMot 1 » Alternatives for the LIMIT 1 additive » 2012-08-16 10:19:07

Another difference: date literals in parameters, for NexusDB they need to be formatted like DATE '2012-01-29'

#78 Re: mORMot 1 » Alternatives for the LIMIT 1 additive » 2012-08-15 11:53:52

How can i determine the database implementation used in i.e. TSQLRest?

#79 mORMot 1 » Alternatives for the LIMIT 1 additive » 2012-08-15 10:43:37

Bascy
Replies: 14

Busy implementing a native NexusDB implementation, i ran into a difference of SQL implementation.
While SQLite is using LIMIT 1 to only retrieve one record with a SQL statement, NexusDB uses TOP 1

How can i best implement this?
SHould we add LIMIT 1 to the implementation specific constants as DB_FIELDS[] and DB_SERVERTIME[] ?

Edit:
The place of TOP 1 in the SELECT statement is also different form the LIMIT 1
<TOP n> should be directly behind the SELECT keyword

#80 Re: mORMot 1 » Isn't there a Step(TRUE) needed in TSQLDBStatement.FetchAllToJSON()? » 2012-08-15 08:22:06

Ok, your first line had me convinced LOL: "Step" is not "Next"

But what confused me was the word "Next"in the first line in the documentation of the Step() method:

    {/ Access the next or first row of data from the SQL Statement result
     - return true on success, with data ready to be retrieved by Column*()
     - return false if no more row is available (e.g. if the SQL statement
      is not a SELECT but an UPDATE or INSERT command)
     - if SeekFirst is TRUE, will put the cursor on the first row of results
     - should raise an Exception on any error }
    function Step(SeekFirst: boolean=false): boolean; virtual; abstract;

#81 Re: mORMot 1 » Isn't there a Step(TRUE) needed in TSQLDBStatement.FetchAllToJSON()? » 2012-08-15 07:45:47

I'm still not convinced ;-)

What will be the result of the following code:

var Customer: Variant;
begin
  with Props.Execute( 'select * from Sales.Customer where AccountNumber like ?', ['AW000001%'],@Customer) do
      assert(Copy(Customer.AccountNumber,1,8)='AW000001');
end;

Or in a question: Isn't the first record the active record after running Props.Execute() ?

#82 mORMot 1 » Isn't there a Step(TRUE) needed in TSQLDBStatement.FetchAllToJSON()? » 2012-08-15 06:52:53

Bascy
Replies: 5

While trying to code a native NexusDB driver something struck me in the base code of TSQLDBStatement.
The FetchAllToJSON method starts the data-processing loop with a call to Step. If I would follow this in my NexusDB implementation (or rather if i did not override it ;-)) then the first record would not be processed, as this is the current record when entering the FetchAllToJSON method.

I would expect the data processing loop to look something like this

while Step(result=0) do 
begin
      ColumnsToJSON(W);
      W.Add(',');
      inc(result);
end;

#83 Re: mORMot 1 » How to start creating native NexusDB support » 2012-08-01 06:18:53

Thanks for your suggestions, they are really usefull!

The reason we are looking for this extra implementation is that the current version of our product make heavy use of NexusDB. We are in the start-up of developing a major overhaul of our application, and of course cannot ignore our installed base, even it is only necessary to convert to a better solution ;-)

We'll will keep you posted on our progress, and of course will supply you with the resulting unit

#84 mORMot 1 » How to start creating native NexusDB support » 2012-07-31 06:43:31

Bascy
Replies: 2

We would like to have native support in mORMot for NexusDB database servers, supporting both the embedded server possibilities of Nexusdb and the remote server possibilities.
Performance is a very big issue, so we don't want to introduce any extra layers by using OLEDB.

How do we start creating this SynDBNexusDB.pas? (or maybe somebody already has started work in this ;-)?)
What kind of already existing unit-tests can we use to determine if our implementation is complete and in working order?

#85 Re: mORMot 1 » Cannot run Perftest on our own Oracle Database » 2012-07-30 10:10:48

ab wrote:

I'll find out how to circumvent this issue. Certainly by setting OCI_ATTR_DATEFORMAT for the column.

Can i set this attribute by hand somehow? maybe with SQLPLUS?

Hahaha you have fixed the problem faster than i can think of a workaround,
thanks

#86 Re: mORMot 1 » Cannot run Perftest on our own Oracle Database » 2012-07-30 09:09:55

Ok, solved this one myself ;-)
Turns out the Perftest uses a SampleRecord table, and that one already existed in our database LOL

Next error:

ORA-01861: Literal does not match format string

probably during "select max(id) from SampleRecord"

no the error occurs during the following statement:

SQLite3DB.TSQLRestServerStaticExternal.ExecuteInlined('insert into SampleRecord (ID,FirstName,LastName,Amount,BirthDate,LastChange,CreatedAt) VALUES (:(1):,:(''Namé 5''):,:(''5''):,:(0.05):,:(''1900-01-05''):,:(135052048292):,:(135052048292):)',False)

#87 mORMot 1 » Cannot run Perftest on our own Oracle Database » 2012-07-30 09:04:07

Bascy
Replies: 6

I'm trying to run the new Perftest project against our own oracle 10g installation, but getting the following error:

..raised exception class ESQLDBOracle with message 'ORA-00904: "CREATEDAT": invalid identifier

Any suggestions?

#88 Re: mORMot 1 » Highest performance when adding a lot of records to Oracle » 2012-07-12 12:49:51

Thanks for the blog-entry! It made a lot of things clearer to me.

One correction: the link at the bottom of the article to this forum thread is not correct

#89 Re: mORMot 1 » Highest performance when adding a lot of records to Oracle » 2012-07-12 07:16:42

Again, thanks for the advise!

One of the things I don't like so much about your approach to the ORM is the mis-use of existing Delphi constructs like "index n" attribute for the maximum length of a string-property. Other ORMs solve this i.e. with official Class-attributes like TMS Aurelius:

type
  [Entity]
  [Table('ARTISTS')]
  [Sequence('SEQ_ARTISTS')]
  [Id('FId', TIdGenerator.IdentityOrSequence)]
  TArtist = class
  private
    [Column('ID', [TColumnProp.Unique, TColumnProp.Required, TColumnProp.NoUpdate])]
    FId: Integer;
    FArtistName: string;
    FGenre: Nullable<string>;
    function GetArtistName: string;
    procedure SetArtistName(const Value: string);
  public
    property Id: integer read FId;

    [Column('ARTIST_NAME', [TColumnProp.Required], 100)]
    property ArtistName: string read GetArtistName write SetArtistName;

    [Column('GENRE', [], 100)]
    property Genre: Nullable<string> read FGenre write FGenre;
  end;

IMHO this approach has a few advantages:

  • Every class can be persisted this way, without any specific demands on class-hierarchy

  • Cleaner code, much better readable because there are no standard language construct that mean something else in this ORM context

What are your thoughts on this?

#90 Re: mORMot 1 » Highest performance when adding a lot of records to Oracle » 2012-07-11 14:16:46

Thanks for the info

Your answer does confuse me a bit. What do you mean by "Client-server ORM" as opposed to SOA mentioned later on?


I have read the SAD Doc, until page 186 and it does appeal to me ;-)
But now i have to convince my boss, when he gets back from holiday

#91 Re: mORMot 1 » Highest performance when adding a lot of records to Oracle » 2012-07-11 11:27:36

We are not sure yet. At the moment we are testing several ORMs learning what can be realistic performances.
At the moment we have a single client windows application, who's design principles date more than 15 years back. It uses an embedded NexusDB engine which is (of course) lightning fast. There are a lot of very data-intensive calculations needed in all kinds of analyses. The dataset can be from just a few Mb to several dozens of GB. Becuase of its design all the analyses were done on a record-by-record basis in the client (Dataset.First - .Next - Until .EOF)

For the newly designed application client-server with multiple clients (windows, HTML, Scripting, ...), SOAP or REST, multiple databases (Oracle and MSSQL required, others are optional) are all MustHave requirements. This all demands a very different approach to the new design, and our way of thinking (guess we need a bit of re-education ;-)).

The application also has to have the possibility to convert external data into its own model, and this could also be a few MB or several GB that has to be converted on a regular basis...

Another important function for us developer ( not so much from a users point of view) is the possibility to update the Model and its underlying database when the model changes with new version of the application (new or dropped classes, new or changed published properties, new indeces etc). This is something we have not found in mORMot yet (while it is present i.e. in InstantObjects. Any plans on such a thing?

#92 Re: mORMot 1 » Highest performance when adding a lot of records to Oracle » 2012-07-11 10:39:09

ab wrote:

TNSPING has a 10 ms resolution, so it is not precise enough when you want to know the latency of multiple requests.
A good old PING does make sense in such case.

Good old Ping says: 0ms average

#93 Re: mORMot 1 » Highest performance when adding a lot of records to Oracle » 2012-07-11 09:14:09

The latency as stated by TNSPING is 10-20 msec

Executed some more tests, directly executing 4 possible scenarios in one go, using the following code:

procedure TForm1.FormCreate(Sender: TObject);
begin
  Props := TSQLDBOracleConnectionProperties.Create('OPFTEST','OPF_MORMOT','secret', 871);
  Model := CreateSampleModel; // from SampleData unit
  VirtualTableExternalRegister(Model,TSQLSampleRecord,Props,'SampleRecord');
  Client := TSQLRestClientDB.Create(Model, nil, 'test.db3', TSQLRestServerDB);
  Client.Server.CreateMissingTables;
end;

function TForm1.Addmany(aUseBatch, aUseTransactions: boolean): double;
var
  i: Integer;
  Starttime, Endtime: cardinal;
  Rec: TSQLSampleRecord;
  Ida: TIntegerDynArray;

begin

  Rec := TSQLSampleRecord.Create;

  Rec.Question := QuestionMemo.Text;
  Rec.Address_ := 'Dit is het adres laan, 23';
  Rec.postalCode := '1234AA';
  Rec.City := 'Nijmegen';

  if aUseBatch then
    Client.BatchStart(TSQLSampleRecord);
  if aUseTransactions then
    Client.TransactionBegin(TSQLSampleRecord);

  StartTime := GetTickCount;
  try
    for i := 0 to cRecordCount do
    begin
      Rec.Name := NameEdit.Text + IntToString(i);
      if aUseBatch then
        begin
          Client.BatchAdd(Rec,false,true);
          if (i mod 1000) = 0 then
          begin
            Client.BatchSend(Ida);
            Client.BatchStart(TSQLSampleRecord);
          end;
        end
      else
        Client.Add(Rec,true);

    end;
  finally
    Rec.Free;
    if aUseBatch then
      Client.BatchSend(Ida);
    if aUseTransactions then
      Client.Commit;
  end;

  EndTime := GetTickCount;
  Result := cRecordCount /((endtime - starttime)/1000);
end;
  • No Batch, No Transactions: 600 obj/sec

  • Batch, No Transactions: 706 obj/sec

  • Batch, Transactions: 681 obj/sec

  • No Batch, Transactions: 624 obj/sec

As to your other questions and suggestions: I will be very happy to provide you with the answers and results, but as we are only investigating the possibilities of mORMot (we haven't reached a conclusion yet), I cannot invest to much time in finding out how these can be realized ;-). If you provide me with the code or hints on how to do those things, I'll execute them and post the result in here.

#94 Re: mORMot 1 » Highest performance when adding a lot of records to Oracle » 2012-07-09 09:43:27

Ok, added the Batch and Transaction parts to the code:

procedure TForm1.btWrite1MilionClick(Sender: TObject);
var
  i: Integer;
  Starttime, Endtime: cardinal;
  Rec: TSQLSampleRecord;
  Ida: TIntegerDynArray;
begin

  Rec := TSQLSampleRecord.Create;

  Rec.Question := QuestionMemo.Text;
  Rec.Address_ := 'Dit is het adres laan, 23';
  Rec.postalCode := '1234AA';
  Rec.City := 'Nijmegen';

  Client.BatchStart(TSQLSampleRecord);
  Client.TransactionBegin(TSQLSampleRecord);

  StartTime := GetTickCount;
  try
    for i := 0 to 10000 do
    begin
      Rec.Name := NameEdit.Text + IntToString(i);
      Client.BatchAdd(Rec,false,true);
      if (i mod 1000) = 0 then
        Client.BatchSend(Ida);
    end;
  finally
    Rec.Free;
    Client.BatchSend(Ida);
    Client.Commit;
  end;

  EndTime := GetTickCount;
  OutputDebugString(PChar(Format('Done. %f Objects/sec ', [ 10.000/ ((endtime - starttime)/1000) ])));
  Beep;
end;

The results:

using Regular Client.Add: 625 objects per sec
Using BatchAdd: 711 objects/sec
Using BatchAdd + transactions: 653 objects/sec

#95 mORMot 1 » Highest performance when adding a lot of records to Oracle » 2012-07-06 13:19:21

Bascy
Replies: 44

We're in the process of comparing different OPFs to see if and which one will be a candidate to the next generation of our application.

I've made a test project based on sample3, that creates 100000 occurences and stores them in the database as follows:

procedure TForm1.btWriteManyClick(Sender: TObject);
var
  i: Integer;
  Starttime, Endtime: cardinal;
  Rec: TSQLSampleRecord;
begin

  Rec := TSQLSampleRecord.Create;

  Rec.Question := QuestionMemo.Text;
  Rec.Address_ := 'Dit is het adres laan, 23';
  Rec.postalCode := '1234AA';
  Rec.City := 'Nijmegen';

  StartTime := GetTickCount;
  try
    for i := 0 to 100000 do
    begin
      Rec.Name := NameEdit.Text + IntToString(i);
      Client.Add(Rec,true);
    end;
    EndTime := GetTickCount;
  finally
    Rec.Free;
  end;

  OutputDebugString(PChar(Format('Elapsed: %d ms', [ endtime - starttime ])));
  Beep;
end;

This way we're getting around 600 object per second.

Is there a way to speed this up?

#96 Re: mORMot 1 » Getting started with Oracle OCI » 2012-07-06 12:19:58

did as you asked, and with the instant client 10.2.0.5 in the exe folder, and the TNSNAMES file there to everything is working!

#97 Re: mORMot 1 » Getting started with Oracle OCI » 2012-07-06 09:01:45

Looking at the registry, i see the Oracle settings in HKLM\SOFTWARE\Wow6432Node\Oracle
so i'm pretty sure its the 32bits client i have installed

#98 Re: mORMot 1 » Getting started with Oracle OCI » 2012-07-06 06:57:46

Tried it on another pc with the same Oracle client installed, and get the same results
I can connect with commandline sqlplus, resulting in the following connect message:

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

#99 Re: mORMot 1 » Getting started with Oracle OCI » 2012-07-05 17:54:23

The Status returned is -1
I can connect to the database with the Enterprise Manager COnsole
I have only one client installed on my PC.

I added ORACLE_HOME environment var. cause that wasnt defined, but doesnt make a difference
I checked all loaded oracle dlls, but loaded libs and dll are from the correct clientfolder ...

#100 Re: mORMot 1 » Getting started with Oracle OCI » 2012-07-05 12:14:20

ab wrote:

Is it triggered with the sample program you sent me?

Yes, same program.

ab wrote:

It is perfectly normal that fEnv=nil the first time it reaches this part of the code.
The EnvNlsCreate() OCI call will initialize the environment and FILL fEnv parameter with the appropriate handle value.

The access violation occurs on SynDBOracle.TSQLDBOracleLib.CodePageToCharSet line 1201

function TSQLDBOracleLib.CodePageToCharSet(env: pointer;
  aCodePage: integer): integer;
var ocp: PUTF8Char;
    i: integer;
begin
  case aCodePage of
  CP_UTF8:
    result := OCI_UTF8;
  CP_UTF16:
    result := OCI_UTF16ID;
  else begin
    ocp := CODEPAGES[0].Text; // default is MS Windows Code Page 1252
    for i := 0 to high(CODEPAGES) do
      if aCodePage=CODEPAGES[i].Num then begin
        ocp := CODEPAGES[i].Text;
        break;
      end;
    result := NlsCharSetNameToID(env,ocp); // <==== ACCESS VIOLATION HERE!!!!
    if result=0 then
      result := OCI_WE8MSWIN1252;
  end;
  end;
end;

The call stack directly after the access violation is: (notice the call to CodePageToCharSet having  the first parameter NIL)

:61d41761 ; D:\oracle\product\10.2.0\client_1\bin\OraClient10.Dll
:61c28fee OraClient10.OCINlsCharSetNameToId + 0xe
:1000e8e0 oci.OCINlsCharSetNameToId + 0x18
SynDBOracle.TSQLDBOracleLib.CodePageToCharSet(nil,1252)
SynDBOracle.TSQLDBOracleConnection.Connect
SynDB.TSQLDBStatement.Prepare('select sysdate from dual',True)
SynDBOracle.TSQLDBOracleStatement.Prepare('select sysdate from dual',True)
SynDB.TSQLDBStatement.Execute('select sysdate from dual',True,(...))
SynDB.TSQLDBConnectionProperties.Execute('select sysdate from dual',(...),nil)
SynDB.TSQLDBConnection.GetServerTimeStamp
SQLite3DB.TSQLRestServerStaticExternal.Create(TSQLSampleRecord,$1FBFBA0,'D:\Sources\Research\mORMot\SQLite3\Samples\03a - Client-Server Oracle\SampleRecord.external',False)
SQLite3Commons.TSQLVirtualTable.Create($1F71350,'SampleRecord',3,$208ABE4)
SQLite3.vt_Create(33589048,$1F71350,6,$208ABD8,$20437C8,nil {#0})
SynSQLite3.sqlite3_create_module_v2(33589048,'ˆ7'#4#2'ÿÿÿÿ',$1FB74A8,$5B9520,$1F91CFC)
:0059865e sqlite3_create_module_v2 + $5BE
SynSQLite3.sqlite3_create_module_v2(33589048,nil {#0},$20437A8,$1F91CFC,$2008738)
:005989fb sqlite3_create_module_v2 + $95B
SynSQLite3.sqlite3_value_numeric_type(33103048)
:00578e6a sqlite3_value_numeric_type + $5CE6
SynSQLite3.sqlite3_result_value(33103048,0)
:00572042 sqlite3_result_value + $1E2
SynSQLite3.sqlite3_step(33103048)
:0057215b sqlite3_step + $4F
SynSQLite3.TSQLRequest.Step
SynSQLite3.TSQLRequest.Execute
SynSQLite3.TSQLRequest.Execute(33589048,'CREATE VIRTUAL TABLE SampleRecord USING External(Time INTEGER, Name TEXT COLLATE SYSTEMNOCASE, Question TEXT COLLATE SYSTEMNOCASE);')
SynSQLite3.TSQLDatabase.Execute('CREATE VIRTUAL TABLE SampleRecord USING External(Time INTEGER, Name TEXT COLLATE SYSTEMNOCASE, Question TEXT COLLATE SYSTEMNOCASE);')
SQLite3.TSQLRestServerDB.CreateMissingTables(0)
ufrmServer.TfrmServer.FormCreate($1FED0C0)
:00501c4b TCustomForm.DoCreate + $37
:0050188f TCustomForm.AfterConstruction + $17
:0050185e TCustomForm.Create + $1A6
:0050c289 TApplication.CreateForm + $79
Project03aServer.Project03aServer
:7605339a kernel32.BaseThreadInitThunk + 0x12
:77299ef2 ntdll.RtlInitializeExceptionChain + 0x63
:77299ec5 ntdll.RtlInitializeExceptionChain + 0x36

Which version of the OCI do you use? Instant Client or Full Client? It needs to support OCI_UTF8 encoding, which should be always the case.

I have oracle client 10.2.0 installed, actually don't really know if its InstantClient or FullClient .. I'm not such a Oracle expert

If you need any more info, feel free to ask. I really like the framework you are setting up here, but for us to use it, Oracle is a MustHave

PS Is there a way to get notified by mail on new entriues of this forum thread? I cannot seem to find a Notify button

Board footer

Powered by FluxBB