#1 Re: mORMot 1 » What is the difference between TSQLRestClientDB, and its server? » 2021-04-14 14:08:12

Dear @wxinix,

You may have issues if you intend to cache external database tables.
Please check this post for ab's clarifications regarding caching.

Regards.

#3 Re: mORMot 1 » External Zeos MySQL tables with underscore ( _ ) cannot be registered. » 2021-04-12 15:39:56

Hi flydev, EgonHugeist

I can not thank you enough for finding the source of this bug.

As flydev explained, the bug is due to this code in ZDbcMetadata.pas in Zeoslib:

{**
  Set the Wildcards character for WildcardsArray variable.
  Overrride this method if the wildcards character is different in other database
}
procedure TZAbstractDatabaseMetadata.FillWildcards;
begin
  SetLength(WildcardsArray,2);
  WildcardsArray[0]:='_';  //<---- seems to be a trublemaker, no idea how to test it with our tests. See http://zeoslib.sourceforge.net/viewtopic.php?f=40&t=13184
  WildcardsArray[1]:='%';
end;

And I confirm that the change EgonHugeist suggested solves the issue.

BTW, I wasn't expecting to find a solution to my problem in the foreseeable future, therefore I found a shortcut to bypass half of the Mormot framework smile
No sqlite3 layer, no TSQLModel, no TSQLRestServerDB instance, no VirtualTableExternalMap, no CreateMissingTables. Sounds to good to be true, doesn't it?
I will share the code here if someone wants to use:

  procedure TMyRestServer.aet_insert(Ctxt: TSQLRestServerURIContext);
  var
    tmp_query: ISQLDBStatement;
    tmp_record: TSQLUserDefs;
    tmp_fields_JSON: RAWUTF8;
    tmp_fields_SQL: RawUTF8;
    tmp_SQL: RawUTF8;
  begin
    tmp_record := TSQLUserDefs.Create;
    tmp_record.USERNAME := 'test_user';
    tmp_record.USERPASS := 'test_pass';
    tmp_record.REAL_NAME := 'test_real_name';
    tmp_record.REAL_SURNAME := 'test_surname';
    tmp_record.REAL_EMAIL := 'test_email';
    tmp_record.USER_COMPANY := 'test_company';
    tmp_record.USER_DISCIPLINE := 'test_discipline';
    tmp_record.EMAIL_RECIPIENT := 1;

    tmp_fields_JSON := tmp_record.GetJSONValues(true,false,soInsert); //{"USERNAME":"test_user","USERPASS":"test_pass",..}
    tmp_fields_SQL := GetJSONObjectAsSQL(tmp_fields_JSON,False,False); //"(USERNAME,USERPASS,..) VALUES ('test_user','test_pass',..)"
    FormatUTF8('INSERT INTO %%;',['user_defs', tmp_fields_SQL], tmp_SQL);
    //Ctxt.Returns('["' + tmp_SQL + '"]'); //
    try
       tmp_query := fConnection.NewThreadSafeStatementPrepared(tmp_SQL, {ExpectResults=}false, {RaiseExceptionOnError=}true);
       tmp_query.ExecutePrepared;
       tmp_query.ReleaseRows;
       Ctxt.Returns('["DONE"]');
    except
       tmp_query := nil;
    end;
  end; 

#4 Re: mORMot 1 » External Zeos MySQL tables with underscore ( _ ) cannot be registered. » 2021-04-10 21:30:11

Hi mpv,

If you check this link, you will notice that VirtualTableExternalRegister function has an "aExternalTableName" argument which  allows you to map the table name
different than the class name for the record. VirtualTableExternalMap function works similarly.

My ORM class is "TSQLUserDefs", which maps to an external MySQL table named "user_defs". You can see a similar usage in SynSelfTests.pas file here.
The same code works for Firebird without any problems.

So the problem should be elsewhere.

Regards.

#5 Re: mORMot 1 » External Zeos MySQL tables with underscore ( _ ) cannot be registered. » 2021-04-10 17:36:22

Dear forum members and ab,

As I mentioned above, Mormot fails to map an external MySQL table with an underscore character such as "user_defs".
I am stuck with this bug, and I couldn't find any workaround so far. I can not rename the table because this table is already in use.

I feel like this bug is more related to sqlite engine than Mormot, but I am not sure.
Here is the code, execution path, and the exception generated after calling "CreateMissingTables":

Code:

aetDBSchema := TSQLModel.Create([TSQLUserDefs], 'root');
VirtualTableExternalMap(aetDBSchema,TSQLUserDefs,fConnection,'mysqltest.user_defs');
aetRestServerDB := TSQLRestServerDB.Create(aetDBSchema, ':memory:', false);
aetRestServerDB.CreateMissingTables();         

Execution Path

1 - TSQLRestServerDB.CreateMissingTables(user_version: cardinal=0; Options: TSQLInitializeTableOptions=[]);
2 - TSQLDataBase.Execute(const aSQL: RawUTF8);
3 - TSQLRequest.Execute(aDB: TSQLite3DB; const aSQL: RawUTF8);
4 - TSQLRequest.Prepare(DB: TSQLite3DB; const SQL: RawUTF8; NoExcept: boolean): integer;
5 - TSQLRequest.Execute;
6 - TSQLRequest.Step: integer;
7 - sqlite3_check(DB: TSQLite3DB; aResult: integer; const SQL: RawUTF8): integer; ----> Creates Exception

Exception and debug log:

SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%';
Step
BEGIN TRANSACTION;
Step
CREATE VIRTUAL TABLE UserDefs USING External(USERNAME TEXT COLLATE SYSTEMNOCASE, USERPASS TEXT COLLATE SYSTEMNOCASE, REAL_NAME TEXT COLLATE SYSTEMNOCASE, REAL_SURNAME TEXT COLLATE SYSTEMNOCASE, REAL_EMAIL TEXT COLLATE SYSTEMNOCASE, USER_COMPANY TEXT COLLATE SYSTEMNOCASE, USER_DISCIPLINE TEXT COLLATE SYSTEMNOCASE, EMAIL_RECIPIENT INTEGER);
Step
Error SQLITE_ERROR (1) [Step] using 3.35.4 - SQL Error: Table 'mysqltest.user\_defs' doesn't exist
Code: 1146 SQL: SHOW INDEX FROM mysqltest.`user\_defs`, extended_errcode=1
ROLLBACK TRANSACTION;
Step
An unhandled exception occurred at $0061B879:
ESQLite3Exception:
  $0061B879
  $0061B2F8
  $0061A6A0
  $0061A70B
  $0061839B
  $004CB33E
  $0040208E

I am out of ideas. Hope someone can find a solution soon.

#6 mORMot 1 » Mormot external table caching seems broken » 2021-04-08 12:19:10

turkerali
Replies: 1

Hi forum members,

I am using Mormot framework with an external MySQL database. I have a very small table
which never changes. So I am trying to activate table cache for this particular table with the
following code:

aetRestServerDB := TSQLRestServerDB.Create(aetDBSchema, ':memory:', false);
aetRestServerDB.CreateMissingTables(0);
aetRestServerDB.Cache.SetCache(TSQLDriveRecord);

Then I retrieve the contents of this external table directly in JSON format, via the SOA method as follows:

procedure TMyRestServer.mysql_test(Ctxt: TSQLRestServerURIContext);
begin
   // ALTERNATIVE 1
   Ctxt.Returns(aetRestServerDB.RetrieveListJSON(TSQLDriveRecord,'',''));

   // ALTERNATIVE 2
   //Ctxt.Returns(VariantSaveJson(aetRestServerDB.RetrieveDocVariantArray(TSQLDriveRecord,'','')));
end;

Whether I activate the cache or not, when I benchmark the above method with ab, I always get around 4,000 req/s (with both ALTERNATIVE 1 and 2).
Therefore the cache seems not working. Is there another method to retrieve all the records at once in JSON from the cache?
Please note that the table I would like to cache is an external table.

Thanks in advance.

#7 mORMot 1 » External Zeos MySQL tables with underscore ( _ ) cannot be registered. » 2021-04-04 13:12:15

turkerali
Replies: 7

Hi,

Any "VirtualTableExternalRegister" call to an external MySQL table with an underscore ( _ ) sign produces an exception as follows:

CODE:

VirtualTableExternalRegister(aetDBSchema, TSQLDriveRecord, fConnection, 'test_table');

RESULT:

ESQLite3Exception: Error SQLITE_ERROR (1) [Step] using 3.35.3 - SQL Error: Table 'testdb.test\_table' doesn't exist
Code: 1146 SQL: SHOW INDEX FROM testdb.`test\_table`, extended_errcode=1
  $0061C92F
  $0061C3B8
  $0061B760
  $0061B7CB
  $0061945B
  $004CC04E
  $004027AD

Any ideas on how to workaround this problem? Thanks in advance.

#8 Re: mORMot 1 » Method based web services, TSQLRestServerDB thread safety concerns » 2021-03-31 11:16:03

Hi Arnaud,

Today I deep dived into the mORMot source code to find the best way for parallel database updates,
and I must say that I am astonished at the beauty of the code.

The excerpt below can be attributed to you I guess:

I have often felt that programming is an art form,
     whose real value can only be appreciated
     by another versed in the same arcane art;
     there are lovely gems and brilliant coups
     hidden from human view and admiration, sometimes forever,
     by the very nature of the process.
     You can learn a lot about an individual
     just by reading through his code,
     even in hexadecimal.

The Story of Mel

#9 Re: mORMot 1 » Method based web services, TSQLRestServerDB thread safety concerns » 2021-03-30 08:46:34

Hi Arnaud,

ab wrote:

Why do you use the TSQLRestClientDB?

I decided to use "TSQLRestClientDB" for inserts/updates after I saw the benchmarks under the 11.7.3. Proven behavior section
of Mormot documentation:

Multi thread process:
  - Create thread pool: 1 assertion passed  3.11ms
  - TSQLRestServerDB: 24,061 assertions passed  903.31ms
     1=41986/s  2=24466/s  5=14041/s  10=9212/s  30=10376/s  50=10028/s
  - TSQLRestClientDB: 24,062 assertions passed  374.93ms
     1=38606/s  2=35823/s  5=30083/s  10=32739/s  30=33454/s  50=30905/s
  - TSQLRestClientURINamedPipe: 12,012 assertions passed  1.68s
     1=4562/s  2=5002/s  5=3177/s
  - TSQLRestClientURIMessage: 16,022 assertions passed  616.00ms
     1=16129/s  2=24873/s  5=8613/s  10=11857/s
  - TSQLHttpClientWinHTTP_HTTPAPI: 24,056 assertions passed  1.63s
     1=5352/s  2=7441/s  5=7563/s  10=7903/s  30=8413/s  50=9106/s
  - TSQLHttpClientWinSock_WinSock: 24,061 assertions passed  1.10s
     1=11528/s  2=10941/s  5=12014/s  10=12039/s  30=9443/s  50=10831/s
  Total failed: 0 / 124,275  - Multi thread process PASSED  6.31s

For direct in-process access, TSQLRestClientDB sounds the best candidate: its abstraction layer is very thin, and much more
multi-thread friendly than straight TSQLRestServerDB calls. It also will feature a cache, on need - see ORM Cache. And it will
allow your code to switch between TSQLRestClientURI kind of classes, from its shared abstract methods.

Is this a different scenario than mine? Or did I misinterpret this part?

ab wrote:

Just use the GlobalRestServerDB instance.
It also has ORM methods. And it will be thread-safe.

If I use the only global TSQLRestServerDB instance for method based services as you suggested, wouldn't the application bottleneck?
Because multiple web requests will be competing for the same resource (GlobalRestServerDB). So there will be say 32 simultaneous insert
requests, but they will need to be serialized due to the global TSQLRestServerDB lock.

If I remember correctly, every application thread created by web server has an associated db connection. Is there a way to utilize this
connection directly and bypass the global TSQLRestServerDB instance?

ab wrote:

So if you need the best performance, use a temporary TSQLRestBatch instance, protect it with its lock, and append the data in it. Then call Send after a while.
It will be almost non blocking with huge performance, since the SQL sent to MySQL will be optimized for insertion.
We do it on production since years, with good stability and no bottlenck.

I tested this solution, and I really like it. Very neat and straightforward. Now the Alternative 3 is as follows:

var
   tmp_record: TSQLDriveRecord;
   tmpBatchRest: TSQLRestBatch;

begin
     tmp_record := TSQLDriveRecord.Create;
     tmp_record.Drive := 'C';
     tmp_record.Active := 1;
     tmp_record.Checked := 1;

     tmpBatchRest := TSQLRestBatch.Create(GlobalRestServerDB, TSQLDriveRecord);
     tmpBatchRest.Add(tmp_record, true);
     Ctxt.Returns('["' + IntToStr(GlobalRestServerDB.BatchSend(tmpBatchRest)) + '"]');

     tmpBatchRest.Free;
     tmp_record.Free;
end;

This method works fine, but I can not catch the SQL Exceptions, it always returns 200, even if the SQL fails. Here is the failed log:

20210330 10412002  +    project1.TMyRestServer(ffffa63271b0).URI GET root/aet_test_mysql_batch in=0 B
20210330 10412002 trace         mORMotSQLite3.TSQLRestServerDB(ffffa630f2f0) BatchSend {"TSQLRestBatch(ffffa448b040)":{"Count":1,"SizeBytes":35}}
20210330 10412002  +            mORMotSQLite3.TSQLRestServerDB(ffffa630f2f0).EngineBatchSend TSQLDriveRecord inlen=36
20210330 10412002 EXC                   EZSQLException ("SQL Error: Duplicate entry 'C' for key 'Drive' \nCode: 1062 SQL: insert into DriveRecord (ID,Drive,Active,Checked) values (?,?,?,?)") [HttpSvr 8888/root THttpSvrResp] at 676fb8 758b74 60d19c 51c540 5189dc 4e0c44 4e2958 4c6aac 4c6bbc 4020ac 4d46c8 4d4118 4da41c 51ed78 655dcc 656e7c 65841c
20210330 10412002 trace                 mORMotSQLite3.TSQLRestServerDB(ffffa630f2f0) EngineBatchSend json=36 B add=1 update=0 delete=0 TSQLDriveRecord

How can I catch the failed SQL errors with TSQLRestBatch?

Thanks in advance.

#10 mORMot 1 » Method based web services, TSQLRestServerDB thread safety concerns » 2021-03-30 06:44:14

turkerali
Replies: 4

Hi everyone,

First of all I would like to thank Arnaud Bouchez for this great ORM framework.

Now back to my question. I have started a new web based project using ExtJS, Mormot and MySQL.
I create the UI with ExtJS library and post the form data to mormot application server for database inserts/updates.
I am using "method based services" via TSQLRestServerFullMemory, and connect to to external MySQL tables using
"VirtualTableExternalRegister. Everything works as expected but I have some concerns.

I handle the incoming web requests using such methods:

procedure TMyRestServer.aet_test_mysql(Ctxt: TSQLRestServerURIContext);
var
     tmp_record: TSQLDriveRecord;
     tmpRestClientDB: TSQLRestClientDB;
begin
     tmp_record := TSQLDriveRecord.Create;
     tmp_record.Drive := 'C';
     tmp_record.Active := 1;
     tmp_record.Checked := 1;

     // ALTERNATIVE 1
     tmpRestClientDB := TSQLRestClientDB.Create(aetDBSchema, nil, ':memory:', TSQLRestServerDB);
     tmpRestClientDB.Server.CreateMissingTables(0);

     // ALTERNATIVE 2, USE THE GLOBAL "GlobalRestServerDB"
     //tmpRestClientDB := TSQLRestClientDB.Create(GlobalRestServerDB);
	
     // INSERT THE RECORD
     if (tmpRestClientDB.Add(tmp_record, true) = 0) then Ctxt.Returns('["MySQL ERROR."]');
     else Ctxt.Returns('["RECORD ADDED."]');

     tmpRestClientDB.Free;
     tmp_record.Free;
end;

As you will see in the above code, there are 2 alternatives for creating the TSQLRestClientDB for database access:
- The first one will create a hidden TSQLRestServerDB server, and call it internaly.
- The second one initializes the class for an existing (global) TSQLRestServerDB.

I don't like the first approach because I need to call the "CreateMissingTables" for every request, which is totally unneccessary
because the database structure does not change in every request. Besides, it creates a hidden TSQLRestServerDB  which is
additional overhead.

The advantage of the second approach is, you don't need to call the "CreateMissingTables" in every single web request,
because it is already called by a global TSQLRestServerDB instance when the application starts:

    // Global TSQLRestServerDB
    GlobalRestServerDB := TSQLRestServerDB.Create(aetDBSchema, ':memory:', false);
    GlobalRestServerDB.CreateMissingTables(0);

But I believe this approach is not thread-safe, because multiple threads can modify the global GlobalRestServerDB simultaneously.

How can I protect the global GlobalRestServerDB? If I use locks, I believe it will be the bottleneck for the application, because 32
threads will need to wait for this lock for database inserts/updates, although the MySQL server can handle multiple requests
simultaneously. Or is there a better multithread-friendly database ORM approach for method-based services?

Thanks in advance.

Board footer

Powered by FluxBB