You are not logged in.
Hello ab,
According to this official sqlite3 document page (https://www.sqlite.org/backup.html), the current implementation of the sqlite backup API can be used to copy data to or from in-memory databases.
According to the mormot doc:
The same backup process can be used e.g. to save an in-memory SQLite3 database into a SQLite3 file,
as such:
if aInMemoryDB.BackupBackground('backup.db3',-1,0,nil) then
aInMemoryDB.BackupBackgroundWaitUntilFinished;
Above code will save the aInMemoryDB database into the ' backup.db3 ' file.
So we can perform memory-to-db.
How about loading a DB from file to memory? Thanks.
@ab,
Does the moRMot framework itself has a mechanism for handling network broken situations? For example, what will happen if the network is broken during a batch-mode update process? Thanks.
Don't use SQL but batch crud operations.
Or use a server side service with raw SQL execution.
This is what I'm doing now - no client-side CURD/SQL nor clident-side transactions.
Is is possible that the instability of the network, such as when transmitting through VPN, would cause such problems?
@ab,
Do you have any other advises at this moment ![]()
This is a follow-up for this original post: (https://synopse.info/forum/viewtopic.php?id=3682).
The mORMot server log and madExcept log (client-side0 are located here: (https://gist.github.com/edwinyzh/053ea7 … ff4257dc2c)
Additional info
===
- Client-side: At the time of the original post, the program in question was using client-side transactions, however by now all client-side transactions have already been removed and have been converted to batch-mode instead.
- Server-side: I have applied checking for "is in transaction" before calling
rollbackor
TransactionBegin.
Note:
===
In the original post, ab said:
I guess there is something wrong with your SQL statement executed in first line, during the Batch sequence.
.
I don't quite understand that, since we cannot compose SQL statements ourselves in batch-mode...
References
===
Discussion about the same SQLite error in the Xaramin forum: (https://forums.xamarin.com/discussion/1 … t-rollback). This error seems to be really mystery...
Thanks zed!
Hello Arnaud,
Today I noticed the "datetime" column, which is the first column of theserver log file, becomes hexadecimal values.
I guess there is a change recently (in the last two months). How to get it back to the normal date time values?
Thanks.
For the record, "[mormot_root]\SQLite3\Samples\04 - HTTP Client-Server\Unit2Static.pas" is the file that demonstrate EMartin's idea ![]()
Thank you two.
@ab, is there a way for us to extend the TSQLRestBatch class to support the real "INSERT OR REPLACE" operation supported by SQLITE? Thanks.
I'm currently using a timestamp-based UUID generated at client sides, so far so good: https://synopse.info/forum/viewtopic.php?id=2522
@ab,
That's a great proposal! And I'm looking forward to it.
@fubail@qq.com,
I think you at least have two options:
1 - As @lele9 said, re-design your data model, make the "detail table" as a property of the master record, in this case I prefer TObjectList as the type of the property, that way, the "detail records" are stored in JSON in the db, you can then take advantage of the SQLite's JSON feature. If you use another db system, I believe most modern db system has support for JSON. ref: (https://synopse.info/files/html/Synopse … ml#TITL_26)
2 - Control the generation of the ID value for your master records.
Wow, you read all those 2000+ pages.
I get started with the simple samples Arnaud did, without reading too much docs in prior, But search and read the doc when I have a problem/question.
IMHO, the basic idea of mORMot is simple, it's just that it includes too much features.
You can define a TSQLRecord class matching the database view.
But of course, it won't create the view, but generate the SQL as expected (SELECT is the same as for a regular table).
Of course, Add/Delete/Update won't work, unless the database supports INSERT/UPDATE/DELETE on views (which is very unlikely, but working on some engines, to some extent).
Oh, it didn't occurred to me that this would work, is this officially supported by the framework? I mean, since it's not officially documented, AFAIK.
Currently, I derive from TSQLTableJSON to define my "view objects".
Pass "nil" for the record class parameter to enable multi table transaction.
Note, if you are using client-side trans, it's strongly recommended to use the batch-mode instead.
@Ab,
Thank you very much for the info! So I've applied the default http authentication using one of the 3 default users defined by the mORMot framework itself.
Have a nice weekend!
ab, the program is being used in a local secure network, so I did not implement the authentication, so in this case, should I still provide a unique "SessionId" for each client? Thanks.
Yes, I know batch-mode is the way to go, but there are some legacy code I cannot convert to batch mode yet.
So back to the SessionID question - when there are multi-clients and all of the clients are using the same default value (which is 1, according to the method definition), will there be conflicts? Thanks.
@ab,
Sorry for posting to this old thread - so with the latest version of mORMot, does the SessionID parameter still make sense for the client-side transactions? Thanks.
Try use a different port other than the one (80 by default) used by ISS.
@ab,
I guess you've just migrated to a new server, and thought I'd let you know that the mORMot pdf doc download link is not working.
Error msg:
404 Not Found
nginx/1.10.0 (Ubuntu)Thanks for the explanation @EMartin. I think @Bo's understand of @juwo's question is correct.
I don't think there are "automatic" ways of that
One idea implementing this:
You set up a global "gDbSchemaVersionNoEach" variable, and each time you upgrade the db schema, you increase that value, and you can expose that version number to the clients via a Method-based API. If you want to store the version number in a sqlite db file, consider the official schema_version PRAGMA (https://www.sqlite.org/pragma.html#prag … ma_version)
Some JS example would get people started easier. I don't have a use case for JWT yet, but when I've got some time I'll do some investigation and post back to here, if I'd have any good findings ![]()
@ab, I see now, put the generated .mab file in the same folder as the .exe file, right? thanks!
Hello ab,
I've log entry:
20161231 20242758 EXC ESQLite3Exception {"ErrorCode":1,"SQLite3ErrorCode":"secERROR","Message":"Error SQLITE_ERROR (1) [Step] using 3.15.2 - cannot rollback - no transaction is active, extended_errcode=1"} at 0075FBA2 stack trace API 00751C33 0040A92C
How to convert the address shown in the log entry to source line position? I tried using the LogView.lpr log viewer, but it couldn't load it (raised error when loading the log file).
Thanks.
I wish the ISO8601 datetime values stored by the framework has milliseconds too!
@EMartin, can you be more obvious? I don't understand it. Thanks.
I think @EMartin's idea is very simple and workable! The "root" as mentioned by EMartin is actually the URI for the data model, and each model/rest-server/http-server combination can have it's own URI, for example, "db1", "db2", "db3"....
It's not not clear to me that, how to create a TSQLHttpClient connection to the specified "db1" or "db2"?...
Multitenancy is a new term to me too ![]()
mORMot is more than an ORM, on the db side, it's also a wrapper around SQLite to transform SQLite the single-user DB into a multi-user DB like other so-called enterprise DBs, while keeping the zero-configuration nature of the amazing SQLite ![]()
It's also a feature-rich web server.
Sometimes it take a little time to realize that for a newcomer ![]()
It's like a balancing server (the main server), and other real db servers (sub servers)?
I think the architecture would depend on how the clients would access the sub servers, for example, would these sub servers hosted on the same machine or on other physical macines?
@mine1961, I don't know if caching is also enabled in the cross-platform units, but just in case you missed the related chapter: http://synopse.info/files/html/Synopse% … DE_TITL_86
@ab,
Is there a way to **partially** update the cache, for example, partially update the existing cache with records modified since a given time (the time the client cache was created previously). I'll illustrate the idea will some pseudo code:
var
//global var
gCacheTime : TDateTime;
procedure ConstructCache;
begin
gCacheTime := Now;
myDb.Cache.SetCache(TSQLMyTable);
myDb.Cache.FillFromQuery(TSQLMyTable, 'ID > 0', []);
end;
// how to achieve the following?
procedure UpdateCache;
var
onlyNewRecords: TSQLMyTable;
begin
// retrieve records only those updated since the last time the client cache was updated.
myTable := TSQLMyTable.CreateAndFillPrepare('LastUpdateTime > ?', [gCacheTime]);
myDb.Cache.UpdateFromQuery(onlyNewRecords);
end;I wish my propose makes sense ![]()
PS, the call of AppCtx.Db.Cache.SetCache() shouldn't necessary before FillFromQuery() is called, right?
I wasn't aware it's necessary until I debugged into the source of FillFromQuery - open source is soooooooo great ![]()
@ab,
Tested TSQLRestCache.FillFromQuery inside a local network, the result is great - went from 6,373 MS to 40 MS.
Result:
Test loading records one by one from db - non-cached:
6,373 MS = 0:0:6
Destroyed server-side caching
Constructing client cache:
100 MS = 0:0:0
Test loading records one by one from db - cached:
40 MS = 0:0:0Testing code:
procedure TfrmDebug.TestClientCaching;
var
Elapsed: TTimeSpan;
myStr: string;
newProd: TSQLProduct;
prodList: TSQLProduct;
stopWatch: TStopwatch;
procedure TestReadProdFromDbOneByOne;
begin
prodList.FillRewind;
while prodList.FillOne do
begin
AppCtx.Db.Retrieve(prodList.ID, newProd);
end;
end;
begin
//memCache.Clear;
AppCtx.Db.ServerCacheFlush;
AppCtx.Db.Cache.Clear;
prodList := TSQLProduct.CreateAndFillPrepare(AppCtx.Db, 'ID > 0');
GC(prodList);
newProd := TSQLProduct.Create;
GC(newProd);
// test without caching;
memCache.Lines.Add('Load one by one from db - non-cached');
stopWatch := TStopwatch.StartNew;
TestReadProdFromDbOneByOne;
Elapsed := Stopwatch.Elapsed;
memCache.Lines.Add(FormatUTF8('% MS = %:%:%', [HumansInt(Trunc(Elapsed.TotalMilliseconds)), Elapsed.Hours, Elapsed.Minutes, Elapsed.Seconds]));
memCache.Lines.Add('Destroying server-side caching');
AppCtx.Db.ServerCacheFlush;
AppCtx.Db.Cache.Clear;
//load the all products into client cache
memCache.Lines.Add('Constructing client cache');
stopWatch := TStopwatch.StartNew;
AppCtx.Db.Cache.SetCache(TSQLProduct);
AppCtx.Db.Cache.FillFromQuery(TSQLProduct, 'ID > 0', []);
Elapsed := Stopwatch.Elapsed;
memCache.Lines.Add(FormatUTF8('% MS = %:%:%', [HumansInt(Trunc(Elapsed.TotalMilliseconds)), Elapsed.Hours, Elapsed.Minutes, Elapsed.Seconds]));
// test with caching;
memCache.Lines.Add('Load one by one from db - cached');
stopWatch := TStopwatch.StartNew;
TestReadProdFromDbOneByOne;
Elapsed := Stopwatch.Elapsed;
memCache.Lines.Add(FormatUTF8('% MS = %:%:%', [HumansInt(Trunc(Elapsed.TotalMilliseconds)), Elapsed.Hours, Elapsed.Minutes, Elapsed.Seconds]));
memCache.Lines.Add('=============================');
end;Wow! Will do the test soon, thanks Arnaud!
@ab, Yes, I'm currently doing it with TDictionary<TSQLMyRecord>, it's already very handy and useful, I just was wondering if mORMot's built-in caching ability might be more handy - it'll make the cache completely transparent to my code, in other words, my code won't have to care about both the self-implemented in-memory TDictionary and TSQLRestClient - all it needs is interfacing with TSQLRestClient ![]()
Thanks ab, I did read the doc before I'm asking the question here, it's just wasn't very clear to me.
From the doc:
Once enabled for a table and a set of IDs on a given table, any further call to TSQLRest.Retrieve(aClass,aID) or TSQLRecord.Create(aRest,aID) will first attempt to retrieve the TSQLRecord of the given aID from the internal TSQLRestCache instance's in-memory cache, if available.
Note that more complex requests, like queries on other fields than the ID primary key, or JOINed queries, won't be cached at REST level. But such requests may benefit of the JSON global cache, at SQLite3 level, on the server side.
So it's not possible to read all records from a table into the cache on the client side, in a single network round-trip (using 'ID > 0' as the retrieving conditions)? because when retrieving with a single ID, you read only a single record at a time.
My intention is to load an entire table into the RAM, to avoid a lot of "retrieve one record a time" operations, to speed things up.
@Ab,
Assume I want to cache all records for a specified table on the client side, pseudo code:
myRestClient.SetCache(TSQLMyTable);Up to this point, there are not real cached records in the memory yet, until any calls to Retrieve or CreateAndFillPrepare, right?
Now assume the following code is executed:
myRestClient.Retrieve('ID > 0', myRec);At this point, all records for TSQLMyTable are cached now. Am I understanding it correct?
Thanks.
@mpv,
Thanks, will include the "symbol reference info" from now on.
@ab,
Thanks, the version I'm using is from Dec 4, 2016 (https://github.com/synopse/mORMot/commi … 1471514b3e), so I think it's ok.
Update 1:
I'll check the version of the executable in question and compare it with the mORMot version. Yes, I remember that fix - it's about "thread b" might wrongly commit transaction started by "thread a", right?
Re the log entries, to be honest, it's only 11 lines, it's far from huge, right? ![]()
Hello Arnaud,
What can you see/guess from the following logs?
Would an inappropriate call to "rollback" cause the db server stop working?
Thanks.
mORMot logs:
20161210 18365549 EXC ESQLite3Exception {"ErrorCode":1,"SQLite3ErrorCode":2,"Message":"Error SQLITE_ERROR (1) [Step] using 3.15.1 - cannot rollback - no transaction is active, extended_errcode=1"} at 0075E95E stack trace API 00750A5A 0040A92C
20161210 18365746 EXC EORMBatchException {"Message":"TDbServer.EngineBatchSend: TDbServer.TransactionBegin timeout"} at 008C3A13 stack trace API 00750A5A 0040A92C
// 3 similar "TDbServer.EngineBatchSend: TDbServer.TransactionBegin timeout" here...
20161210 18371327 EXCOS EAccessViolation (C0000005) at 0076CB08 stack trace 0040B440 0040B940 7C80A284 00E31FCF 008F9D9B 008FE9D1 004AEC7D 00542ABD 0040B34A 004AEB63 004AEBCB 7C80B713
// 2 similar "TDbServer.EngineBatchSend: TDbServer.TransactionBegin timeout" here...
20161210 18374706 EXC ESynException {"Message":"Reentrant TSynMonitor.ProcessStart"} at 0073CFBC stack trace API 00750A5A 0040A92C
20161210 18383854 EXCOS EAccessViolation (C0000005) at 0076CB08 stack trace 0040B440 0040B940 7C80A284 00E31FCF 008F9D9B 008FE9D1 004AEC7D 00542ABD 0040B34A 004AEB63 004AEBCB 7C80B713
// 2 similar "TDbServer.EngineBatchSend: TDbServer.TransactionBegin timeout" here...
20161210 18384949 EXCOS EAccessViolation (C0000005) at 004059DA stack trace 03837615 004AEC7D 00542ABD 0040B34A 004AEB63 004AEBCB 7C80B713
20161210 18385149 EXCOS EAccessViolation (C0000005) at 00405925 stack trace 00542ABD 0040B34A 004AEB63 004AEBCB 7C80B713
// 3 more "EXCOS EAccessViolation (C0000005) at 00405925" errors here...
Joining the TechEmpower Framework Benchmarks would be very interesting ![]()
@edwinsn Did you read this thread? Hope it's help
I haven't, but I will do so soon! Thanks!
@mohsenti,
Sorry for hijacking the thread, Although I think if you are asking about the web client accessing a mORMot server, my questions posted here are relavant and might be also helpful to you ![]()
@ab,
Then the mORMot's schema is very cool! But how to interact it with JavaScript inside a web browser? Can you show me some example code? Pseudo code would be enough ![]()
AFAIK JWT seems not integrated to intercooler.js, but we may easily add its support within mORMot.
There are other standalone js lib for JWT. But I don't think we need to follow the JWT spec strictly? The "Token-based authentication" described in the link below is quite sufficient, I guess:
OK, came up with a idea for integrating authentication for web browsers. Does the following make sense?
Handling user login:
===
1 - The browser submits a form, containing the username and password.
2 - Somewhere on the server side, such as TMyRestServer.Login(), check the username and password, if both of them are correct, generate a ramdom string, such as "b6tUVDzb5vGs6kMuF3SfSv2v8CTemHTBmJmmwrA8", push that string into a server-side 'LoggedInUserTokenList', and store that string as a cookie value in the client side, via TSQLRestServerURIContext.OutSetCookie, something like:
OutSetCookie := '"userToken=b6tUVDzb5vGs6kMuF3SfSv2v8CTemHTBmJmmwrA8"';3 - done. From this point, according to the HTTP protocol, every http requests from that browser will automatically include the userToken cookie we set just now.
Check if the current user is logged in:
===
1 - In OnBeforeURI, check the input cookie value called userToken, via TSQLRestServerURIContext.InCookie, if the value can be found in 'LoggedInUserTokenList', then it means the user is logged in, we allow the execution to continue, otherwise return False and redirect it to TMyRestServer.Login().
Am I correct? Thanks.
Edit 1:
A similar idea is better described here: http://stackoverflow.com/a/32218069/133516
My next reading is about JWT, which seems to go furhter than the approach described above.
Folloing ab's instruction, Mainly we need to change two places:
function CreateSampleModel: TSQLModel;
begin
//result := TSQLModel.Create([TSQLSampleRecord]); // default is 'root'
// edwin
result := TSQLModel.Create([TSQLSampleRecord], 'API');
end;And, note the 3rd param below:
DB := TMyServer.Create(Model,ChangeFileExt(ExeVersion.ProgramFileName,'.db3'),False);Now my question to Arnaud:
Would you or other knowledgeable people give us further some more instruction on authentication from the browser, using plain js? I mean:
1- Login, we need to transfer the username and password to the server;
2 - Session management, each client (web browsers) need to maintain its login status, maybe using standard cookies? (Note, I read your doc, you mentioned cookies, but did not provide sample code
.
BTW, I found a very **simple** and easy to understand frond end framework, called intercooler.js
Thanks.
Except from the doc:
If your purpose is not to have a full SQLite3 engine available, you may create your server from a
TSQLRestServerFullMemory class instead of TSQLRestServerDB : this will implement a fast
in-memory engine (using TSQLRestStorageInMemory instances), with basic CRUD features (for ORM),
and persistence on disk as JSON or optimized binary files - this kind of server is enough to handle
authentication, and host services in a stand-alone way.
You might instead want to use TSQLRestServerDB with ':memory:' (without the quotes) as the filename? reference is here: https://www.sqlite.org/inmemorydb.html
Thanks Maciej.
Does your code allow serializing multiple batch operation controlled by an instance of TSQLRestBatch? I mean something equals to:
var
myBat, newBatch: TSQLRestBatch;
begin
myBat.Add(...);
myBat.Delete(...);
myBat.Update(...);
myBat.SaveToFile('c:\test.txt');
// now reload the saved data.
newBatch := TSQLRestBatch.Create(myDb);
newBatch.LoadFromFile('c:\test.txt');
myDb.Send(myBatch, ...);
end;Actually I'm talking about the server side ![]()
I see now, access TSQLRestServerDB.DB.LastChangeCount.
Thanks!
Sometimes after executing a DELETE/UPDATE statement, we need to examine the affected row count and I remember the SQLITE engine itself provides this feature. How to get that info out of TSQLRest? Thanks.
Is it possible save the content of a TSQLRestBatch instance, in case of a network error, and later reload the content from file, so that it can be sent by TSQLRest.BatchSend() again?
Thanks.