Real high-concurrency & replication is coming to SQLite!

edwinsn · 2023-02-07 10:14:36

Good news from the SQLite tean:

@ab,

Since you already marked mORMot 1 to be in maintenance mode, will you keep updating the SQLite engine for mORMot 1 in the future?

ab · 2023-02-07 11:02:08

Yes, mORMot 1 will continue to be upgraded with the latest SQLite3 statics.
There is a single build script for both mORMot 1 and mORMot 2.

This hctree prototype is very interesting for sure.
The fact that it is made by SQLite3 authors make it a good choice for the long term support - if it is not only a prototype but a stable enough product.
But note that this hctree database is a new DB format. It can open SQlite3 databases, but it is a new file layout.

edwinsn · 2023-02-07 11:13:31

Great!

ab wrote:

But note that this hctree database is a new DB format. It can open SQlite3 databases, but it is a new file layout.

I don't think it's an issue as of now...

mpv · 2023-02-07 18:38:57

For sure, it's huge improvement. Currently I can't use sqlite3 on real productions where long transactions are commonplace (because of sqlite3_busy error), but hctree should solve this problem, and (at least where DB size < 500Gb) sqlite can be usable.

edwinsn · 2023-02-08 03:17:04

mpv wrote:

...Currently I can't use sqlite3 on real productions where long transactions are commonplace (because of sqlite3_busy error)...

Would you share with us some pseudo code about how do you deal with the sqlite3_busy error?
I guess you use a retrying mechanism?

mpv · 2023-02-08 15:44:44

mORMot uses global lock for SQLITE3 updates, so problem not happens where.

I use my own TubSQLite3ThreadSafeConnection = class(TSQLDBConnectionThreadSafe) without global lock (it actually exists inside sqlite) and sets

TSQLDataBase.WalMode := true;
TSQLDataBase.BusyTimeout := 10000; // 10 seconds

In case of sqlite3_busy - this error is passed to user and on UI message "try again later" is displayed.

edwinsn · 2023-02-09 02:10:08

Thanks for sharing, when not using the ORM but the database layer directly, I also increase the value of BusyTimeout

mORMot Open Source

#1 2023-02-07 10:14:36

Real high-concurrency & replication is coming to SQLite!

#2 2023-02-07 11:02:08

Re: Real high-concurrency & replication is coming to SQLite!

#3 2023-02-07 11:13:31

Re: Real high-concurrency & replication is coming to SQLite!

#4 2023-02-07 18:38:57

Re: Real high-concurrency & replication is coming to SQLite!

#5 2023-02-08 03:17:04

Re: Real high-concurrency & replication is coming to SQLite!

#6 2023-02-08 15:44:44

Re: Real high-concurrency & replication is coming to SQLite!

#7 2023-02-09 02:10:08

Re: Real high-concurrency & replication is coming to SQLite!

Board footer