mORMot Open Source

Great work. I would really appreciate if you could give some feedback about the problems you experienced while adapting the SQLite3MultipleCiphers code. Especially the Linux side, since I develop under Windows most of the time and have only limited Linux experience. Additionally, I would be interested in getting some information about your fast AES implementation. Maybe in a private message?

I have fixed this bug already. However, there might be other bugs lurking around. Not all aspects of the new VFS based implementation have been tested yet.

Another SQLite update is coming soon. Version 3.32.3 will probably be released within the next few days.

I don't think that D.R. Hipp made this decision to annoy users/developers. In the long run it certainly frees him to further improve SQLite and to introduce new exciting features.

Absolutely right. This post on the SQLite Forum shows that obviously the commercial SQLite Encryption Extension (SEE) was updated as well. And my guess is that it is now based on a VFS implementation (although I can't be absolutely sure about that).

A VFS shim is currently the only way to intercept SQLite's file I/O without requiring to tamper too much with SQLite's core implementation. So a VFS based implementation of the encryption extension is a rather logical path.

I fully agree, but it will be quite difficult to accomplish this goal, if one wants to preserve full compatibility with the previous SQLITE_HAS_CODEC API. For example, SQLite's ATTACH statement has an undocumented KEY parameter for setting the passphrase for an attached database. In the public SQLite version 3.32.x and later this parameter is not passed forward anywhere, but silently ignored. If this feature is used in user applications there is no way to avoid patching the SQLite code, unfortunately. In user applications the use of the KEY could be avoided by using an URI parameter in the database file name. However, this requires to modify the code of the user application accordingly.

As the author of wxSQLite3, a SQLite3 wrapper with encryption support, and  SQLite3MultipleCiphers, which is based on a VFS shim, I'd like to add that I was confronted with the very same decision: (1) fork SQLite3 and merge upstream changes, or (2) implement a VFS shim. I came to the conclusion that maintaining a fork would incur growing pains in the long run. Implementing a VFS shim is indeed quite some work, but IMHO much less than maintaining a SQLite3 fork.

According to an answer I got on Zetetic's SQLCipher forum, they seem to intend to maintain their own SQLite3 fork. However, looking at a recent SQLCipher post it is unclear at the moment, whether they will make their solution publicly available or not. If yes, you may be able to adapt it to your implementation of the SQLite3 encryption extension.

Regarding performance of a VFS shim I doubt that it will significantly decrease. Yes, the VFS shim adds one level of indirection, but compared to the time required for encrypting or decrypting a database page this will be neglectable.

The VFS layer is rather dumb and "knows" (almost) nothing about internal SQLite3 structures. Therefore the "hard work" in implementing the VFS shim is to analyze how SQLite accesses the database files and how essential information (like the page number) can be deduced. However, these obstacles are already out of the way.

A disadvantage of SQLite3MultipleCiphers is that it requires patches to the SQLite3 amalgamation. This was not necessary for my previous solution, based on the SQLITE_HAS_CODEC API. Although there are only 4 patches - all of them very small -, I would have preferred a solution without them. Since you intend to support a single encryption scheme only, you might get away without patching the SQLite3 amalgamation code. However, you mentioned "messing with SQLite3 VM opcodes", so it is likely that you will be forced to apply patches, too.