#1 Re: mORMot 1 » 10.4 sydney issues » 2020-06-10 19:58:34

ab wrote:

Our AES implementation uses low-level AES-NI opcodes, so is very fast, with no need of using external libraries.
You will find some reference asm e.g. in https://github.com/synopse/mORMot2/blob … asmx64.inc

Thanks for the pointer. I'll take a look. Although the generic AES implementation used in wxSQLite3/SQLite3MultipleCiphers is fast enough in many use cases, there are of course use cases where speed really matters. So I'm looking for other AES implementations making use of AES instructions of the processor.

ab wrote:

Under Linux, everything is fine.

Good to hear. Thanks.

ab wrote:

The only problem I saw is that we usually use not the default 'unix' VFS on Linux, but 'unix-excl' which is better for WAL support - and you can't use it in sqlite3_openv2() otherwise our encryption is not triggered.
So I changed https://github.com/synopse/mORMot/blob/ … 3mc.c#L243

Yes, this is a problem I'm aware of. Choosing a different default VFS has to be done either before the SQLite3 initialization function is called for the first time, or the additional initialization function has to be modified (as you did).

In principle, VFSs can be nested. However, there is no API which would allow to check whether VFSs are compatible with each other. Quite often they are not. I will have to do further research, how to handle VFS nesting.

ab wrote:

Another small problem is that your code requires a C99 compiler, which is not the case e.g. for Embarcadero C++ Builder XE7 I use. All variable declarations should be moved at the beginning of the function. I have fixed this in my own fork.

Thanks for mentioning this. Some time ago I had a report from a user who used an old Visual C++ compiler; he had similar problems. Maybe I can lift off the requirement of C99.

ab wrote:

You can check the latest state at https://github.com/synopse/mORMot/tree/ … algamation
It includes your fix - thanks!

The next weeks or months may bring additional changes, fixes etc to SQLite3MultipleCiphers. The project is still "Work in progress".

#2 Re: mORMot 1 » 10.4 sydney issues » 2020-06-10 11:34:49

ab wrote:

After a lot of painful debugging and adaptation in plain C, we just updated SQLite3 to latest 3.32.2 revision
- switched to VFS pattern for encryption
- sqlite3 cross-compilation is now done from Linux scripts

Great work. I would really appreciate if you could give some feedback about the problems you experienced while adapting the SQLite3MultipleCiphers code. Especially the Linux side, since I develop under Windows most of the time and have only limited Linux experience. Additionally, I would be interested in getting some information about your fast AES implementation. Maybe in a private message?

I have fixed this bug already. However, there might be other bugs lurking around. Not all aspects of the new VFS based implementation have been tested yet.

Another SQLite update is coming soon. Version 3.32.3 will probably be released within the next few days.

#3 Re: mORMot 1 » 10.4 sydney issues » 2020-06-07 11:50:16

MC wrote:

I use sqlite3.serialize and sqlite3.deserialize in order to encrypt/decrypt SQLite database with ECC to/from disk.

In pratice I use :memory: database, save it to memory stream and encrypt the stream to disk.

Forloading, I load file to memory stream, decrypt it, and load in memory database with decrypted stream.

There are certainly use cases for which the serialize/deserialize approach makes sense, e.g. read-only databases which are not too big.

In general, using solely a memory based database is not such a good idea, because it violates the D(uration) of the ACID principle. For example, in case of a power loss all database modifications are gone.

An alternative to transparent encryption of the whole database can be to encrypt only security critical database content. For example, the commercial variant of SQLCipher offers additional SQL functions to encrypt and decrypt data. The advantage of that approach is that is doesn't require any modifications of the SQLite code. The disadvantage is that you can't use SQL's strengths to operate on encrypted data, like indexing, searching etc.

#4 Re: mORMot 1 » 10.4 sydney issues » 2020-06-06 10:39:09

Greg0r wrote:

Agree with utelle, that author will not help there, probably intentional block even

I don't think that D.R. Hipp made this decision to annoy users/developers. In the long run it certainly frees him to further improve SQLite and to introduce new exciting features.

Greg0r wrote:

I don't think that  this change will be ever reverted

Absolutely right. This post on the SQLite Forum shows that obviously the commercial SQLite Encryption Extension (SEE) was updated as well. And my guess is that it is now based on a VFS implementation (although I can't be absolutely sure about that).

Greg0r wrote:

I have no idea for now which method will be the best and work stable and fast, but my imagination and experience tells me that forking SQLite is the worst possible idea.

A VFS shim is currently the only way to intercept SQLite's file I/O without requiring to tamper too much with SQLite's core implementation. So a VFS based implementation of the encryption extension is a rather logical path.

Greg0r wrote:

Perfect solution should also do not require to touch amalgamation c oryginal files.

I fully agree, but it will be quite difficult to accomplish this goal, if one wants to preserve full compatibility with the previous SQLITE_HAS_CODEC API. For example, SQLite's ATTACH statement has an undocumented KEY parameter for setting the passphrase for an attached database. In the public SQLite version 3.32.x and later this parameter is not passed forward anywhere, but silently ignored. If this feature is used in user applications there is no way to avoid patching the SQLite code, unfortunately. In user applications the use of the KEY could be avoided by using an URI parameter in the database file name. However, this requires to modify the code of the user application accordingly.

#5 Re: mORMot 1 » 10.4 sydney issues » 2020-06-06 08:08:27

edwinsn wrote:

Is there any way to ask the original SQLite author for help?

It is very unlikely to get help from the original author, D.R. Hipp in this matter. In fact, it was his decision to remove the SQLITE_HAS_CODEC API, on which most independent SQLite encryption extensions were based up to now. Although that API was part of SQLite for over 15 years, it was an undocumented feature, and D.R. Hipp is of course free to remove undocumented features any time he wants.

D.R. Hipp offers a commercial SQLite Encryption Extension (SEE) at a price of 2000 USD. This is the "help" you can get ... if you can afford it.

#6 Re: mORMot 1 » 10.4 sydney issues » 2020-06-06 07:58:37

ab wrote:

About SQlite3 3.32.xxx there is a big problem with codecs.

Critical changes to the public SQLite code were introduced on Feb 7, 2020: “Simplify the code by removing the unsupported and undocumented SQLITE_HAS_CODEC compile-time option”. With the release of SQLite version 3.32.0 on May 22, 2020 these changes finally took officially effect, although they weren't officially announced.

As an unexpected consequence, we are NOT ANY MORE able to compile the new SQlite3 amalgamation with our encryption patch.

Reverting the patch is not an option, since it not a few lines of code, and it mess with SQLite3 VM opcodes.
Perhaps a dedicated VFS may be the solution - this is the path choosen by https://github.com/utelle/SQLite3MultipleCiphers for instance.

But a VFS is a lot of work, and performance may also decrease.

As the author of wxSQLite3, a SQLite3 wrapper with encryption support, and  SQLite3MultipleCiphers, which is based on a VFS shim, I'd like to add that I was confronted with the very same decision: (1) fork SQLite3 and merge upstream changes, or (2) implement a VFS shim. I came to the conclusion that maintaining a fork would incur growing pains in the long run. Implementing a VFS shim is indeed quite some work, but IMHO much less than maintaining a SQLite3 fork.

According to an answer I got on Zetetic's SQLCipher forum, they seem to intend to maintain their own SQLite3 fork. However, looking at a recent SQLCipher post it is unclear at the moment, whether they will make their solution publicly available or not. If yes, you may be able to adapt it to your implementation of the SQLite3 encryption extension.

Regarding performance of a VFS shim I doubt that it will significantly decrease. Yes, the VFS shim adds one level of indirection, but compared to the time required for encrypting or decrypting a database page this will be neglectable.

The VFS layer is rather dumb and "knows" (almost) nothing about internal SQLite3 structures. Therefore the "hard work" in implementing the VFS shim is to analyze how SQLite accesses the database files and how essential information (like the page number) can be deduced. However, these obstacles are already out of the way.

A disadvantage of SQLite3MultipleCiphers is that it requires patches to the SQLite3 amalgamation. This was not necessary for my previous solution, based on the SQLITE_HAS_CODEC API. Although there are only 4 patches - all of them very small -, I would have preferred a solution without them. Since you intend to support a single encryption scheme only, you might get away without patching the SQLite3 amalgamation code. However, you mentioned "messing with SQLite3 VM opcodes", so it is likely that you will be forced to apply patches, too.

Board footer

Powered by FluxBB