mORMot Open Source

ncesi

Member

Registered: 2013-03-01

Posts: 3

SQLite FTS5 and unicode support

I have a big problem using FTS4 and latin script characters. It seems that unicode characters are not supported in current implementation of the FTS engine and as I understand SQLITE_ENABLE_ICU is not a good option on windows platform.

It seems to me that FTS5 extension handles diacritics in correct way. Do you have any plans to support FTS5 in your framework?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,238

Website

Re: SQLite FTS5 and unicode support

AFAIK FTS5 is still an experimental extension.
See https://www.sqlite.org/releaselog/3_8_11.html

Sqlite team is still fixing bugs, and potentially changing the API.
We will support it as soon as it would be stabilized.

In the meanwhile, did you try to use the "unicode61" tokenizer in FTS3/FTS4?
We just added TSQLRecordFTS3Unicode61 and TSQLRecordFTS4Unicode61 classes.
See http://synopse.info/fossil/info/b072487ada

edwinsn

Member

Registered: 2010-07-02

Posts: 1,215

Re: SQLite FTS5 and unicode support

I'm glad to know there is the new fts5! Even it's experiential, I suggest to add it to the mORMot framework anyway!

---

Delphi XE4 Pro on Windows 7 64bit.
Lazarus trunk built with fpcupdelux on Windows with cross-compile for Linux 64bit.

edwinsn

Member

Registered: 2010-07-02

Posts: 1,215

Re: SQLite FTS5 and unicode support

Hi Arnaud,

If you want to use fts5, you should compile the sqlitefts3.obj using the free c++ builder command line tool, right?

---

Delphi XE4 Pro on Windows 7 64bit.
Lazarus trunk built with fpcupdelux on Windows with cross-compile for Linux 64bit.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,238

Website

Re: SQLite FTS5 and unicode support

Yes, just add the corresponding conditional to the c.bat file.