You are not logged in.
- Topics: Active | Unanswered
#1 2011-03-09 15:36:19
- edwinsn
- Member
- Registered: 2010-07-02
- Posts: 1,218
Stop words and international characters with FTS4
Hi AB,
Sorry for so much questions in a single day, since I'm building a new app and I some requirements need to be well thought 
Two questions about using FTS4 with this excellent sqlite framework:
1 - Do you have any tips/info as to support international characters when using FTS4?
2 - Do you have any hints/info as to removing stop words? I see in this article that this can significant reduce the database size.
Thank you again!
Delphi XE4 Pro on Windows 7 64bit.
Lazarus trunk built with fpcupdelux on Windows with cross-compile for Linux 64bit.
Offline
#2 2011-03-09 19:07:23
Re: Stop words and international characters with FTS4
You can have multiple tokenizer in FTS3/FTS4.
But you'll have to create itself.
See http://www.sqlite.org/fts3.html#tokenizer
But it needs to write some low-level C-like code in Delphi.
Not very easy....
About stop words, they should be deleted BEFORE sending the text to the FTS3 engine.
Offline
#3 2011-03-10 10:00:33
- edwinsn
- Member
- Registered: 2010-07-02
- Posts: 1,218
Re: Stop words and international characters with FTS4
Thanks for the link, ab.
Is Synopse SQLITE compiled with the SQLITE_ENABLE_ICU pre-processor symbol defined? Thanks.
Delphi XE4 Pro on Windows 7 64bit.
Lazarus trunk built with fpcupdelux on Windows with cross-compile for Linux 64bit.
Offline
#4 2011-03-10 10:13:12
Re: Stop words and international characters with FTS4
No, it is not, because ICU is a very big part of code, with a lot of tables.
Since we use static linking, the .obj (therefore .exe) files will be much bigger.
And ICU is the "Linux-way" of handling Unicode... there are other native API under Windows, which may be used instead.
But it's not available in SQLite3 by now.
Offline
#5 2011-03-11 04:30:47
- edwinsn
- Member
- Registered: 2010-07-02
- Posts: 1,218
Re: Stop words and international characters with FTS4
I'm wondering if you think that it's a good idea that offering multiple versions of the sqlite3 objects, for example, one with this ICUenabled option defined...
Delphi XE4 Pro on Windows 7 64bit.
Lazarus trunk built with fpcupdelux on Windows with cross-compile for Linux 64bit.
Offline