Stop words and international characters with FTS4

edwinsn · 2011-03-09 15:36:19

Hi AB,

Sorry for so much questions in a single day, since I'm building a new app and I some requirements need to be well thought

Two questions about using FTS4 with this excellent sqlite framework:

1 - Do you have any tips/info as to support international characters when using FTS4?

2 - Do you have any hints/info as to removing stop words? I see in this article that this can significant reduce the database size.

Thank you again!

ab · 2011-03-09 19:07:23

You can have multiple tokenizer in FTS3/FTS4.
But you'll have to create itself.
See http://www.sqlite.org/fts3.html#tokenizer

But it needs to write some low-level C-like code in Delphi.
Not very easy....

About stop words, they should be deleted BEFORE sending the text to the FTS3 engine.

edwinsn · 2011-03-10 10:00:33

Thanks for the link, ab.

Is Synopse SQLITE compiled with the SQLITE_ENABLE_ICU pre-processor symbol defined? Thanks.

ab · 2011-03-10 10:13:12

No, it is not, because ICU is a very big part of code, with a lot of tables.
Since we use static linking, the .obj (therefore .exe) files will be much bigger.

And ICU is the "Linux-way" of handling Unicode... there are other native API under Windows, which may be used instead.
But it's not available in SQLite3 by now.

edwinsn · 2011-03-11 04:30:47

I'm wondering if you think that it's a good idea that offering multiple versions of the sqlite3 objects, for example, one with this ICUenabled option defined...

ab · 2011-03-11 08:31:21

As I wrote above, IMHO under Windows ICU is not a good option.

mORMot Open Source

#1 2011-03-09 15:36:19

Stop words and international characters with FTS4

#2 2011-03-09 19:07:23

Re: Stop words and international characters with FTS4

#3 2011-03-10 10:00:33

Re: Stop words and international characters with FTS4

#4 2011-03-10 10:13:12

Re: Stop words and international characters with FTS4

#5 2011-03-11 04:30:47

Re: Stop words and international characters with FTS4

#6 2011-03-11 08:31:21

Re: Stop words and international characters with FTS4

Board footer