mORMot Open Source

zed

Member

From: Belarus

Registered: 2015-02-26

Posts: 105

FTS5 rises error: no such tokenizer: simple

mORMot version: 1.18.6171

Test code:

uses
  System.SysUtils,
  SynCommons,
  SynSQLite3Static,
  mORMot,
  mORMotSQLite3;

type
  TSQLTextRecord = class(TSQLRecordFTS5)
  protected
    FText: RawUTF8;
  published
    property Text: RawUTF8 read FText write FText;
  end;

var
  VModel: TSQLModel;
  VClient: TSQLRestClientDB;
begin
  Writeln(SYNOPSE_FRAMEWORK_FULLVERSION);
  try
    VModel := TSQLModel.Create([TSQLTextRecord]);
    VClient := TSQLRestClientDB.Create(VModel, nil, 'fts5.db3', TSQLRestServerDB);
    try
      VClient.Server.CreateMissingTables;
      // ...
    finally
      VClient.Free;
      VModel.Free;
    end;
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

rises exception:

ESQLite3Exception: Error SQLITE_ERROR (1) [Step] using 3.33.0 - no such tokenizer: simple, extended_errcode=1

Same code works fine with FTS3 and FTS4.

FTS5 have three built in tokenizers:

- The unicode61 tokenizer, based on the Unicode 6.1 standard. This is the default.
- The ascii tokenizer, which assumes all characters outside of the ASCII codepoint range (0-127) are to be treated as token characters.
- The porter tokenizer, which implements the porter stemming algorithm.

It seems, that default "simple" tokenizer was renamed to "ascii" in FTS5 and it isn't a default anymore.

Last edited by zed (2020-11-24 14:57:40)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,655

Website

Re: FTS5 rises error: no such tokenizer: simple

Nice catch.

Please check https://synopse.info/fossil/info/b515edfb7a

Thanks!