To search Non-Latin characters(CJK for example) in SQLite, https://www.sqlite.org/fts5.html and a custom tokenizer are required since the 3 built-in tokenizers can not handle CJK search correctly.
I wonder if it is possible to provide a tokenizer such as https://icu.unicode.org/ to support search in CJK characters?
Thanks
5 Likes