Enable stemming and choosing tokenizer, when doing full text search in tantivy #1315
Labels
enhancement
New feature or request
Comments
|
This all sounds good to me. Feel free to make a PR :) |
Merged
wjones127
pushed a commit
that referenced
this issue
Jun 20, 2024
Added the ability to specify tokenizer_name, when creating a full text search index using tantivy. This enables the use of language specific stemming. Also updated the [guide on full text search](https://lancedb.github.io/lancedb/fts/) with a short section on choosing tokenizer. Fixes #1315
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
SDK
Python
Description
Enabling stemming and using a language specific tokenizer tend to improve recall quite a bit, when doing full text search.
Tantivy has support for this through the tokenizer_name argument in add_text_field.
As far as I can tell the change needed is to add tokenizer_name argument to the following line
And then add the tokenizer_name argument to the create_fts_index method.
I would personally really prefer if the argument could be exposed instead of just enabling the usage of the english stemmer. Tantivy supports a few different language tokenizers, which I think a lot of people would like to use instead of english
I can create a pull request with the suggested changes if you think it is a good idea :-).
The text was updated successfully, but these errors were encountered: