Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Features/add korean & japanese "kagome" tokenizers #5023

Draft
wants to merge 19 commits into
base: main
Choose a base branch
from
Draft

Features/add korean & japanese "kagome" tokenizers #5023

wants to merge 19 commits into from

Conversation

databyjp
Copy link
Contributor

What's being changed:

  • Add the "Kagome" tokenizer for Korean & Japanese text tokenization

Review checklist

  • Documentation has been updated, if necessary. Link to changed documentation:
  • Chaos pipeline run or not necessary. Link to pipeline:
  • All new code is covered by tests where it is reasonable.
  • Performance tests have been run or not necessary.

@databyjp databyjp requested a review from donomii May 27, 2024 13:42
databyjp and others added 17 commits May 27, 2024 16:31
@databyjp
Generate code from OpenAPI spec
fb31079
@databyjp
Reuse dict objects
2c9f955
@databyjp
typo fix
54c6502
@donomii
re-import modules
63e7d00
@databyjp
Refactor tokenization functions to reuse tokenizer instances
c7b2034
@databyjp
Merge branch 'features/add-korean-tokenizer-inv-index' of github.com:…
ea7c4a6 
…semi-technologies/weaviate into features/add-korean-tokenizer-inv-index
@databyjp
Update expected Korean tokenization in tokenizer_test.go
dc734b3
@databyjp
Further streamline tokenizer
6022e26
@databyjp
Split initialisers; add them to init
ead70fc
@databyjp
Refactor tokenizer initialization to improve code reusability
665be4a
@databyjp
Refactor Kagome tokenizers to only instantiate if env var set, or sel…
23cf76a 
…ected.
@databyjp
gofumpt
ccdb59e
@donomii
Merge branch 'main' into features/add-korean-tokenizer-inv-index
a0e5bad
@donomii
Merge branch 'main' into features/add-korean-tokenizer-inv-index
bac7e38
@databyjp
Merge branch 'refs/heads/main' into features/add-korean-tokenizer-inv…
35381c1 
…-index
@parkerduckworth
resolve merge conflicts with main
49c0d22
@parkerduckworth
increase weaviate testcontainer timeout
ac9f465
@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Jun 12, 2024

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@donomii donomii Awaiting requested review from donomii

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@databyjp @donomii @parkerduckworth