-
Notifications
You must be signed in to change notification settings - Fork 17.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/text/collate: Compare and CompareString broken in "ka-shifted" collation mode #68166
Comments
Similar Issues
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.) |
Pretty simple bug: https://cs.opensource.google/go/x/text/+/refs/tags/v0.16.0:collate/collate.go;l=166 . When the collator is in the shifted alternate mode, Looking at the history, x/text/collate seems unmaintained (last change that wasn't fmts or Unicode upgrades was 2016 :( ). Given this, my suggestion tactically would be to rip out the custom codepaths for Compare/CompareString, and have them generate and bytes.Compare full collation keys. It's less efficient for one-off compares, but I think it would be correct, and a future maintainer can always make it faster again since it's all internal package gubbins. |
CC @mpvl |
I hacked up a more comprehensive debugging tool: https://go.dev/play/p/GcsuboreVW8 . For a bunch of comparisons and collations, it prints:
There's also main() in there suitable for running from the CLI as well. What I reported above can be seen in the Further weirdness I discovered while writing the tool:
Summary:
I have no idea where these new behaviors come from, I haven't paged in enough of the code to understand what's happening. |
Looking more closely at UTS 35, the multi-key language tags above are not well formed because multiple I tried collations |
Correction: Given that numeric sorting is a wrapper around the standard weight, I'm assuming that it lacks awareness of |
Go version
go version go1.22.3 linux/amd64
Output of
go env
in your module/workspace:What did you do?
Tried to compare strings with the "shifted" variable weighting option, which you can select with e.g. language tag
en-u-ka-shifted
.Demo: https://go.dev/play/p/afAOLzqh3Oe
What did you see happen?
99% of Compare() and CompareString() calls on different strings return 0.
What did you expect to see?
Comparison results that match the configured collation, or at least a courtesy panic letting me know it's known broken (spoiler alert...)
The text was updated successfully, but these errors were encountered: