Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why deacc? #19

Open
emillundhcodemill opened this issue Feb 1, 2018 · 3 comments
Open

Why deacc? #19

emillundhcodemill opened this issue Feb 1, 2018 · 3 comments

Comments

@emillundhcodemill
Copy link
Contributor

I see that the parameter deacc is set to True for all languages when tokenize is called from clean_text_by_word and tokenize_by_word. This means that in Nordic languages, characters with umlaut like å, ä, ö are rendered without umlaut as a, a, o; this drastically reduces the quality of the keyword extraction for Swedish which makes heavy use of umlauts and where the umlauted characters count as characters in their own right (For example snö=snow, sno=twist; äga=own, aga=spank).
AFAICS, it's easy to set deacc=False and obtain much nicer results for Swedish; it should probably be done on a per-language basis (I suggest False for danish, finnish, german, norwegian, and swedish).
I'll be happy to file a PR but I'd like to hear if you have any comments?

@fbarrios
Copy link
Contributor

fbarrios commented Feb 2, 2018

Hey @emillundhcodemill, thanks for your contributions!

I actually don't recall why the deaccent is always set to true. In our mother tongue (Spanish) few words can be written with or without the accent (solo, for instance). In most cases, though, the accent mark is used consistently to show intonation or to differentiate meaning between one syllable words that are spelled the same.

I would suggest to default deacc to False in all cases and to make it a parameter in the keywords method. What do you think @fedelopez77?

@fedelopez77
Copy link
Contributor

I totally agree with @fbarrios. Honestly we don't remember why we left that with True.

@emillundhcodemill if you could just set it to False by default, and add it as an optional parameter in the keywords method, we would be more than happy to merge your contribution.

Thanks in advance!

@emillundhcodemill
Copy link
Contributor Author

OK, done. #22

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants