Skip to content

Add Ukrainian stemmer vocabulary data #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
polaz wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Add Ukrainian stemmer vocabulary data #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
f8096b1
Add Ukrainian stemmer vocabulary data
polaz Apr 12, 2026
aabdee1
Ukrainian: add мигтіння and боління to test corpus
polaz Apr 23, 2026
61bdbd6
Ukrainian: add 65 words from stemmer_tests.tsv to corpus
polaz Apr 23, 2026
6faac6e
test(ukrainian): add лошатко for -атко diminutive and refresh output
polaz Apr 24, 2026
399976e
test(ukrainian): add diminutive suffix forms missing from corpus
polaz Apr 24, 2026
9232861
test(ukrainian): add words for diminutive_stem suffix paths
polaz Apr 24, 2026
8e4c923
test(ukrainian): add братунь for diminutive -унь suffix
polaz Apr 24, 2026
98d9489
fix(ukrainian): update expected output after -онн- compound adj fix
polaz Apr 24, 2026
d8fd349
feat(ukrainian): add participle neuter test words живуче and зростаюче
polaz Apr 24, 2026
b71a27d
fix(ukrainian): update expected output for adj R2-blocking fix and pV…
polaz May 6, 2026
b402640
test(ukrainian): update expected output for -ості/-остей oblique stems
polaz May 6, 2026
8c9b5a7
test(ukrainian): update вартості expected stem (вартост, not варт)
polaz May 6, 2026
9576b20
test(ukrainian): update expected output for derivational setlimit fix
polaz Jun 6, 2026
473f82a
test(ukrainian): add product vocabulary and -нти output
polaz Jun 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions ukrainian/COPYING
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
voc.txt and output.txt are licensed as CC BY-SA 3.0.

@ojwb ojwb Jun 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are they?

Taking a large dataset and extracting the most frequent N words is clearly highly transformative and it's not clear to me that the licence of the original text applies to the wordlist. (It's also not clear to me that it automatically doesn't, but apparently an LLM can ignore licensing of code with impunity despite being capable of regenerating significant verbatim sections of the original, whereas you couldn't recreate even one sentence of a wikipedia article working solely from this word list.)

So that's the reason why for the other languages I've just stated how the source data was licensed. Perhaps I should seek a qualified legal opinion on this.


voc.txt was generated from a dump of the Ukrainian Wikipedia like so:

perl -CS scripts/wikipedia-dump-to-freq ukwiki-latest-pages-articles.xml.bz2 10 Cyrillic > voc.freq
scripts/freq-to-voc 300 < voc.freq | grep -v "^'" > ukrainian/voc.txt

The dump used was dated 2025-01-02.

output.txt was generated from voc.txt by running it through the stemmer:

stemwords -l ukrainian -c UTF_8 -i ukrainian/voc.txt -o ukrainian/output.txt

Wikipedia is licensed as: https://creativecommons.org/licenses/by-sa/3.0/
Loading