https://github.com/LanguageMachines/uctodata/
Website and documentation: https://languagemachines.github.io/ucto
uctodata provides datafiles for the tokeniser ucto for several languages. The
language code can be supplied to ucto using the -L paramater (e.g. ucto -L nld input.txt):
eng- Englishnld- Dutchdeu- Germanfra- Frenchita- Italianspa- Spanishpor- Portugueserus- Russianswe- Swedishtur- Turkishfry- Frisian
uctodata is architecture independent.
To install uctodata, first consult whether your distribution's package manager has an up-to-date package. If not, for easy installation of ucto and uctodata, it is included as part of our software distribution LaMachine: https://proycon.github.io/LaMachine .
To compile and install manually from source instead:
$ bash bootstrap.sh
$ ./configure
$ make
$ make install