A re-implementation of redpony/cdec's tokenize-anything.pl script in python
samples/ is a bunch of data pulled from Wikipedia in a bunch of languages.
tok/ holds the same data, as tokenized by the original tokenize-anything.pl.
-
Notifications
You must be signed in to change notification settings - Fork 2
armatthews/TokenizeAnything
About
A re-implementation of redpony/cdec's tokenize-anything.pl script in python
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published