support Japanese grammar

[iostreamatlab]

Good code.

require 'words_counted'

counter = WordsCounted.count("私の见た寿司の神の映画、だから私も恋に落ちて寿司")
puts counter.token_frequency

then ，puts：

だから私も恋に落ちて寿司
1
私の见た寿司の神の映画
1
[Finished in 0.1s]

[abitdodgy]

Thanks for the feature request. I'll have to have a look to see how the tokeniser works again. But I don't think it should be an issue. If you use a regular expression, can't you tokenise the string as desired?

[iostreamatlab]

Thank you for your reply, I know that there are some simple python regular libraries, so ruby must have, thank you.