Add/expose WordTokenizer transform

[ganik]

[pieths]

WordTokenizer is currently blocked by the variable length vector column type not supported in NimbusML issue.

Here is a branch which contains a working set of modifications which demonstrates the issue. The example includes passing in a custom character separator. See below.

import pandas
from nimbusml import Pipeline, Role
from nimbusml.preprocessing.schema import ColumnConcatenator
from nimbusml.preprocessing.text import WordTokenizer

# create the data
customer_reviews = pandas.DataFrame(data=dict(review=[
    "I really did not like the taste of it",
    "It was surprisingly quite good!",
    "I will never ever ever go to that place again!!",
    "The best ever!! It was amazingly good and super fast",
    "I wish I had gone earlier, it was that great",
    "somewhat dissapointing. I'd probably wont try again",
    "Never visit again... rascals!"]))

tokenize = WordTokenizer(char_array_term_separators=[" ", "n"]) << ['review']
pipeline = Pipeline([tokenize])
tokenize.fit(customer_reviews)
y = tokenize.transform(customer_reviews)

and its output:

[I, really, did, ot, like, the, taste, of, it, ]
[It, was, surprisi, gly, quite, good!, ]
[I, will, ever, ever, ever, go, to, that, place, agai, !!, ]
[The, best, ever!!, It, was, amazi, gly, good, a, d, super, fast, ]
[I, wish, I, had, go, e, earlier,, it, was, that, great, ]
[somewhat, dissapoi, ti, g., I'd, probably, wo, t, try, agai, ]
[Never, visit, agai, ..., rascals!, ]

Error: *** System.NotSupportedException: 'Column has variable length vector: review. Not supported in python. Drop column before sending to Python'Traceback (most recent call last):