Provide Pre Sentence Split and Tokenized Data for Input to Model

[jrsperry]

I have an NLP stack where I currently already run sentence splitting and tokenization (in addition to many other tasks) and would like to run the SpanBertCorefModel for my coreference.

Since I already have sentences and tokens, is there are way to provide them as inputs to the model? This would help me line up the results of the coreference with the rest of my NLP data as I wouldn't need to worry about doing any complicated alignment checks.

I've tried the following already in python but got an error as I'm new to spark-nlp and took my best guess at what would make sense.

import sparknlp
from sparknlp.annotator import SpanBertCorefModel

spark = sparknlp.start(m1=True)

corefResolution = SpanBertCorefModel().pretrained("spanbert_base_coref").setInputCols(["sentences", "tokens"]).setOutputCol("corefs")

data_dict = [{'sentences': ['Mary was mad at John when he broke her phone'], 'tokens': ['Mary', 'was', 'mad', 'at', 'John', 'when', 'he', 'broke', 'her', 'phone']}]
test_input = spark.createDataFrame(data_dict)
coref_result = corefResolution.transform(test_input)

*** Error
pyspark.sql.utils.IllegalArgumentException: requirement failed: Wrong or missing inputCols annotators in SPANBERTCOREFMODEL_20b2660eea55.
Current inputCols: sentences,tokens. Dataset's columns:
(column_name=sentences,is_nlp_annotator=false)
(column_name=tokens,is_nlp_annotator=false).

I have been able to successfully run the example for this model as detailed in the documentation, using the Pipeline. Thanks in advance

[maziyarpanahi]

Hi @jrsperry
There is no straightforward to do this, we have to wrap the existing sentence/tokens around Annotations so they are ready for the next stags in the pipeline. (it is totally possible)

But I am thinking of making this part a bit easier, so maybe a feature request can come out of this question. (How to use sentences / tokens coming from outside in Spark NLP)

Just out of curiosity, are those sentences/tokens generated by spaCy? (just wondering which format we can accept and simply run them through a function to get an actual SENTENCE/TOKEN in Spark NLP pipeline so users can continue with their own tokens)

[jrsperry]

I sentence split and tokenize with corenlp, but I store it separately in a custom protobuf. Ideally in my mind, for a given document supplying a list of strings representing the sentences, and a list of list of strings representing the tokens would make providing your own sentences/tokens straightforward. This would be more 'sentence' centric.

Example:
Document Text: John went to the store last night. He bought some bread.
Sentences: ['John went to the store last night.', 'He bought some bread.']
Tokens: [['John', 'went', 'to', 'the', 'store', 'last', 'night', '.'], ['He', 'bought', 'some', 'bread', '.']]

Alternatively, if you wanted to mimic spacy more closely you could pass in only a list of tokens, along with a list of tokens denoting whether the tokens is the start of a sentence. That's how spacy supports creating a doc from already sentence-split and tokenized inputs. This would be more 'token' centric.

Tokens: [John', 'went', 'to', 'the', 'store', 'last', 'night', '.', 'He', 'bought', 'some', 'bread', '.']
Tokens_Sent_Start = [True, False, False, False, False, False, False, False, True, False, False, False, False]

I'm not as familiar with which approach would make most sense for spark-nlp, but either of these approaches would be great in my mind.

Do you have a code example of how to add those annotations? or just a link to those annotations which I would need to wrap the text with?

[maziyarpanahi]

Thanks @jrsperry for a detailed examples. I think we might be able to support this as a feature (bring your own tokens) and then detect sentence's boundaries from either each item of the lists. (we'll make this possible to be read from a column in a DataFrame and transform it into what SentenceDetector and Tokenizer annotators would have outputted)

I'll add this as a feature request

[jrsperry]

Awesome. This would help me out a ton in my use case.