Extracting features on for long sequences / SQuAD

[ethanjperez]

Would it be difficult for you to release a feature extractor that works for sequences longer than 512, in the same way long SQuAD passages are handled? That would be quite helpful for being able to train small models on longer sequences (e.g., some SQuAD passages) and especially tasks (e.g. in summarization) that were previously prohibitively long for those without much compute.

Also, for SQuAD specifically, do you know how many passages are omitted for being longer than 512 tokens?

Thanks for your help and for releasing the pre-trained weights and code. This is a great service to the community 👍

[jacobdevlin-google]

For SQuAD we actually used 384 and about 5% of passages were longer than that. My recommendation would be to modify extract_features.py to take a sliding window approach. So if you have:

the man went to the store and bought a gallon of milk.

With a maximum length of 6, you don't want to do;

the man went to the store
and bought a gallon of milk

Because then the words on the edge wont have context. Instead what you want to do is have some overlap window:

the man went to the store
to the store and bought a
and bought a gallon of milk

And then take the representation of each word with maximal context. (E.g., the version of store in the second sentence has maximum context).

You can see run_squad.py for how we do it in the fine-tuning method, and then apply the same technique in extract_features.py.

We will probably not release a feature extractor do this specifically because there are too many use cases and details to get this right in every case (e.g., SQuAD is different from other scenarios because it needs to include the question. But others might not need to.)

[ethanjperez]

Okay got it, that makes sense. Thanks for your help!

[wayfarerjing]

I understand that in the Squad task sliding window technique can be applied. But what if it is a classification task? How should ground-truth label be assigned to sub-texts? I don't think assigning the same label as its original long text very harmful. But I don't think it's reasonable either.

Have you been able to figure this out?
Thank you.

[billiebaechan]

Hi there,

Does anyone have advice on applying the sliding window technique for classification? As @wayfarerjing commented above, assigning the same label to all the windows produced from the same long body of text might be detrimental to the performance since it produces an unbalanced distribution of labels in the training set.

I appreciate any advice you might have. Thank you.

[MatthijsRijlaarsdam]

Hi there,

We've made a sliding window approach that somewhat works for classification, at least for non-training purposes.

We have made our own dataset of (long) scientific papers and multiple choice questions that we want BERT to answer. Because of the labelling issue @billiebaechan mentions, we train it on the RACE dataset, which has sequences that are shorter than 512. For our own dataset, we split all the input texts in snippets of 350 characters with a stride of 35, keeping the rest for our questions and answers. The labels are kept the same for all snippets. We then print the questions+given answers+logits to a file, and keep as a final prediction the answer with the highest logit. This improves our results quite a bit (from 32 to 39% correct), although our dataset might still be a bit too hard for BERT right now (best result we got is around 50%).

Hope this helps at least somewhat.

[jind11]

@MatthijsRijlaarsdam hi, thanks for your answer. did you average the logits obtained from each snippet and then do the prediction of the answer with the highest logit? for this case, would it possible that the improvement comes from the ensemble effects?