weird documents in COVID-19 Open Research Dataset paragraph index

[stephaniewhoo]

While iterating through the paragraph index( lucene-index-cord19-paragraph-2020-04-17), some outliers are identified.

1. There are files e.g.(docid: 'ij3ncdb' ,https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4896250/) has a really long abstract and the abstract is appended to each paragraph index caused the index file size explosion.

2. There is a weird file. Cord-uid is 'hwjkbpqp', doi is '10.1007/s12529-010-9106-9'.
   This document has an empty abstract and meanwhile has 5849 paragraphs. It appears as an extreme outlier regarding to the size per index.

[hscells]

I also found that the paragraph index is strange: for some reason the title and abstract are concatenated to each paragraph. Is there a reason for this @lintool?

[lintool]

Yes, the concatenation is intentional: https://github.com/castorini/anserini/blob/master/docs/experiments-covid.md#paragraph

See: https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-10-46

Prepending title definitely helps, we believe (based on eyeballing results). Unclear whether title + paragraph > title + abstract + paragraph.

[lintool]

• ij3ncdb was noted in New indexes CORD-19 04/17 data drop #1109
• hwjkbpqp is a new observation.