LuceneSearcher.from_prebuilt_index returns empty contents

[joelrorseth]

Hi there, I'm using one of the pre-built indexes via LuceneSearcher.from_prebuilt_index('msmarco-passage'). I noticed that the contents field of each JLuceneSearcherResult returned by the search() method is not populated, however, the raw field is populated (and appears to contain the entire contents encoded in JSON).

Do the prebuilt indexes store the full document contents? If so, is there any way to access them without decoding the raw JSON myself)? I noticed there is a -storeContents arg for the actual indexer, but I figure that isn't exposed in the pre-built methods.

Thanks!

[lintool]

raw stores the raw document in its original format, contents stores the "parsed" document. So, for example, raw might give the original HTML doc, and contents provides what's actually indexed after tag cleanup. Thus, you're always able to reconstruct contents form raw (i.e., just re-parse the document), but not vice versa. For this reason, we only store raw in the prebuilt indexes.

In this case, you get contents from raw by parsing out the JSON and pulling out the right field.

If you want an index with contents but not raw, you'll have to build a fresh index yourself.

Hope this helps!

[joelrorseth]

Thanks @lintool, that clears things up!