Searching within a subset of documents

[agmontpetit]

I've gone over the intro2new.ipynb notebook, and I'm wondering if it is possible to accomplish the following:

1. Map passages from a collection to their passage_id. As far as I can tell, when calling indexer.index(...), there is nothing returned. I'd need a way to link the original passage text with its passage_id.
2. Restrict a search over a subset of passage ids. The use case would be to only search within the passages from a large document.

[detaos]

The first part is easy. The passages you pass to the indexer either already have them from the TSV or are given a sequential ID.

TSV Format we use:
[ID] \t [Passage Text] \t [Passage title / other meta data]

So, you don't get the passage IDs from the indexer. You give the passage IDs to the indexer. That way you know them.

The second part is best done with filtering. I'm assuming you have some way to do the subsetting. We do that with page metadata. Here's the search line to pass a filtering function:

        results = searcher.search(query.query, k=query.k, filter_fn=lambda pids: torch.tensor(
            [index for index in pids.numpy().tolist() if keepResult(query.conditions, index)], dtype=pids.dtype))

The keepResult function takes the "conditions", which for us is a JSON object representing the filter, and the passage ID, which we use to pull the metadata for that passage. It returns true of the passage's metadata meets the filter criteria, or false otherwise.

[agmontpetit]

Thanks for the answer. It looks straightforward to implement.