forked from castorini/anserini
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add regression for MS MARCO doc with per-passage docTTTTTquery expans…
…ions (castorini#1414)
- Loading branch information
Showing
7 changed files
with
171 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| # Anserini: Regressions for MS MARCO Document Ranking | ||
|
|
||
| This page documents regression experiments for the [MS MARCO document ranking task](https://github.com/microsoft/MSMARCO-Document-Ranking) with per-passage docTTTTTquery document expansion, which is integrated into Anserini's regression testing framework. | ||
| For more complete instructions on how to run end-to-end experiments, refer to [this page](https://github.com/castorini/docTTTTTquery#Replicating-MS-MARCO-Document-Ranking-Results-with-Anserini). | ||
|
|
||
| The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/msmarco-doc-docTTTTTquery-per-passage.yaml). | ||
| Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/msmarco-doc-docTTTTTquery-per-passage.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead. | ||
|
|
||
| ## Indexing | ||
|
|
||
| Typical indexing command: | ||
|
|
||
| ``` | ||
| nohup sh target/appassembler/bin/IndexCollection -collection JsonCollection \ | ||
| -input /path/to/msmarco-doc-docTTTTTquery-per-passage \ | ||
| -index indexes/lucene-index.msmarco-doc-docTTTTTquery-per-passage.pos+docvectors+raw \ | ||
| -generator DefaultLuceneDocumentGenerator \ | ||
| -threads 1 -storePositions -storeDocvectors -storeRaw \ | ||
| >& logs/log.msmarco-doc-docTTTTTquery-per-passage & | ||
| ``` | ||
|
|
||
| The directory `/path/to/msmarco-doc-docTTTTTquery-per-passage/` should be a directory containing the expanded document collection; see [this link](https://github.com/castorini/docTTTTTquery#replicating-ms-marco-document-ranking-results-with-anserini) for how to prepare this collection. | ||
|
|
||
| For additional details, see explanation of [common indexing options](common-indexing-options.md). | ||
|
|
||
| ## Retrieval | ||
|
|
||
| Topics and qrels are stored in [`src/main/resources/topics-and-qrels/`](../src/main/resources/topics-and-qrels/). | ||
| The regression experiments here evaluate on the 5193 dev set questions. | ||
|
|
||
| After indexing has completed, you should be able to perform retrieval as follows: | ||
|
|
||
| ``` | ||
| nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.msmarco-doc-docTTTTTquery-per-passage.pos+docvectors+raw \ | ||
| -topicreader TsvInt -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \ | ||
| -output runs/run.msmarco-doc-docTTTTTquery-per-passage.bm25-default.topics.msmarco-doc.dev.txt \ | ||
| -bm25 -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 1000 & | ||
| ``` | ||
|
|
||
| Evaluation can be performed using `trec_eval`: | ||
|
|
||
| ``` | ||
| tools/eval/trec_eval.9.0.4/trec_eval -m map -c -m recall.1000 -c src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt runs/run.msmarco-doc-docTTTTTquery-per-passage.bm25-default.topics.msmarco-doc.dev.txt | ||
| ``` | ||
|
|
||
| ## Effectiveness | ||
|
|
||
| With the above commands, you should be able to replicate the following results: | ||
|
|
||
| MAP | BM25 (Default)| | ||
| :---------------------------------------|-----------| | ||
| [MS MARCO Document Ranking: Dev Queries](https://github.com/microsoft/MSMARCO-Document-Ranking)| 0.3182 | | ||
|
|
||
|
|
||
| R@1000 | BM25 (Default)| | ||
| :---------------------------------------|-----------| | ||
| [MS MARCO Document Ranking: Dev Queries](https://github.com/microsoft/MSMARCO-Document-Ranking)| 0.9490 | | ||
|
|
||
| See [this page](https://github.com/castorini/docTTTTTquery#Replicating-MS-MARCO-Document-Ranking-Results-with-Anserini) for more details. | ||
| Note that here we are using `trec_eval` to evaluate the top 1000 hits for each query; beware, the runs provided by MS MARCO organizers for reranking have only 100 hits per query. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
45 changes: 45 additions & 0 deletions
45
src/main/resources/docgen/templates/msmarco-doc-docTTTTTquery-per-passage.template
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # Anserini: Regressions for MS MARCO Document Ranking | ||
|
|
||
| This page documents regression experiments for the [MS MARCO document ranking task](https://github.com/microsoft/MSMARCO-Document-Ranking) with per-passage docTTTTTquery document expansion, which is integrated into Anserini's regression testing framework. | ||
| For more complete instructions on how to run end-to-end experiments, refer to [this page](https://github.com/castorini/docTTTTTquery#Replicating-MS-MARCO-Document-Ranking-Results-with-Anserini). | ||
|
|
||
| The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/msmarco-doc-docTTTTTquery-per-passage.yaml). | ||
| Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/msmarco-doc-docTTTTTquery-per-passage.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead. | ||
|
|
||
| ## Indexing | ||
|
|
||
| Typical indexing command: | ||
|
|
||
| ``` | ||
| ${index_cmds} | ||
| ``` | ||
|
|
||
| The directory `/path/to/msmarco-doc-docTTTTTquery-per-passage/` should be a directory containing the expanded document collection; see [this link](https://github.com/castorini/docTTTTTquery#replicating-ms-marco-document-ranking-results-with-anserini) for how to prepare this collection. | ||
|
|
||
| For additional details, see explanation of [common indexing options](common-indexing-options.md). | ||
|
|
||
| ## Retrieval | ||
|
|
||
| Topics and qrels are stored in [`src/main/resources/topics-and-qrels/`](../src/main/resources/topics-and-qrels/). | ||
| The regression experiments here evaluate on the 5193 dev set questions. | ||
|
|
||
| After indexing has completed, you should be able to perform retrieval as follows: | ||
|
|
||
| ``` | ||
| ${ranking_cmds} | ||
| ``` | ||
|
|
||
| Evaluation can be performed using `trec_eval`: | ||
|
|
||
| ``` | ||
| ${eval_cmds} | ||
| ``` | ||
|
|
||
| ## Effectiveness | ||
|
|
||
| With the above commands, you should be able to replicate the following results: | ||
|
|
||
| ${effectiveness} | ||
|
|
||
| See [this page](https://github.com/castorini/docTTTTTquery#Replicating-MS-MARCO-Document-Ranking-Results-with-Anserini) for more details. | ||
| Note that here we are using `trec_eval` to evaluate the top 1000 hits for each query; beware, the runs provided by MS MARCO organizers for reranking have only 100 hits per query. |
60 changes: 60 additions & 0 deletions
60
src/main/resources/regression/msmarco-doc-docTTTTTquery-per-passage.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| --- | ||
| name: msmarco-doc-docTTTTTquery-per-passage | ||
| index_command: target/appassembler/bin/IndexCollection | ||
| index_utils_command: target/appassembler/bin/IndexReaderUtils | ||
| search_command: target/appassembler/bin/SearchCollection | ||
| topic_root: src/main/resources/topics-and-qrels/ | ||
| qrels_root: src/main/resources/topics-and-qrels/ | ||
| index_root: | ||
| ranking_root: | ||
| collection: JsonCollection | ||
| generator: DefaultLuceneDocumentGenerator | ||
| threads: 1 | ||
| index_options: | ||
| - -storePositions | ||
| - -storeDocvectors | ||
| - -storeRaw | ||
| topic_reader: TsvInt | ||
| evals: | ||
| - command: tools/eval/trec_eval.9.0.4/trec_eval | ||
| params: | ||
| - -m map | ||
| - -c | ||
| separator: "\t" | ||
| parse_index: 2 | ||
| metric: map | ||
| metric_precision: 4 | ||
| can_combine: true | ||
| - command: tools/eval/trec_eval.9.0.4/trec_eval | ||
| params: | ||
| - -m recall.1000 | ||
| - -c | ||
| separator: "\t" | ||
| parse_index: 2 | ||
| metric: R@1000 | ||
| metric_precision: 4 | ||
| can_combine: true | ||
| input_roots: | ||
| - /tuna1/ # on tuna | ||
| - /store/ # on orca | ||
| - /scratch2/ # on damiano | ||
| input: collections/msmarco/doc-docTTTTTquery-per-passage | ||
| index_path: indexes/lucene-index.msmarco-doc-docTTTTTquery-per-passage.pos+docvectors+raw | ||
| index_stats: | ||
| documents: 20544550 | ||
| documents (non-empty): 20544550 | ||
| total terms: 4203956960 | ||
| topics: | ||
| - name: "[MS MARCO Document Ranking: Dev Queries](https://github.com/microsoft/MSMARCO-Document-Ranking)" | ||
| path: topics.msmarco-doc.dev.txt | ||
| qrel: qrels.msmarco-doc.dev.txt | ||
| models: | ||
| - name: bm25-default | ||
| display: BM25 (Default) | ||
| params: | ||
| - -bm25 -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 1000 | ||
| results: | ||
| map: | ||
| - 0.3182 | ||
| R@1000: | ||
| - 0.9490 |