diff --git a/README.md b/README.md index 7b2f035400..600f95b290 100644 --- a/README.md +++ b/README.md @@ -323,8 +323,8 @@ See individual pages for details. | BGE-base-en-v1.5 w/ HNSW fp32 (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-onnx.md) | | BGE-base-en-v1.5 w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8.md) | | BGE-base-en-v1.5 w/ HNSW int8 (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.md) | -| OpenAI Ada2 w/ HNSW fp32 | [✓](docs/regressions/regressions-msmarco-passage-openai-ada2.md) | [✓](docs/regressions/regressions-dl19-passage-openai-ada2.md) | [✓](docs/regressions/regressions-dl20-passage-openai-ada2.md) | -| OpenAI Ada2 w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-passage-openai-ada2-int8.md) | [✓](docs/regressions/regressions-dl19-passage-openai-ada2-int8.md) | [✓](docs/regressions/regressions-dl20-passage-openai-ada2-int8.md) | +| OpenAI Ada2 w/ HNSW fp32 | [✓](docs/regressions/regressions-msmarco-v1-passage.openai-ada2.hnsw.md) | [✓](docs/regressions/regressions-dl19-passage.openai-ada2.hnsw.md) | [✓](docs/regressions/regressions-dl20-passage.openai-ada2.hnsw.md) | +| OpenAI Ada2 w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-v1-passage.openai-ada2.hnsw-int8.md) | [✓](docs/regressions/regressions-dl19-passage.openai-ada2.hnsw-int8.md) | [✓](docs/regressions/regressions-dl20-passage.openai-ada2.hnsw-int8.md) | | Cohere English v3.0 w/ HNSW fp32 | [✓](docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3.0-hnsw.md) | [✓](docs/regressions/regressions-dl19-passage-cohere-embed-english-v3.0-hnsw.md) | [✓](docs/regressions/regressions-dl20-passage-cohere-embed-english-v3.0-hnsw.md) | | Cohere English v3.0 w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3.0-hnsw-int8.md) | [✓](docs/regressions/regressions-dl19-passage-cohere-embed-english-v3.0-hnsw-int8.md) | [✓](docs/regressions/regressions-dl20-passage-cohere-embed-english-v3.0-hnsw-int8.md) | | **Learned Dense** (Inverted; experimental) | | | | diff --git a/docs/regressions.md b/docs/regressions.md index 73639d3dc5..bfa9bc528d 100644 --- a/docs/regressions.md +++ b/docs/regressions.md @@ -48,23 +48,23 @@ nohup python src/main/python/run_regression.py --index --verify --search --regre nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-unicoil-tilde-expansion >& logs/log.msmarco-passage-unicoil-tilde-expansion & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-distill-splade-max >& logs/log.msmarco-passage-distill-splade-max & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-splade-pp-ed >& logs/log.msmarco-passage-splade-pp-ed & +nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-splade-pp-ed-onnx >& logs/log.msmarco-passage-splade-pp-ed-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-splade-pp-sd >& logs/log.msmarco-passage-splade-pp-sd & +nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-splade-pp-sd-onnx >& logs/log.msmarco-passage-splade-pp-sd-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-hnsw >& logs/log.msmarco-passage-cos-dpr-distil-hnsw & +nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-hnsw-onnx >& logs/log.msmarco-passage-cos-dpr-distil-hnsw-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-hnsw-int8 >& logs/log.msmarco-passage-cos-dpr-distil-hnsw-int8 & +nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-hnsw-int8-onnx >& logs/log.msmarco-passage-cos-dpr-distil-hnsw-int8-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-fw >& logs/log.msmarco-passage-cos-dpr-distil-fw & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-lexlsh >& logs/log.msmarco-passage-cos-dpr-distil-lexlsh & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-bge-base-en-v1.5-hnsw >& logs/log.msmarco-passage-bge-base-en-v1.5-hnsw & +nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-bge-base-en-v1.5-hnsw-onnx >& logs/log.msmarco-passage-bge-base-en-v1.5-hnsw-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-bge-base-en-v1.5-hnsw-int8 >& logs/log.msmarco-passage-bge-base-en-v1.5-hnsw-int8 & +nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx >& logs/log.msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cohere-embed-english-v3.0-hnsw >& logs/log.msmarco-passage-cohere-embed-english-v3.0-hnsw & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cohere-embed-english-v3.0-hnsw-int8 >& logs/log.msmarco-passage-cohere-embed-english-v3.0-hnsw-int8 & -nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2 >& logs/log.msmarco-passage-openai-ada2 & - -nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-splade-pp-ed-onnx >& logs/log.msmarco-passage-splade-pp-ed-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-splade-pp-sd-onnx >& logs/log.msmarco-passage-splade-pp-sd-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-hnsw-onnx >& logs/log.msmarco-passage-cos-dpr-distil-hnsw-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-hnsw-int8-onnx >& logs/log.msmarco-passage-cos-dpr-distil-hnsw-int8-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-bge-base-en-v1.5-hnsw-onnx >& logs/log.msmarco-passage-bge-base-en-v1.5-hnsw-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx >& logs/log.msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx & +nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2-hnsw >& logs/log.msmarco-passage-openai-ada2-hnsw & +nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2-hnsw-int8 >& logs/log.msmarco-passage-openai-ada2-hnsw-int8 & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-doc >& logs/log.msmarco-doc & nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-doc-wp >& logs/log.msmarco-doc-wp & @@ -87,24 +87,24 @@ nohup python src/main/python/run_regression.py --index --verify --search --regre nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-unicoil >& logs/log.dl19-passage-unicoil & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-unicoil-noexp >& logs/log.dl19-passage-unicoil-noexp & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-splade-pp-ed >& logs/log.dl19-passage-splade-pp-ed & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-splade-pp-ed-onnx >& logs/log.dl19-passage-splade-pp-ed-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-splade-pp-sd >& logs/log.dl19-passage-splade-pp-sd & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-splade-pp-sd-onnx >& logs/log.dl19-passage-splade-pp-sd-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cos-dpr-distil-hnsw >& logs/log.dl19-passage-cos-dpr-distil-hnsw & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cos-dpr-distil-hnsw-onnx >& logs/log.dl19-passage-cos-dpr-distil-hnsw-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cos-dpr-distil-hnsw-int8 >& logs/log.dl19-passage-cos-dpr-distil-hnsw-int8 & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cos-dpr-distil-hnsw-int8-onnx >& logs/log.dl19-passage-cos-dpr-distil-hnsw-int8-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cos-dpr-distil-fw >& logs/log.dl19-passage-cos-dpr-distil-fw & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cos-dpr-distil-lexlsh >& logs/log.dl19-passage-cos-dpr-distil-lexlsh & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-bge-base-en-v1.5-hnsw >& logs/log.dl19-passage-bge-base-en-v1.5-hnsw & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-bge-base-en-v1.5-hnsw-onnx >& logs/log.dl19-passage-bge-base-en-v1.5-hnsw-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-bge-base-en-v1.5-hnsw-int8 >& logs/log.dl19-passage-bge-base-en-v1.5-hnsw-int8 & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-openai-ada2 >& logs/log.dl19-passage-openai-ada2 & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx >& logs/log.dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-openai-ada2-hnsw >& logs/log.dl19-passage-openai-ada2-hnsw & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-openai-ada2-hnsw-int8 >& logs/log.dl19-passage-openai-ada2-hnsw-int8 & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cohere-embed-english-v3.0-hnsw >& logs/log.dl19-passage-cohere-embed-english-v3.0-hnsw & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cohere-embed-english-v3.0-hnsw-int8 >& logs/log.dl19-passage-cohere-embed-english-v3.0-hnsw-int8 & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-splade-pp-ed-onnx >& logs/log.dl19-passage-splade-pp-ed-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-splade-pp-sd-onnx >& logs/log.dl19-passage-splade-pp-sd-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cos-dpr-distil-hnsw-onnx >& logs/log.dl19-passage-cos-dpr-distil-hnsw-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cos-dpr-distil-hnsw-int8-onnx >& logs/log.dl19-passage-cos-dpr-distil-hnsw-int8-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-bge-base-en-v1.5-hnsw-onnx >& logs/log.dl19-passage-bge-base-en-v1.5-hnsw-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx >& logs/log.dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx & - nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-doc >& logs/log.dl19-doc & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-doc-wp >& logs/log.dl19-doc-wp & nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-doc-hgf-wp >& logs/log.dl19-doc-hgf-wp & @@ -126,24 +126,24 @@ nohup python src/main/python/run_regression.py --index --verify --search --regre nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-unicoil >& logs/log.dl20-passage-unicoil & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-unicoil-noexp >& logs/log.dl20-passage-unicoil-noexp & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-splade-pp-ed >& logs/log.dl20-passage-splade-pp-ed & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-splade-pp-ed-onnx >& logs/log.dl20-passage-splade-pp-ed-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-splade-pp-sd >& logs/log.dl20-passage-splade-pp-sd & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-splade-pp-sd-onnx >& logs/log.dl20-passage-splade-pp-sd-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cos-dpr-distil-hnsw >& logs/log.dl20-passage-cos-dpr-distil-hnsw & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cos-dpr-distil-hnsw-onnx >& logs/log.dl20-passage-cos-dpr-distil-hnsw-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cos-dpr-distil-hnsw-int8 >& logs/log.dl20-passage-cos-dpr-distil-hnsw-int8 & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cos-dpr-distil-hnsw-int8-onnx >& logs/log.dl20-passage-cos-dpr-distil-hnsw-int8-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cos-dpr-distil-fw >& logs/log.dl20-passage-cos-dpr-distil-fw & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cos-dpr-distil-lexlsh >& logs/log.dl20-passage-cos-dpr-distil-lexlsh & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-bge-base-en-v1.5-hnsw >& logs/log.dl20-passage-bge-base-en-v1.5-hnsw & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-bge-base-en-v1.5-hnsw-onnx >& logs/log.dl20-passage-bge-base-en-v1.5-hnsw-onnx & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-bge-base-en-v1.5-hnsw-int8 >& logs/log.dl20-passage-bge-base-en-v1.5-hnsw-int8 & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-openai-ada2 >& logs/log.dl20-passage-openai-ada2 & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx >& logs/log.dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-openai-ada2-hnsw >& logs/log.dl20-passage-openai-ada2-hnsw & +nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-openai-ada2-hnsw-int8 >& logs/log.dl20-passage-openai-ada2-hnsw-int8 & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cohere-embed-english-v3.0-hnsw >& logs/log.dl20-passage-cohere-embed-english-v3.0-hnsw & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cohere-embed-english-v3.0-hnsw-int8 >& logs/log.dl20-passage-cohere-embed-english-v3.0-hnsw-int8 & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-splade-pp-ed-onnx >& logs/log.dl20-passage-splade-pp-ed-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-splade-pp-sd-onnx >& logs/log.dl20-passage-splade-pp-sd-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cos-dpr-distil-hnsw-onnx >& logs/log.dl20-passage-cos-dpr-distil-hnsw-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cos-dpr-distil-hnsw-int8-onnx >& logs/log.dl20-passage-cos-dpr-distil-hnsw-int8-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-bge-base-en-v1.5-hnsw-onnx >& logs/log.dl20-passage-bge-base-en-v1.5-hnsw-onnx & -nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx >& logs/log.dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx & - nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-doc >& logs/log.dl20-doc & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-doc-wp >& logs/log.dl20-doc-wp & nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-doc-hgf-wp >& logs/log.dl20-doc-hgf-wp & diff --git a/docs/regressions/regressions-dl19-passage-openai-ada2-int8.md b/docs/regressions/regressions-dl19-passage.openai-ada2.hnsw-int8.md similarity index 86% rename from docs/regressions/regressions-dl19-passage-openai-ada2-int8.md rename to docs/regressions/regressions-dl19-passage.openai-ada2.hnsw-int8.md index 5dca5b05f5..587bd66d83 100644 --- a/docs/regressions/regressions-dl19-passage-openai-ada2-int8.md +++ b/docs/regressions/regressions-dl19-passage.openai-ada2.hnsw-int8.md @@ -1,7 +1,5 @@ # Anserini Regressions: TREC 2019 Deep Learning Track (Passage) -**NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. - **Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: @@ -13,13 +11,13 @@ In these experiments, we are using pre-encoded queries (i.e., cached results of Note that the NIST relevance judgments provide far more relevant passages per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast). For additional instructions on working with MS MARCO passage collection, refer to [this page](experiments-msmarco-passage.md). -The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl19-passage-openai-ada2-int8.yaml). -Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl19-passage-openai-ada2-int8.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. +The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl19-passage.openai-ada2.hnsw-int8.yaml). +Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl19-passage.openai-ada2.hnsw-int8.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-openai-ada2-int8 +python src/main/python/run_regression.py --index --verify --search --regression dl19-passage.openai-ada2.hnsw-int8 ``` We make available a version of the MS MARCO Passage Corpus that has already been encoded with the OpenAI-ada2 embedding model. @@ -27,7 +25,7 @@ We make available a version of the MS MARCO Passage Corpus that has already been From any machine, the following command will download the corpus and perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --download --index --verify --search --regression dl19-passage-openai-ada2-int8 +python src/main/python/run_regression.py --download --index --verify --search --regression dl19-passage.openai-ada2.hnsw-int8 ``` The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results. @@ -45,7 +43,7 @@ To confirm, `msmarco-passage-openai-ada2.tar` is 109 GB and has MD5 checksum `a4 With the corpus downloaded, the following command will perform the remaining steps below: ```bash -python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-openai-ada2-int8 \ +python src/main/python/run_regression.py --index --verify --search --regression dl19-passage.openai-ada2.hnsw-int8 \ --corpus-path collections/msmarco-passage-openai-ada2 ``` @@ -58,7 +56,7 @@ bin/run.sh io.anserini.index.IndexHnswDenseVectors \ -collection JsonDenseVectorCollection \ -input /path/to/msmarco-passage-openai-ada2 \ -generator HnswDenseVectorDocumentGenerator \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2-int8/ \ + -index indexes/lucene-hnsw-int8.msmarco-v1-passage.openai-ada2/ \ -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge -quantize.int8 \ >& logs/log.msmarco-passage-openai-ada2 & ``` @@ -81,7 +79,7 @@ After indexing has completed, you should be able to perform retrieval as follows ```bash bin/run.sh io.anserini.search.SearchHnswDenseVectors \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2-int8/ \ + -index indexes/lucene-hnsw-int8.msmarco-v1-passage.openai-ada2/ \ -topics tools/topics-and-qrels/topics.dl19-passage.openai-ada2.jsonl.gz \ -topicReader JsonIntVector \ -output runs/run.msmarco-passage-openai-ada2.openai-ada2-cached_q.topics.dl19-passage.openai-ada2.jsonl.txt \ @@ -103,16 +101,16 @@ With the above commands, you should be able to reproduce the following results: | **AP@1000** | **OpenAI-ada2**| |:-------------------------------------------------------------------------------------------------------------|-----------| -| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.479 | +| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.478 | | **nDCG@10** | **OpenAI-ada2**| -| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.704 | +| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.707 | | **R@100** | **OpenAI-ada2**| -| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.624 | +| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.617 | | **R@1000** | **OpenAI-ada2**| -| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.857 | +| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.853 | Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. -Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/dl19-passage-openai-ada2-int8.yaml). +Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/dl19-passage.openai-ada2.hnsw-int8.yaml). Also note that retrieval metrics are computed to depth 1000 hits per query (as opposed to 100 hits per query for document ranking). Also, for computing nDCG, remember that we keep qrels of _all_ relevance grades, whereas for other metrics (e.g., AP), relevance grade 1 is considered not relevant (i.e., use the `-l 2` option in `trec_eval`). @@ -120,4 +118,4 @@ The experimental results reported here are directly comparable to the results re ## Reproduction Log[*](reproducibility.md) -To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl19-passage-openai-ada2-int8.template) and run `bin/build.sh` to rebuild the documentation. +To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl19-passage.openai-ada2.hnsw-int8.template) and run `bin/build.sh` to rebuild the documentation. diff --git a/docs/regressions/regressions-dl19-passage-openai-ada2.md b/docs/regressions/regressions-dl19-passage.openai-ada2.hnsw.md similarity index 89% rename from docs/regressions/regressions-dl19-passage-openai-ada2.md rename to docs/regressions/regressions-dl19-passage.openai-ada2.hnsw.md index b7a5d46215..4a2fb713fa 100644 --- a/docs/regressions/regressions-dl19-passage-openai-ada2.md +++ b/docs/regressions/regressions-dl19-passage.openai-ada2.hnsw.md @@ -11,13 +11,13 @@ In these experiments, we are using pre-encoded queries (i.e., cached results of Note that the NIST relevance judgments provide far more relevant passages per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast). For additional instructions on working with MS MARCO passage collection, refer to [this page](experiments-msmarco-passage.md). -The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl19-passage-openai-ada2.yaml). -Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl19-passage-openai-ada2.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. +The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl19-passage.openai-ada2.hnsw.yaml). +Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl19-passage.openai-ada2.hnsw.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-openai-ada2 +python src/main/python/run_regression.py --index --verify --search --regression dl19-passage.openai-ada2.hnsw ``` We make available a version of the MS MARCO Passage Corpus that has already been encoded with the OpenAI-ada2 embedding model. @@ -25,7 +25,7 @@ We make available a version of the MS MARCO Passage Corpus that has already been From any machine, the following command will download the corpus and perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --download --index --verify --search --regression dl19-passage-openai-ada2 +python src/main/python/run_regression.py --download --index --verify --search --regression dl19-passage.openai-ada2.hnsw ``` The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results. @@ -43,7 +43,7 @@ To confirm, `msmarco-passage-openai-ada2.tar` is 109 GB and has MD5 checksum `a4 With the corpus downloaded, the following command will perform the remaining steps below: ```bash -python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-openai-ada2 \ +python src/main/python/run_regression.py --index --verify --search --regression dl19-passage.openai-ada2.hnsw \ --corpus-path collections/msmarco-passage-openai-ada2 ``` @@ -56,7 +56,7 @@ bin/run.sh io.anserini.index.IndexHnswDenseVectors \ -collection JsonDenseVectorCollection \ -input /path/to/msmarco-passage-openai-ada2 \ -generator HnswDenseVectorDocumentGenerator \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2/ \ + -index indexes/lucene-hnsw.msmarco-v1-passage.openai-ada2/ \ -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge -maxThreadMemoryBeforeFlush 1945 \ >& logs/log.msmarco-passage-openai-ada2 & ``` @@ -77,7 +77,7 @@ After indexing has completed, you should be able to perform retrieval as follows ```bash bin/run.sh io.anserini.search.SearchHnswDenseVectors \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2/ \ + -index indexes/lucene-hnsw.msmarco-v1-passage.openai-ada2/ \ -topics tools/topics-and-qrels/topics.dl19-passage.openai-ada2.jsonl.gz \ -topicReader JsonIntVector \ -output runs/run.msmarco-passage-openai-ada2.openai-ada2-cached_q.topics.dl19-passage.openai-ada2.jsonl.txt \ @@ -108,7 +108,7 @@ With the above commands, you should be able to reproduce the following results: | [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.857 | Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. -Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/dl19-passage-openai-ada2.yaml). +Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/dl19-passage.openai-ada2.hnsw.yaml). Also note that retrieval metrics are computed to depth 1000 hits per query (as opposed to 100 hits per query for document ranking). Also, for computing nDCG, remember that we keep qrels of _all_ relevance grades, whereas for other metrics (e.g., AP), relevance grade 1 is considered not relevant (i.e., use the `-l 2` option in `trec_eval`). @@ -116,4 +116,4 @@ The experimental results reported here are directly comparable to the results re ## Reproduction Log[*](reproducibility.md) -To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl19-passage-openai-ada2.template) and run `bin/build.sh` to rebuild the documentation. +To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl19-passage.openai-ada2.hnsw.template) and run `bin/build.sh` to rebuild the documentation. diff --git a/docs/regressions/regressions-dl20-passage-openai-ada2-int8.md b/docs/regressions/regressions-dl20-passage.openai-ada2.hnsw-int8.md similarity index 86% rename from docs/regressions/regressions-dl20-passage-openai-ada2-int8.md rename to docs/regressions/regressions-dl20-passage.openai-ada2.hnsw-int8.md index 5b493a6162..16a344f9fd 100644 --- a/docs/regressions/regressions-dl20-passage-openai-ada2-int8.md +++ b/docs/regressions/regressions-dl20-passage.openai-ada2.hnsw-int8.md @@ -1,7 +1,5 @@ # Anserini Regressions: TREC 2020 Deep Learning Track (Passage) -**NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. - **Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: @@ -13,13 +11,13 @@ In these experiments, we are using pre-encoded queries (i.e., cached results of Note that the NIST relevance judgments provide far more relevant passages per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast). For additional instructions on working with MS MARCO passage collection, refer to [this page](experiments-msmarco-passage.md). -The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl20-passage-openai-ada2-int8.yaml). -Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl20-passage-openai-ada2-int8.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. +The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl20-passage.openai-ada2.hnsw-int8.yaml). +Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl20-passage.openai-ada2.hnsw-int8.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-openai-ada2-int8 +python src/main/python/run_regression.py --index --verify --search --regression dl20-passage.openai-ada2.hnsw-int8 ``` We make available a version of the MS MARCO Passage Corpus that has already been encoded with the OpenAI-ada2 embedding model. @@ -27,7 +25,7 @@ We make available a version of the MS MARCO Passage Corpus that has already been From any machine, the following command will download the corpus and perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --download --index --verify --search --regression dl20-passage-openai-ada2-int8 +python src/main/python/run_regression.py --download --index --verify --search --regression dl20-passage.openai-ada2.hnsw-int8 ``` The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results. @@ -45,7 +43,7 @@ To confirm, `msmarco-passage-openai-ada2.tar` is 109 GB and has MD5 checksum `a4 With the corpus downloaded, the following command will perform the remaining steps below: ```bash -python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-openai-ada2-int8 \ +python src/main/python/run_regression.py --index --verify --search --regression dl20-passage.openai-ada2.hnsw-int8 \ --corpus-path collections/msmarco-passage-openai-ada2 ``` @@ -58,7 +56,7 @@ bin/run.sh io.anserini.index.IndexHnswDenseVectors \ -collection JsonDenseVectorCollection \ -input /path/to/msmarco-passage-openai-ada2 \ -generator HnswDenseVectorDocumentGenerator \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2-int8/ \ + -index indexes/lucene-hnsw-int8.msmarco-v1-passage.openai-ada2/ \ -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge -quantize.int8 \ >& logs/log.msmarco-passage-openai-ada2 & ``` @@ -81,7 +79,7 @@ After indexing has completed, you should be able to perform retrieval as follows ```bash bin/run.sh io.anserini.search.SearchHnswDenseVectors \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2-int8/ \ + -index indexes/lucene-hnsw-int8.msmarco-v1-passage.openai-ada2/ \ -topics tools/topics-and-qrels/topics.dl20.openai-ada2.jsonl.gz \ -topicReader JsonIntVector \ -output runs/run.msmarco-passage-openai-ada2.openai-ada2-cached_q.topics.dl20.openai-ada2.jsonl.txt \ @@ -105,14 +103,14 @@ With the above commands, you should be able to reproduce the following results: |:-------------------------------------------------------------------------------------------------------------|-----------| | [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.477 | | **nDCG@10** | **OpenAI-ada2**| -| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.676 | +| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.675 | | **R@100** | **OpenAI-ada2**| -| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.723 | +| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.727 | | **R@1000** | **OpenAI-ada2**| -| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.867 | +| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.866 | Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. -Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/dl20-passage-openai-ada2-int8.yaml). +Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/dl20-passage.openai-ada2.hnsw-int8.yaml). Also note that retrieval metrics are computed to depth 1000 hits per query (as opposed to 100 hits per query for document ranking). Also, for computing nDCG, remember that we keep qrels of _all_ relevance grades, whereas for other metrics (e.g., AP), relevance grade 1 is considered not relevant (i.e., use the `-l 2` option in `trec_eval`). @@ -120,4 +118,4 @@ The experimental results reported here are directly comparable to the results re ## Reproduction Log[*](reproducibility.md) -To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl20-passage-openai-ada2-int8.template) and run `bin/build.sh` to rebuild the documentation. +To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl20-passage.openai-ada2.hnsw-int8.template) and run `bin/build.sh` to rebuild the documentation. diff --git a/docs/regressions/regressions-dl20-passage-openai-ada2.md b/docs/regressions/regressions-dl20-passage.openai-ada2.hnsw.md similarity index 89% rename from docs/regressions/regressions-dl20-passage-openai-ada2.md rename to docs/regressions/regressions-dl20-passage.openai-ada2.hnsw.md index ad15fca4e1..98120c892e 100644 --- a/docs/regressions/regressions-dl20-passage-openai-ada2.md +++ b/docs/regressions/regressions-dl20-passage.openai-ada2.hnsw.md @@ -11,13 +11,13 @@ In these experiments, we are using pre-encoded queries (i.e., cached results of Note that the NIST relevance judgments provide far more relevant passages per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast). For additional instructions on working with MS MARCO passage collection, refer to [this page](experiments-msmarco-passage.md). -The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl20-passage-openai-ada2.yaml). -Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl20-passage-openai-ada2.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. +The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl20-passage.openai-ada2.hnsw.yaml). +Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl20-passage.openai-ada2.hnsw.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-openai-ada2 +python src/main/python/run_regression.py --index --verify --search --regression dl20-passage.openai-ada2.hnsw ``` We make available a version of the MS MARCO Passage Corpus that has already been encoded with the OpenAI-ada2 embedding model. @@ -25,7 +25,7 @@ We make available a version of the MS MARCO Passage Corpus that has already been From any machine, the following command will download the corpus and perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --download --index --verify --search --regression dl20-passage-openai-ada2 +python src/main/python/run_regression.py --download --index --verify --search --regression dl20-passage.openai-ada2.hnsw ``` The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results. @@ -43,7 +43,7 @@ To confirm, `msmarco-passage-openai-ada2.tar` is 109 GB and has MD5 checksum `a4 With the corpus downloaded, the following command will perform the remaining steps below: ```bash -python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-openai-ada2 \ +python src/main/python/run_regression.py --index --verify --search --regression dl20-passage.openai-ada2.hnsw \ --corpus-path collections/msmarco-passage-openai-ada2 ``` @@ -56,7 +56,7 @@ bin/run.sh io.anserini.index.IndexHnswDenseVectors \ -collection JsonDenseVectorCollection \ -input /path/to/msmarco-passage-openai-ada2 \ -generator HnswDenseVectorDocumentGenerator \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2/ \ + -index indexes/lucene-hnsw.msmarco-v1-passage.openai-ada2/ \ -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge -maxThreadMemoryBeforeFlush 1945 \ >& logs/log.msmarco-passage-openai-ada2 & ``` @@ -77,7 +77,7 @@ After indexing has completed, you should be able to perform retrieval as follows ```bash bin/run.sh io.anserini.search.SearchHnswDenseVectors \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2/ \ + -index indexes/lucene-hnsw.msmarco-v1-passage.openai-ada2/ \ -topics tools/topics-and-qrels/topics.dl20.openai-ada2.jsonl.gz \ -topicReader JsonIntVector \ -output runs/run.msmarco-passage-openai-ada2.openai-ada2-cached_q.topics.dl20.openai-ada2.jsonl.txt \ @@ -108,7 +108,7 @@ With the above commands, you should be able to reproduce the following results: | [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.867 | Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. -Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/dl20-passage-openai-ada2.yaml). +Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/dl20-passage.openai-ada2.hnsw.yaml). Also note that retrieval metrics are computed to depth 1000 hits per query (as opposed to 100 hits per query for document ranking). Also, for computing nDCG, remember that we keep qrels of _all_ relevance grades, whereas for other metrics (e.g., AP), relevance grade 1 is considered not relevant (i.e., use the `-l 2` option in `trec_eval`). @@ -116,4 +116,4 @@ The experimental results reported here are directly comparable to the results re ## Reproduction Log[*](reproducibility.md) -To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl20-passage-openai-ada2.template) and run `bin/build.sh` to rebuild the documentation. +To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl20-passage.openai-ada2.hnsw.template) and run `bin/build.sh` to rebuild the documentation. diff --git a/docs/regressions/regressions-msmarco-passage-openai-ada2-int8.md b/docs/regressions/regressions-msmarco-v1-passage.openai-ada2.hnsw-int8.md similarity index 85% rename from docs/regressions/regressions-msmarco-passage-openai-ada2-int8.md rename to docs/regressions/regressions-msmarco-v1-passage.openai-ada2.hnsw-int8.md index 33e56cc43c..6fea8aa781 100644 --- a/docs/regressions/regressions-msmarco-passage-openai-ada2-int8.md +++ b/docs/regressions/regressions-msmarco-v1-passage.openai-ada2.hnsw-int8.md @@ -1,7 +1,5 @@ # Anserini Regressions: MS MARCO Passage Ranking -**NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. - **Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: @@ -10,13 +8,13 @@ This page describes regression experiments, integrated into Anserini's regressio In these experiments, we are using pre-encoded queries (i.e., cached results of query encoding). -The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/msmarco-passage-openai-ada2-int8.yaml). -Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/msmarco-passage-openai-ada2-int8.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. +The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/msmarco-v1-passage.openai-ada2.hnsw-int8.yaml). +Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/msmarco-v1-passage.openai-ada2.hnsw-int8.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2-int8 +python src/main/python/run_regression.py --index --verify --search --regression msmarco-v1-passage.openai-ada2.hnsw-int8 ``` We make available a version of the MS MARCO Passage Corpus that has already been encoded with the OpenAI-ada2 embedding model. @@ -24,7 +22,7 @@ We make available a version of the MS MARCO Passage Corpus that has already been From any machine, the following command will download the corpus and perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --download --index --verify --search --regression msmarco-passage-openai-ada2-int8 +python src/main/python/run_regression.py --download --index --verify --search --regression msmarco-v1-passage.openai-ada2.hnsw-int8 ``` The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results. @@ -42,7 +40,7 @@ To confirm, `msmarco-passage-openai-ada2.tar` is 109 GB and has MD5 checksum `a4 With the corpus downloaded, the following command will perform the remaining steps below: ```bash -python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2-int8 \ +python src/main/python/run_regression.py --index --verify --search --regression msmarco-v1-passage.openai-ada2.hnsw-int8 \ --corpus-path collections/msmarco-passage-openai-ada2 ``` @@ -55,7 +53,7 @@ bin/run.sh io.anserini.index.IndexHnswDenseVectors \ -collection JsonDenseVectorCollection \ -input /path/to/msmarco-passage-openai-ada2 \ -generator HnswDenseVectorDocumentGenerator \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2-int8/ \ + -index indexes/lucene-hnsw-int8.msmarco-v1-passage.openai-ada2/ \ -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge -quantize.int8 \ >& logs/log.msmarco-passage-openai-ada2 & ``` @@ -77,7 +75,7 @@ After indexing has completed, you should be able to perform retrieval as follows ```bash bin/run.sh io.anserini.search.SearchHnswDenseVectors \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2-int8/ \ + -index indexes/lucene-hnsw-int8.msmarco-v1-passage.openai-ada2/ \ -topics tools/topics-and-qrels/topics.msmarco-passage.dev-subset.openai-ada2.jsonl.gz \ -topicReader JsonIntVector \ -output runs/run.msmarco-passage-openai-ada2.openai-ada2-cached_q.topics.msmarco-passage.dev-subset.openai-ada2.jsonl.txt \ @@ -99,18 +97,18 @@ With the above commands, you should be able to reproduce the following results: | **AP@1000** | **OpenAI-ada2**| |:-------------------------------------------------------------------------------------------------------------|-----------| -| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.350 | -| **RR@10** | **OpenAI-ada2**| | [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.343 | +| **RR@10** | **OpenAI-ada2**| +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.336 | | **R@100** | **OpenAI-ada2**| -| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.898 | +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.894 | | **R@1000** | **OpenAI-ada2**| -| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.985 | +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.983 | Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. -Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/msmarco-passage-openai-ada2-int8.yaml). +Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/msmarco-v1-passage.openai-ada2.hnsw-int8.yaml). ## Reproduction Log[*](../../docs/reproducibility.md) -To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/msmarco-passage-openai-ada2-int8.template) and run `bin/build.sh` to rebuild the documentation. +To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/msmarco-v1-passage.openai-ada2.hnsw-int8.template) and run `bin/build.sh` to rebuild the documentation. diff --git a/docs/regressions/regressions-msmarco-passage-openai-ada2.md b/docs/regressions/regressions-msmarco-v1-passage.openai-ada2.hnsw.md similarity index 88% rename from docs/regressions/regressions-msmarco-passage-openai-ada2.md rename to docs/regressions/regressions-msmarco-v1-passage.openai-ada2.hnsw.md index 73a6b2e604..301cb0fd3d 100644 --- a/docs/regressions/regressions-msmarco-passage-openai-ada2.md +++ b/docs/regressions/regressions-msmarco-v1-passage.openai-ada2.hnsw.md @@ -8,13 +8,13 @@ This page describes regression experiments, integrated into Anserini's regressio In these experiments, we are using pre-encoded queries (i.e., cached results of query encoding). -The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/msmarco-passage-openai-ada2.yaml). -Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/msmarco-passage-openai-ada2.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. +The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/msmarco-v1-passage.openai-ada2.hnsw.yaml). +Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/msmarco-v1-passage.openai-ada2.hnsw.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2 +python src/main/python/run_regression.py --index --verify --search --regression msmarco-v1-passage.openai-ada2.hnsw ``` We make available a version of the MS MARCO Passage Corpus that has already been encoded with the OpenAI-ada2 embedding model. @@ -22,7 +22,7 @@ We make available a version of the MS MARCO Passage Corpus that has already been From any machine, the following command will download the corpus and perform the complete regression, end to end: ```bash -python src/main/python/run_regression.py --download --index --verify --search --regression msmarco-passage-openai-ada2 +python src/main/python/run_regression.py --download --index --verify --search --regression msmarco-v1-passage.openai-ada2.hnsw ``` The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results. @@ -40,7 +40,7 @@ To confirm, `msmarco-passage-openai-ada2.tar` is 109 GB and has MD5 checksum `a4 With the corpus downloaded, the following command will perform the remaining steps below: ```bash -python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2 \ +python src/main/python/run_regression.py --index --verify --search --regression msmarco-v1-passage.openai-ada2.hnsw \ --corpus-path collections/msmarco-passage-openai-ada2 ``` @@ -53,7 +53,7 @@ bin/run.sh io.anserini.index.IndexHnswDenseVectors \ -collection JsonDenseVectorCollection \ -input /path/to/msmarco-passage-openai-ada2 \ -generator HnswDenseVectorDocumentGenerator \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2/ \ + -index indexes/lucene-hnsw.msmarco-v1-passage.openai-ada2/ \ -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge -maxThreadMemoryBeforeFlush 1945 \ >& logs/log.msmarco-passage-openai-ada2 & ``` @@ -73,7 +73,7 @@ After indexing has completed, you should be able to perform retrieval as follows ```bash bin/run.sh io.anserini.search.SearchHnswDenseVectors \ - -index indexes/lucene-hnsw.msmarco-passage-openai-ada2/ \ + -index indexes/lucene-hnsw.msmarco-v1-passage.openai-ada2/ \ -topics tools/topics-and-qrels/topics.msmarco-passage.dev-subset.openai-ada2.jsonl.gz \ -topicReader JsonIntVector \ -output runs/run.msmarco-passage-openai-ada2.openai-ada2-cached_q.topics.msmarco-passage.dev-subset.openai-ada2.jsonl.txt \ @@ -104,9 +104,9 @@ With the above commands, you should be able to reproduce the following results: | [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.985 | Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. -Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/msmarco-passage-openai-ada2.yaml). +Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/msmarco-v1-passage.openai-ada2.hnsw.yaml). ## Reproduction Log[*](../../docs/reproducibility.md) -To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/msmarco-passage-openai-ada2.template) and run `bin/build.sh` to rebuild the documentation. +To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/msmarco-v1-passage.openai-ada2.hnsw.template) and run `bin/build.sh` to rebuild the documentation. diff --git a/src/main/python/regressions-batch03.txt b/src/main/python/regressions-batch03.txt index 4c282dfc9b..0fae7b8491 100644 --- a/src/main/python/regressions-batch03.txt +++ b/src/main/python/regressions-batch03.txt @@ -5,7 +5,8 @@ python src/main/python/run_regression.py --index --verify --search --regression python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-bge-base-en-v1.5-hnsw-int8 > logs/log.msmarco-passage-bge-base-en-v1.5-hnsw-int8 2>&1 python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-hnsw > logs/log.msmarco-passage-cos-dpr-distil-hnsw 2>&1 python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-hnsw-int8 > logs/log.msmarco-passage-cos-dpr-distil-hnsw-int8 2>&1 -python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2 > logs/log.msmarco-passage-openai-ada2 2>&1 +python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2-hnsw > logs/log.msmarco-passage-openai-ada2-hnsw 2>&1 +python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2-hnsw-int8 > logs/log.msmarco-passage-openai-ada2-hnsw-int8 2>&1 python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-splade-pp-ed > logs/log.msmarco-passage-splade-pp-ed 2>&1 python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-splade-pp-sd > logs/log.msmarco-passage-splade-pp-sd 2>&1 @@ -89,7 +90,8 @@ python src/main/python/run_regression.py --search --regression dl19-passage-cos- python src/main/python/run_regression.py --search --regression dl19-passage-cos-dpr-distil-hnsw-int8 > logs/log.dl19-passage-cos-dpr-distil-hnsw-int8 2>&1 python src/main/python/run_regression.py --search --regression dl19-passage-cos-dpr-distil-fw > logs/log.dl19-passage-cos-dpr-distil-fw 2>&1 python src/main/python/run_regression.py --search --regression dl19-passage-cos-dpr-distil-lexlsh > logs/log.dl19-passage-cos-dpr-distil-lexlsh 2>&1 -python src/main/python/run_regression.py --search --regression dl19-passage-openai-ada2 > logs/log.dl19-passage-openai-ada2 2>&1 +python src/main/python/run_regression.py --search --regression dl19-passage-openai-ada2-hnsw > logs/log.dl19-passage-openai-ada2-hnsw 2>&1 +python src/main/python/run_regression.py --search --regression dl19-passage-openai-ada2-hnsw-int8 > logs/log.dl19-passage-openai-ada2-hnsw-int8 2>&1 python src/main/python/run_regression.py --search --regression dl19-passage-cohere-embed-english-v3.0-hnsw > logs/log.dl19-passage-cohere-embed-english-v3.0-hnsw 2>&1 python src/main/python/run_regression.py --search --regression dl19-passage-cohere-embed-english-v3.0-hnsw-int8 > logs/log.dl19-passage-cohere-embed-english-v3.0-hnsw-int8 2>&1 @@ -133,7 +135,8 @@ python src/main/python/run_regression.py --search --regression dl20-passage-cos- python src/main/python/run_regression.py --search --regression dl20-passage-cos-dpr-distil-hnsw-int8 > logs/log.dl20-passage-cos-dpr-distil-hnsw-int8 2>&1 python src/main/python/run_regression.py --search --regression dl20-passage-cos-dpr-distil-fw > logs/log.dl20-passage-cos-dpr-distil-fw 2>&1 python src/main/python/run_regression.py --search --regression dl20-passage-cos-dpr-distil-lexlsh > logs/log.dl20-passage-cos-dpr-distil-lexlsh 2>&1 -python src/main/python/run_regression.py --search --regression dl20-passage-openai-ada2 > logs/log.dl20-passage-openai-ada2 2>&1 +python src/main/python/run_regression.py --search --regression dl20-passage-openai-ada2-hnsw > logs/log.dl20-passage-openai-ada2-hnsw 2>&1 +python src/main/python/run_regression.py --search --regression dl20-passage-openai-ada2-hnsw-int8 > logs/log.dl20-passage-openai-ada2-hnsw-int8 2>&1 python src/main/python/run_regression.py --search --regression dl20-passage-cohere-embed-english-v3.0-hnsw > logs/log.dl20-passage-cohere-embed-english-v3.0-hnsw 2>&1 python src/main/python/run_regression.py --search --regression dl20-passage-cohere-embed-english-v3.0-hnsw-int8 > logs/log.dl20-passage-cohere-embed-english-v3.0-hnsw-int8 2>&1 diff --git a/src/main/resources/docgen/templates/dl19-passage-openai-ada2-int8.template b/src/main/resources/docgen/templates/dl19-passage.openai-ada2.hnsw-int8.template similarity index 97% rename from src/main/resources/docgen/templates/dl19-passage-openai-ada2-int8.template rename to src/main/resources/docgen/templates/dl19-passage.openai-ada2.hnsw-int8.template index 274bfc48e5..a540b45cca 100644 --- a/src/main/resources/docgen/templates/dl19-passage-openai-ada2-int8.template +++ b/src/main/resources/docgen/templates/dl19-passage.openai-ada2.hnsw-int8.template @@ -1,7 +1,5 @@ # Anserini Regressions: TREC 2019 Deep Learning Track (Passage) -**NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. - **Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl19-passage-openai-ada2.template b/src/main/resources/docgen/templates/dl19-passage.openai-ada2.hnsw.template similarity index 100% rename from src/main/resources/docgen/templates/dl19-passage-openai-ada2.template rename to src/main/resources/docgen/templates/dl19-passage.openai-ada2.hnsw.template diff --git a/src/main/resources/docgen/templates/dl20-passage-openai-ada2-int8.template b/src/main/resources/docgen/templates/dl20-passage.openai-ada2.hnsw-int8.template similarity index 97% rename from src/main/resources/docgen/templates/dl20-passage-openai-ada2-int8.template rename to src/main/resources/docgen/templates/dl20-passage.openai-ada2.hnsw-int8.template index e205721637..ea6c92c852 100644 --- a/src/main/resources/docgen/templates/dl20-passage-openai-ada2-int8.template +++ b/src/main/resources/docgen/templates/dl20-passage.openai-ada2.hnsw-int8.template @@ -1,7 +1,5 @@ # Anserini Regressions: TREC 2020 Deep Learning Track (Passage) -**NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. - **Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl20-passage-openai-ada2.template b/src/main/resources/docgen/templates/dl20-passage.openai-ada2.hnsw.template similarity index 100% rename from src/main/resources/docgen/templates/dl20-passage-openai-ada2.template rename to src/main/resources/docgen/templates/dl20-passage.openai-ada2.hnsw.template diff --git a/src/main/resources/docgen/templates/msmarco-passage-openai-ada2-int8.template b/src/main/resources/docgen/templates/msmarco-v1-passage.openai-ada2.hnsw-int8.template similarity index 97% rename from src/main/resources/docgen/templates/msmarco-passage-openai-ada2-int8.template rename to src/main/resources/docgen/templates/msmarco-v1-passage.openai-ada2.hnsw-int8.template index fec329dff5..d1773246fa 100644 --- a/src/main/resources/docgen/templates/msmarco-passage-openai-ada2-int8.template +++ b/src/main/resources/docgen/templates/msmarco-v1-passage.openai-ada2.hnsw-int8.template @@ -1,7 +1,5 @@ # Anserini Regressions: MS MARCO Passage Ranking -**NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. - **Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/src/main/resources/docgen/templates/msmarco-passage-openai-ada2.template b/src/main/resources/docgen/templates/msmarco-v1-passage.openai-ada2.hnsw.template similarity index 100% rename from src/main/resources/docgen/templates/msmarco-passage-openai-ada2.template rename to src/main/resources/docgen/templates/msmarco-v1-passage.openai-ada2.hnsw.template diff --git a/src/main/resources/regression/dl19-passage-openai-ada2-int8.yaml b/src/main/resources/regression/dl19-passage.openai-ada2.hnsw-int8.yaml similarity index 92% rename from src/main/resources/regression/dl19-passage-openai-ada2-int8.yaml rename to src/main/resources/regression/dl19-passage.openai-ada2.hnsw-int8.yaml index e96365c34a..3c232f6645 100644 --- a/src/main/resources/regression/dl19-passage-openai-ada2-int8.yaml +++ b/src/main/resources/regression/dl19-passage.openai-ada2.hnsw-int8.yaml @@ -5,7 +5,7 @@ corpus_path: collections/msmarco/msmarco-passage-openai-ada2/ download_url: https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-openai-ada2.tar download_checksum: a4d843d522ff3a3af7edbee789a63402 -index_path: indexes/lucene-hnsw.msmarco-passage-openai-ada2-int8/ +index_path: indexes/lucene-hnsw-int8.msmarco-v1-passage.openai-ada2/ index_type: hnsw collection_class: JsonDenseVectorCollection generator_class: HnswDenseVectorDocumentGenerator @@ -56,10 +56,10 @@ models: params: -generator VectorQueryGenerator -topicField vector -threads 16 -hits 1000 -efSearch 1000 results: AP@1000: - - 0.479 + - 0.478 nDCG@10: - - 0.704 + - 0.707 R@100: - - 0.624 + - 0.617 R@1000: - - 0.857 + - 0.853 diff --git a/src/main/resources/regression/dl19-passage-openai-ada2.yaml b/src/main/resources/regression/dl19-passage.openai-ada2.hnsw.yaml similarity index 96% rename from src/main/resources/regression/dl19-passage-openai-ada2.yaml rename to src/main/resources/regression/dl19-passage.openai-ada2.hnsw.yaml index ebe909dbc4..b5c198106b 100644 --- a/src/main/resources/regression/dl19-passage-openai-ada2.yaml +++ b/src/main/resources/regression/dl19-passage.openai-ada2.hnsw.yaml @@ -5,7 +5,7 @@ corpus_path: collections/msmarco/msmarco-passage-openai-ada2/ download_url: https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-openai-ada2.tar download_checksum: a4d843d522ff3a3af7edbee789a63402 -index_path: indexes/lucene-hnsw.msmarco-passage-openai-ada2/ +index_path: indexes/lucene-hnsw.msmarco-v1-passage.openai-ada2/ index_type: hnsw collection_class: JsonDenseVectorCollection generator_class: HnswDenseVectorDocumentGenerator diff --git a/src/main/resources/regression/dl20-passage-openai-ada2-int8.yaml b/src/main/resources/regression/dl20-passage.openai-ada2.hnsw-int8.yaml similarity index 93% rename from src/main/resources/regression/dl20-passage-openai-ada2-int8.yaml rename to src/main/resources/regression/dl20-passage.openai-ada2.hnsw-int8.yaml index 66f2b19830..574c5fa04e 100644 --- a/src/main/resources/regression/dl20-passage-openai-ada2-int8.yaml +++ b/src/main/resources/regression/dl20-passage.openai-ada2.hnsw-int8.yaml @@ -5,7 +5,7 @@ corpus_path: collections/msmarco/msmarco-passage-openai-ada2/ download_url: https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-openai-ada2.tar download_checksum: a4d843d522ff3a3af7edbee789a63402 -index_path: indexes/lucene-hnsw.msmarco-passage-openai-ada2-int8/ +index_path: indexes/lucene-hnsw-int8.msmarco-v1-passage.openai-ada2/ index_type: hnsw collection_class: JsonDenseVectorCollection generator_class: HnswDenseVectorDocumentGenerator @@ -58,8 +58,8 @@ models: AP@1000: - 0.477 nDCG@10: - - 0.676 + - 0.675 R@100: - - 0.723 + - 0.727 R@1000: - - 0.867 + - 0.866 diff --git a/src/main/resources/regression/dl20-passage-openai-ada2.yaml b/src/main/resources/regression/dl20-passage.openai-ada2.hnsw.yaml similarity index 96% rename from src/main/resources/regression/dl20-passage-openai-ada2.yaml rename to src/main/resources/regression/dl20-passage.openai-ada2.hnsw.yaml index 49db70ac5d..652a951003 100644 --- a/src/main/resources/regression/dl20-passage-openai-ada2.yaml +++ b/src/main/resources/regression/dl20-passage.openai-ada2.hnsw.yaml @@ -5,7 +5,7 @@ corpus_path: collections/msmarco/msmarco-passage-openai-ada2/ download_url: https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-openai-ada2.tar download_checksum: a4d843d522ff3a3af7edbee789a63402 -index_path: indexes/lucene-hnsw.msmarco-passage-openai-ada2/ +index_path: indexes/lucene-hnsw.msmarco-v1-passage.openai-ada2/ index_type: hnsw collection_class: JsonDenseVectorCollection generator_class: HnswDenseVectorDocumentGenerator diff --git a/src/main/resources/regression/msmarco-passage-openai-ada2-int8.yaml b/src/main/resources/regression/msmarco-v1-passage.openai-ada2.hnsw-int8.yaml similarity index 93% rename from src/main/resources/regression/msmarco-passage-openai-ada2-int8.yaml rename to src/main/resources/regression/msmarco-v1-passage.openai-ada2.hnsw-int8.yaml index 935bd69ea3..356af81b0e 100644 --- a/src/main/resources/regression/msmarco-passage-openai-ada2-int8.yaml +++ b/src/main/resources/regression/msmarco-v1-passage.openai-ada2.hnsw-int8.yaml @@ -5,7 +5,7 @@ corpus_path: collections/msmarco/msmarco-passage-openai-ada2/ download_url: https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-openai-ada2.tar download_checksum: a4d843d522ff3a3af7edbee789a63402 -index_path: indexes/lucene-hnsw.msmarco-passage-openai-ada2-int8/ +index_path: indexes/lucene-hnsw-int8.msmarco-v1-passage.openai-ada2/ index_type: hnsw collection_class: JsonDenseVectorCollection generator_class: HnswDenseVectorDocumentGenerator @@ -56,10 +56,10 @@ models: params: -generator VectorQueryGenerator -topicField vector -threads 16 -hits 1000 -efSearch 1000 results: AP@1000: - - 0.350 - RR@10: - 0.343 + RR@10: + - 0.336 R@100: - - 0.898 + - 0.894 R@1000: - - 0.985 + - 0.983 diff --git a/src/main/resources/regression/msmarco-passage-openai-ada2.yaml b/src/main/resources/regression/msmarco-v1-passage.openai-ada2.hnsw.yaml similarity index 96% rename from src/main/resources/regression/msmarco-passage-openai-ada2.yaml rename to src/main/resources/regression/msmarco-v1-passage.openai-ada2.hnsw.yaml index 1069abf82f..17ff730755 100644 --- a/src/main/resources/regression/msmarco-passage-openai-ada2.yaml +++ b/src/main/resources/regression/msmarco-v1-passage.openai-ada2.hnsw.yaml @@ -5,7 +5,7 @@ corpus_path: collections/msmarco/msmarco-passage-openai-ada2/ download_url: https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-openai-ada2.tar download_checksum: a4d843d522ff3a3af7edbee789a63402 -index_path: indexes/lucene-hnsw.msmarco-passage-openai-ada2/ +index_path: indexes/lucene-hnsw.msmarco-v1-passage.openai-ada2/ index_type: hnsw collection_class: JsonDenseVectorCollection generator_class: HnswDenseVectorDocumentGenerator