RECITE: Recitation-Augmented Language Models

See "Recitation-Augmented Language Models" (ICLR 2023) for the paper associated with this codebase.

Setup

conda env create -f environment.yml
conda activate recite

Reproduction

Natural Questions

Direct:

export PYTHONPATH="$PWD:$PYTHONPATH"

python -u codex_experiments/run_hotpot_qa.py \
  --num_examples 1024 \
  --inference_scheme "direct" \
  --batch_size 8 \
  --verbose

RECITE:

export PYTHONPATH="$PWD:$PYTHONPATH"

python -u codex_experiments/run_hotpot_qa.py \
  --num_examples 1024 \
  --self_consistency_k 20 \
  --inference_scheme "recite" \
  --batch_size 8 \
  --verbose

Direct + BM25 (start the BM25 server from bm25_retrieval_server first):

export PYTHONPATH="$PWD:$PYTHONPATH"

python -u codex_experiments/run_natural_questions.py \
  --num_examples 1024 \
  --inference_scheme "with_bm25_context" \
  --batch_size 8 \
  --verbose

Direct + Ground-truth Context:

export PYTHONPATH="$PWD:$PYTHONPATH"

python -u codex_experiments/run_natural_questions.py \
  --num_examples 1024 \
  --inference_scheme "with_ground_truth_context" \
  --batch_size 8 \
  --verbose

TriviaQA

Direct:

export PYTHONPATH="$PWD:$PYTHONPATH"

python -u codex_experiments/run_trivia_qa.py \
  --num_examples 1024 \
  --inference_scheme "direct" \
  --batch_size 8 \
  --verbose

RECITE:

export PYTHONPATH="$PWD:$PYTHONPATH"

python -u codex_experiments/run_trivia_qa.py \
  --num_examples 1024 \
  --inference_scheme "recite" \
  --batch_size 8 \
  --verbose

HotpotQA

Direct:

export PYTHONPATH="$PWD:$PYTHONPATH"

python -u codex_experiments/run_hotpot_qa.py \
  --num_examples 1024 \
  --inference_scheme "direct" \
  --batch_size 8 \
  --verbose

RECITE:

export PYTHONPATH="$PWD:$PYTHONPATH"

python -u codex_experiments/run_hotpot_qa.py \
  --num_examples 1024 \
  --self_consistency_k 20 \
  --inference_scheme "recite" \
  --batch_size 8 \
  --verbose

Chain-of-thought:

export PYTHONPATH="$PWD:$PYTHONPATH"

python -u codex_experiments/run_hotpot_qa.py \
  --num_examples 1024 \
  --inference_scheme "cot" \
  --batch_size 1 \
  --verbose

Reference

If you found this codebase useful, please consider citing the paper:

@inproceedings{
    sun2023recitationaugmented,
    title={Recitation-Augmented Language Models},
    author={Zhiqing Sun and Xuezhi Wang and Yi Tay and Yiming Yang and Denny Zhou},
    booktitle={International Conference on Learning Representations},
    year={2023},
    url={https://openreview.net/forum?id=-cqvvvb-NkI}
}