Skip to content

add TREC RAG 2026 answer capability - #9

Open
nourj98 wants to merge 1 commit into
castorini:mainfrom
nourj98:codex/trec-rag-prompt
Open

add TREC RAG 2026 answer capability#9
nourj98 wants to merge 1 commit into
castorini:mainfrom
nourj98:codex/trec-rag-prompt

Conversation

@nourj98

@nourj98 nourj98 commented Jul 25, 2026

Copy link
Copy Markdown
Member

Summary

Adds end-to-end support for the official TREC RAG 2026 answer task over ClimbMix-400b.

Changes

  • Registers the official 119-topic trec-rag-2026 benchmark as an unjudged collection.
  • Makes qrels optional and skips retrieval evaluation when judgments are unavailable.
  • Adds the TREC RAG answer prompt and validation:
    • Sentence-level answers with ClimbMix document-ID citations
    • Maximum 1,024 words
    • Maximum three citations per sentence
    • Citation/reference consistency checks
    • Deterministically generated run metadata
  • Adds team_id, run_id, and run_desc configuration.
  • Persists validated structured output in per-query artifacts.
  • Generates the official rag_output_trec_rag_2026.jsonl submission.
  • Improves sharded merging, completion checks, resumability, and failed-topic retries.
  • Adds retry/backoff handling for transient Pyserini REST errors.
  • Updates documentation and focused tests.

Validation

  • Verified authenticated retrieval against climbmix-400b.
  • Completed an isolated rag2026-0 smoke test.
  • Validated metadata, narratives, word limits, citation structure, and referenced document IDs.
  • Ran focused tests, type checking, linting, and formatting checks.

@nourj98 nourj98 changed the title feat: add TREC RAG 2026 answer runs add TREC RAG 2026 answer capability Jul 25, 2026

@lilyjge lilyjge left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

roughly lgtm but I'm not the most familiar with typescript, @ricky42613 ptal?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants