[Roadmap] BigCodeBench Q3 2024 Roadmap

[terryyz]

This document includes the features of BigCodeBench Q3 2024. Please feel free to discuss and contribute, as this roadmap is shaped by the BigCodeBench community.

Help Wanted

• Lingering processes [bug] Lingering processes after eval run completion #42
• Better documentation [docs] Add working end-to-end example for a local model #40, [docs] Clarify risks of running evals locally. #41
• Dataset Repair, e.g., 🐛 [TestRemoval/TestRepair] - 211, 215- include status code in mock response #33
• Tests & CI/CD
• Flexible Pass@k Support Pass@100 not supported? #50

Feature

• Batch generation (to be released) #46, Batch size configuration when using greedy decoding #36
• Customized direct completion setup (to be released)
• Catch up on the progress of EvalPlus

Dataset

• More investigations on the BigCodeBench tasks

Ongoing Research

• Benchmarking more languages -- Verilog & R, cc @shailja-thakur @ThreeCirclesK @marianna13
• Agentic Evaluation (proof-of-concept infra to be scaled up), cc @JoshuaPurtell
• Grounded Zero-Shot Tool Use, cc @terryyz @SivilTaram