Which evaluation framework is used ?

[ChaofanTao]

Which evaluation framework is used to reproduce the results on LiveCodeBench, MBPP+, HumanEval+ in the ReadMe?

[ganler]

LiveCodeBench: https://github.com/LiveCodeBench/LiveCodeBench for both generation and eval
HumanEval+ / MBPP+: using an internal batched generation framework for generation and https://github.com/evalplus/evalplus for eval.

[ChaofanTao]

Thanks! IIn addition, have you tried to evaluate Code-R1 on SWE-Bench?

[ganler]

Code-R1 is not trained to act agentically, so it's not quite for SWE-Bench.