OpenR1-Qwen-7B achieves 47.40 on AIME24, better than reported!

[Hasuer]

The reported OpenR1-Qwen-7B results on AIME24 is 36.7.

While I download the model from huggingface, and use lighteval to evaluate it, I get the results below:

Task	Version	Metric	Value		Stderr
all		math_pass@1:32_samples	0.4740	±	0.0651
		extractive_match	0.4667	±	0.0926
lighteval:aime24:0	1	math_pass@1:32_samples	0.4740	±	0.0651
		extractive_match	0.4667	±	0.0926

Which is much higher than reported!

The evaluation code:

MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,tensor_parallel_size=4,max_model_length=32768,max_num_batched_tokens=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"

lighteval vllm $MODEL_ARGS "lighteval|aime24|0|0" \
      --use-chat-template \
      --output-dir "$OUTPUT_DIR"

I tried to use data_parallel_size, but encounter with this issue.

Besides, the vllm version I use is 0.8.3, ray 2.43.0, lighteval 0.8.1.dev0.

Has anyone ever faced this situation? Thanks in advance.

@lewtun Do you have any idea? Any comment can be helpful.

[Hasuer]

1. I don't add export VLLM_WORKER_MULTIPROC_METHOD=spawn in my evaluation code, which is added in tensor parallel version of evaluated code. I am not sure whether this would has an effect on the evaluation result?
2. I notice that lighteval apply different prompt for different tasks. For AIME24, the prompt can be found here. I wonder the reported result using the same prompt or other settings?

@NathanHB Do you have any idea? Any comment can be helpful.

[StarLooo]

I get a similar result (math_pass@1:32_samples 0.482 on AIME24 using the downloaded OpenR1-Qwen-7B weights).
But I cannot run the original evaluation code directly, so I try to make some modifications (See: #602) on it and then successfully run the lighteval evaluation.

[ahatamiz]

@Hasuer how did you compute math_pass@1:32_samples ? lighteval|aime24|0|0 does not seem to be giving you this.

[StarLooo]

@Hasuer how did you compute math_pass@1:32_samples ? lighteval|aime24|0|0 does not seem to be giving you this.

I think the latest version of lighteval has already integrated the aime24 task into its official supported task (see: https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/default_tasks.py#L315), which support the computation of pass@1 metric.
However, the old version of this open-r1 repository implement the evaluation of aime24 by adding implementions from itself, since the old version of lighteval didn't support aime24 directly.

[Hasuer]

I get a similar result (math_pass@1:32_samples 0.482 on AIME24 using the downloaded OpenR1-Qwen-7B weights). But I cannot run the original evaluation code directly, so I try to make some modifications (See: #602) on it and then successfully run the lighteval evaluation.

I can not run original evaluation code with data parallel either, use the tensor_parallel_size is ok. And I do not make other modifications.

[StarLooo]

I get a similar result (math_pass@1:32_samples 0.482 on AIME24 using the downloaded OpenR1-Qwen-7B weights). But I cannot run the original evaluation code directly, so I try to make some modifications (See: #602) on it and then successfully run the lighteval evaluation.

I can not run original evaluation code with data parallel either, use the tensor_parallel_size is ok. And I do not make other modifications.

Maybe the old version of open-r1 with the old version of lighteval can run the evaluation code without modification.
The data parallel problem is also reported here: huggingface/lighteval#670

[Hasuer]

I get a similar result (math_pass@1:32_samples 0.482 on AIME24 using the downloaded OpenR1-Qwen-7B weights). But I cannot run the original evaluation code directly, so I try to make some modifications (See: #602) on it and then successfully run the lighteval evaluation.

I can not run original evaluation code with data parallel either, use the tensor_parallel_size is ok. And I do not make other modifications.

Maybe the old version of open-r1 with the old version of lighteval can run the evaluation code without modification. The data parallel problem is also reported here: huggingface/lighteval#670

But I just clone the repo two days ago, and use make install to create the uv environment. The vllm version I use is 0.8.3, ray 2.43.0, lighteval 0.8.1.dev0 .

What modifications do you make to run the evaluation code? Can you run the evalaution code with data_parallel_size param in MODEL_ARGS?

[StarLooo]

But I just clone the repo two days ago, and use make install to create the uv environment. The vllm version I use is 0.8.3, ray 2.43.0, lighteval 0.8.1.dev0 .

What modifications do you make to run the evaluation code? Can you run the evalaution code with data_parallel_size param in MODEL_ARGS?

1. You can refer to the modifications that I made to run the evaluation code from here: Is vllm==0.8.3 causing some incompatible problems #602 (comment).
2. I installed the lighteval from source code. When I use "pip show lighteval" to check its version, it shows: 0.8.1.dev0.
3. I met the similar problem when using lighteval vllm with data parallel similar to this issue: [BUG] vLLM backend hangs with DDP lighteval#670
4. I'm not very sure about the detailed influence of different versions of open-r1, lighteval and vllm. Especially the open-r1's repository updates very frequently.
   And all the reference issues I mentioned above may contain useful information about how to run the evaluation.

[Hasuer]

Thanks for your instructions. And I’m really wondering how the reported score of OpenR1-Qwen-7B on AIME24 could be 36.7. Even I calculate the math_pass@1:1_samples, the result also acheives 40+.

[StarLooo]

Thanks for your instructions. And I’m really wondering how the reported score of OpenR1-Qwen-7B on AIME24 could be 36.7. Even I calculate the math_pass@1:1_samples, the result also acheives 40+.

As far as I know, the exact_match metric on AIME24 has a large variance especially for small models, different run may casue very different performance. Since the AIME24 only contains 30 questions, the math_pass@1:32_samples which sample 32 times on this dataset could be a better metric to monitor the model performance.
The recent reproduced evaluation also shows a better performance than the open-r1 reported values:
#545

[Hasuer]

Exactly, so it seems that huggingface underestimated the performance of their model (at least in AIME24)

[StarLooo]

Exactly, so it seems that huggingface underestimated the performance of their model (at least in AIME24)

I guess with the updation of lighteval, the metric computation (extract answer in the box and compare with ground truth) improves.
And the reported performance is computed using old version of lighteval a few months ago.

[StarLooo]

Also, according to the comment on lighteval's related codes:
https://github.com/huggingface/lighteval/blob/main/src/lighteval/metrics/dynamic_metrics.py#L200
There are known issues that should concern but difficult to adress.
Since the method to extract answer of math question and compare it with the ground-truth is complicated, as well as the influence of different prompts, the fluctuation and difference of different version of one evaluation framework / different evaluation frameworks are understandable.

[ahatamiz]

@Hasuer @StarLooo Thanks for the interesting discussions. Just to recap, the following computes math_pass@1:1_samples which is basically pass@1.

MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,tensor_parallel_size=4,max_model_length=32768,max_num_batched_tokens=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"

lighteval vllm $MODEL_ARGS "lighteval|aime24|0|0" \
      --use-chat-template \
      --output-dir "$OUTPUT_DIR"

How can one compute pass @K which samples K times, with the newest version of lighteval ?

Would it simply be

Also to note that in the new version (or probably the old one as well), something like lighteval|aime24|0|0 is basically following the format of suite|task|few_shot|truncate_few_shots. So you can only control the num few_shots but not the generations.

I rely on this package for comprehensive evaluations but it is really slow.

[Hasuer]

| How can one compute pass @k which samples K times, with the newest version of lighteval ?

You can make the following modifications:

1. This line define a math_pass_at_1_4n, which means k is 1 and generate 4 samples per question. To calculate pass@2, you can deifne math_pass_at_2_4n by passing k = 2 when initializing PassAtK clas, which means k is 2 and generate 4 samples per question.
2. Add the metirc you just define here if you want to evaluate AIME24 by pass@2.

For other benchmarks, you can use the same steps.