Fail to reproduce MATH-500 Score on DeepSeek-R1-Distill-Qwen-1.5B

[superdocker]

Here is my script and the corresponding results:

MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
TASK=math_500
OUTPUT_DIR=data/evals/$MODEL

NUM_GPUS=1
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilisation=0.8,tensor_parallel_size=$NUM_GPUS"
lighteval vllm $MODEL_ARGS "custom|$TASK|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --output-dir $OUTPUT_DIR

[2025-02-18 06:01:32,413] [    INFO]: --- COMPUTING METRICS --- (pipeline.py:299)
[2025-02-18 06:01:42,431] [    INFO]: --- DISPLAYING RESULTS --- (pipeline.py:342)
|      Task       |Version|     Metric     |Value|   |Stderr|
|-----------------|------:|----------------|----:|---|-----:|
|all              |       |extractive_match|0.766|_  | 0.019|
|custom:math_500:0|      1|extractive_match|0.766|_  | 0.019|

The score is 5 points lower than the one reported in the README.

I also referred to issue #194 but was unable to reproduce the reported score of 81.6.
My environment:

# CUDA 12.4
torch                             2.5.1
latex2sympy2_extended             1.0.6
vllm                              0.7.2 
math-verify                       0.5.2
lighteval                         0.6.0.dev0

Are there any additional steps I can try to debug or improve my results? Any suggestions would be greatly appreciated.

[gauss-clb]

I get the score of 0.836.

[superdocker]

@gauss-clb - Thanks for your response.

I’m not sure why, but when I ran it on a different A100 server, I got a result of 83.6.
I still don’t know the reason—despite using the same Docker image, the same requirements.txt, the same model files, and the same script, the results were normal on a different server.

Anyway, I appreciate your help.

[air-tea]

Here is my script and the corresponding results:这里是我的脚本及其对应的结果：

MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
TASK=math_500
OUTPUT_DIR=data/evals/$MODEL

NUM_GPUS=1
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilisation=0.8,tensor_parallel_size=$NUM_GPUS"
lighteval vllm $MODEL_ARGS "custom|$TASK|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --output-dir $OUTPUT_DIR

[2025-02-18 06:01:32,413] [    INFO]: --- COMPUTING METRICS --- (pipeline.py:299)
[2025-02-18 06:01:42,431] [    INFO]: --- DISPLAYING RESULTS --- (pipeline.py:342)
|      Task       |Version|     Metric     |Value|   |Stderr|
|-----------------|------:|----------------|----:|---|-----:|
|all              |       |extractive_match|0.766|_  | 0.019|
|custom:math_500:0|      1|extractive_match|0.766|_  | 0.019|

The score is 5 points lower than the one reported in the README.得分比 README 中报告的低 5 分。

I also referred to issue #194 but was unable to reproduce the reported score of 81.6.我也参考了问题 #194 ，但无法重现报告中的 81.6 分。 My environment:  我的环境：

# CUDA 12.4
torch                             2.5.1
latex2sympy2_extended             1.0.6
vllm                              0.7.2 
math-verify                       0.5.2
lighteval                         0.6.0.dev0

Are there any additional steps I can try to debug or improve my results? Any suggestions would be greatly appreciated.有哪些额外的步骤我可以尝试来调试或改进我的结果？任何建议都将非常感激。

I use the code in the repo and the default hyperparameters to finetune (use Lora) DeepSeek-R1-Distill-Qwen-1.5B on the Bespoke-Stratos-17k dataset, but I find that the accuracy decreases to around 20% on the MATH 500 dataset. Do you use the same setting? It seems that your result is very good, are there suggestions for me to improve my results? I really appreciate your help.

[superdocker]

@air-tea - I did not perform any additional fine-tuning; I directly evaluated the checkpoint of DeepSeek-R1-Distill-Qwen-1.5B. Therefore, the discrepancy between my lower and improved scores is likely due to environmental factors such as package versions rather than the effect of training. As a result, I may not be able to provide the answer you are looking for regarding your question about the lower score after LoRA fine-tuning.

[air-tea]

@air-tea - I did not perform any additional fine-tuning; I directly evaluated the checkpoint of DeepSeek-R1-Distill-Qwen-1.5B. Therefore, the discrepancy between my lower and improved scores is likely due to environmental factors such as package versions rather than the effect of training. As a result, I may not be able to provide the answer you are looking for regarding your question about the lower score after LoRA fine-tuning.我没有进行任何额外的微调；我直接评估了 DeepSeek-R1-Distill-Qwen-1.5B 的检查点。因此，我的低分和改进分之间的差异可能是由于环境因素，如包版本，而不是训练效果。因此，我可能无法提供您关于 LoRA 微调后分数较低的问题的答案。

Thanks for your kind reply~, I will keep trying to improve the model training.

[zwxandy]

Hi, I want to evaluate DeepSeek-R1-Distill-Qwen-7B on different tasks, so I use the provided script below:

NUM_GPUS=1 # Set to 8 for 32B and 70B models
MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,data_parallel_size=$NUM_GPUS,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

lighteval vllm $MODEL_ARGS "custom|aime24|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --output-dir $OUTPUT_DIR

However, I met the error is shown below:

It seems like an error when loading the dataset. How can I correctly download and load the dataset to fix the error?

Thanks for your help!

[RewindL]

maybe you should add eos_token_id=151645 in generation_config.json

[lewtun]

maybe you should add eos_token_id=151645 in generation_config.json

Good catch @RewindL ! This only affect generation in transformers with e.g. the pipeline() function since I believe vllm infers the EOS token from the tokenizer config.

In any case, this PR should fix the transformers side too: #663