Description
Describe the bug
In regards to below lighteval command using accelerate backend, the generation params for sampling and max new tokens are not being applied correctly.
Observing the following warning (moreover, setting do_sample:true throws a transformer config validation error)
2025-05-19 10:08:27,393] [ WARNING]: /usr/local/lib/python3.11/dist-packages/transformers/generation/configuration_utils.py:631: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.5` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. warnings.warn( (warnings.py:110)
To Reproduce
Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.
Command:
lighteval accelerate "model_name=Qwen/Qwen3-1.7B,generation_parameters={temperature:0.6, top_k:20, top_p:0.95, max_new_tokens:32768}" "lighteval|gsm8k|0|0" --system-prompt "You are an expert in math. Provide the final numeric answer in the following format: #### 123" --save-details --use-chat-template
Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.
OS - Ubuntu 22.04.5 LTS
Python - 3.11.10
pip - 24.2
lighteval - 0.9.2