Error with value of `n`

[alielfilali01]

Hey there,
while evaluating Qwen2.5-1.5B on these tasks TASKS="lighteval|aime25|0|0,lighteval|gpqa:diamond|0|0,harness|bbh|0|0,lighteval|simpleqa|0|0,extended|lcb|0|0" using lighteval vllm, i get the following error:

ValueError: n must be 1 when using greedy sampling, got 64

I'm using :

(lighteval) :~$ pip show lighteval
Name: lighteval
Version: 0.9.2

I don't know how or where to pass the value of n !
Should it be in the command itself or the yaml config ? and what should be the name ?
Removing the aime25 task, we get ValueError: n must be 1 when using greedy sampling, got 16 instead of 64 !

[alielfilali01]

A quick fix was to edit directly in the /path_to/lighteval/models/vllm/vllm_model.py file, line 283

            vllm_outputs = self._generate(
                inputs=inputs,
                max_new_tokens=max_new_tokens,
                stop_tokens=stop_tokens,
                returns_logits=returns_logits,
                # num_samples=num_samples,
            )

Comment out num_samples=num_samples, so we can always use the default value num_samples: int = 1, in the def _generate() function

I'm not sure this is the best way to fix this. So still waiting for @NathanHB input on this one before marking this as CLOSED.