[FT] Controlling the number of experiments/trials to run

[ljvmiranda921]

Issue encountered

Hi, I noticed that when running something like lighteval vllm for an MCF task, I also get an stderr value outside of the acc_ metric. Does that mean that multiple experiments were run? I can't seem to find in the docs how many trials were run for a given experiment.

Is there also a way to control this?

[NathanHB]

Hi ! stderr does not run multiple trials, it just uses a bootstrapping function on the metrics.

lighteval/src/lighteval/metrics/stderr.py

Line 52 in d3da6b9

class _bootstrap_internal:

[ljvmiranda921]

Got it thanks!