[FT] Support computing/displaying stddev in the final table from the variation in sample scores (when using sampling metrics) instead of a bootstrap

[jamesbraza]

Issue encountered

I was hoping to run an evaluation with N=5 replicates, as well as an get easy way to get error bars.

Solution/Feature

Some parameter in PipelineParameters.

Possible alternatives

It's possible that PipelineParameters.max_samples does this, but it's unclear to me.

[clefourrier]

When you say replicates, do you mean sampling different parameters?
If yes, you need to define your own custom task (and possibly metric if you don't want to use a pass at or maj at metric).

However, you can also get error bars using the bootstrapped stderr

[clefourrier]

max_samples allows you to run the evaluation on a limited amount of different samples (for example 5) to test that the pipeline is working well for you before starting a full run.

[jamesbraza]

When you say replicates, do you mean sampling different parameters?

I mean something like prompting a QA task to the language model N times, evaluating the responses for accuracy, and delivering mean/std dev on the responses.

max_samples allows you to run the evaluation on a limited amount of different samples (for example 5) to test that the pipeline is working well for you before starting a full run.

Ah then this is not a num replicates concept, thanks for clarifying.

---

To give a concrete example, perhaps something like prompting an LLM 10 times in parallel: "What is 2 + 2?"

And calculating mean and standard deviation on the answer containing "4".

[clefourrier]

Ok, so you indeed want a sampling metric (= prompting the model with a number of samples to generate, then computing a score on the different versions of each sample, then aggregating) - it would not exactly be trivial to add the stddev as such in the bootstrapping section (where we compute the value for the stderr) but you could probably extract the score from the details generated.