Create a lightEval converter

[borgr]

Like inspect and eval harness etc. lightEval is another framework we should support
https://github.com/huggingface/lighteval

[gbemike]

i'll work on a plan for this

[gbemike]

can you assign this to me?

[Mandark-droid]

@gbemike are you still on this one? Don't want to duplicate your work. If you're tied up
I'm happy to take a crack at it, or put up a draft you can build on, whichever suits.

For context we ingest EEE downstream to feed a model recommender, so lightEval coverage
is directly useful to us and I don't mind doing the legwork.

I spent a bit of time reading the lighteval source since the README doesn't cover the
output format. Notes here in case they save someone the trip.

Results land in {output_dir}/results/{model_name}/results_{date_id}.json
(EvaluationTracker.save_results in logging/evaluation_tracker.py). Top level is:

config_general   asdict(GeneralConfigLogger)
results          metric_aggregated, i.e. dict[task][metric] = float
versions         task versions
config_tasks     task configs
summary_tasks    compiled per-task details

and config_general carries lighteval_sha, num_fewshot_seeds, max_samples,
job_id, start_time, end_time, total_evaluation_time_secondes, model_config,
model_name.

Two traps I ran into:

MetricsLogger writes {metric}_stderr into the same dict as the metric itself, so
iterating results[task] naively gets you acc_stderr as its own metric. It's set to
float("nan") when stderr can't be computed, which I'd read as "no uncertainty
reported" and omit rather than write as 0.

metric_aggregated holds both the per-subtask entries and an aggregate across them, so
emitting every key gives you the aggregate and its parts as siblings.

Also total_evaluation_time_secondes is genuinely spelled that way upstream, worth not
"fixing".

On lighteval's nested ModelConfig (its fields differ by backend: transformers / vllm /
endpoint) I think ModelInfo already settles it: name, id, developer,
inference_platform and inference_engine are normalised, and everything else goes to
additional_details, which is dict[str, str] so the leftovers need stringifying. Plus
the deployment_type / model_availability axes the validator defaults to unknown.
I'd follow that unless someone wants it done differently.

If it helps I can open a draft PR with the adapter plus an offline fixture test following
the lm_eval layout, easy to close or take over.

[gbemike]

@Mandark-droid Yeah, I'm still working on it, had an issue with my pc hence the delay. I'd be happy to build on your draft if that's fine and see if there's anything I'm currently missing from my own approach. Thanks

[Mandark-droid]

@gbemike no worries, and thanks for the heads up — hope the PC is sorted. This stays yours, take the time you need, I'm not going to touch the issue.

Draft is up as a PR so it's easier to read than a wall of code in a comment: #235. Take whatever's useful and throw away the rest, genuinely — if your approach is different I'd be curious which parts you'd have done another way.

The one thing I couldn't decide on my own is in the PR description: results mixes the tasks lighteval measured with the ones it averaged itself, and I've dropped the averaged rows. If you or a maintainer land somewhere else on that, it's a small change.

[gbemike]

@Mandark-droid yeah the pc issue is sorted, thanks. I'll check out the PR.

[gbemike]

@Mandark-droid just looked through your PR, I think you can go ahead and finish it up I wouldn't mind at all. Seems like you've gained some momentum and I can help review.

[borgr]

Wait I missed this, I think I am also working on lightEval, at some point Avijit asked for it. How far are you? Should I let an agent just finish off merging what I have and what you've started?

[Mandark-droid]

@borgr pretty far, but not finished — #235 is up as a draft. Adapter, CLI wiring, 31 tests,
ruff clean, and a convert → validate round-trip passing 3/3 locally with semantic checks
on.

Two gaps worth knowing before you decide anything. CI hasn't run at all — it's gated because
I'm a first-time contributor, so it needs a maintainer to approve the workflow. And the
fixture isn't captured output: lighteval pulls torch, so I built it from the v0.13.0 writers
(logging/evaluation_tracker.py, logging/info_loggers.py). Every field traces to a line
there, but it has never been checked against a real results file. If you've actually run
lighteval, that's the single thing that would improve this most.

Merging sounds good to me, and I'd rather fold into whatever you have than have two of these.
The one call you might disagree with is in the PR description: results mixes the tasks
lighteval measured with the ones it averaged itself, and I drop the averaged rows — aggregate()
builds them from keys that include the _stderr entries, so they carry a mean of standard
errors rather than the standard error of a mean. I record which keys were dropped instead of
discarding them silently. Easy to flip if you'd rather keep them.

[borgr]

Sounds good I'll take a look (I'm making a new CI thing anyway so don't worry about that(it was the reason I was working on it)

…

On Fri, Aug 7, 2026, 8:27 AM kshitij thakkar ***@***.***> wrote: *Mandark-droid* left a comment (evaleval/every_eval_ever#189) <#189 (comment)> @borgr <https://github.com/borgr> pretty far, but not finished — #235 <#235> is up as a draft. Adapter, CLI wiring, 31 tests, ruff clean, and a convert → validate round-trip passing 3/3 locally with semantic checks on. Two gaps worth knowing before you decide anything. CI hasn't run at all — it's gated because I'm a first-time contributor, so it needs a maintainer to approve the workflow. And the fixture isn't captured output: lighteval pulls torch, so I built it from the v0.13.0 writers (logging/evaluation_tracker.py, logging/info_loggers.py). Every field traces to a line there, but it has never been checked against a real results file. If you've actually run lighteval, that's the single thing that would improve this most. Merging sounds good to me, and I'd rather fold into whatever you have than have two of these. The one call you might disagree with is in the PR description: results mixes the tasks lighteval measured with the ones it averaged itself, and I drop the averaged rows — aggregate() builds them from keys that include the _stderr entries, so they carry a mean of standard errors rather than the standard error of a mean. I record which keys were dropped instead of discarding them silently. Easy to flip if you'd rather keep them. — Reply to this email directly, view it on GitHub <#189?email_source=notifications&email_token=ABQ6RWB3NDI3QQNZDPAKKMT5IXDLBA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRRG4YDIMZUGEZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5217043413>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABQ6RWB425N3NEHM5QOBCE35IXDLBAVCNFSNUABGKJSXA33TNF2G64TZHMYTANZSGQYTQNZZGY5US43TOVSTWNBWGY4TGNBUHEZTNILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/ABQ6RWCQXK4TJTIQUJK7VM35IXDLBA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRRG4YDIMZUGEZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/ABQ6RWEXBEEH5KRJUALXTHL5IXDLBA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRRG4YDIMZUGEZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>