Description
Describe the bug
When running lighteval with a custom evaluator, pydantic throws an error.
To Reproduce
Custom class (per https://huggingface.co/docs/lighteval/en/evaluating-a-custom-model):
from lighteval.models.abstract_model import LightevalModel
class DumpDataset(LightevalModel):
def __init__(self, config):
super().__init__(config)
pass
def greedy_until(self, requests, max_tokens=None, stop_sequences=None):
for request in requests:
print(request)
def loglikelihood(self, requests, log=True):
# Implement loglikelihood computation
pass
def loglikelihood_rolling(self, requests):
# Implement rolling loglikelihood computation
pass
def loglikelihood_single_token(self, requests):
# Implement single token loglikelihood computation
passCommand:
$ lighteval custom "dump" custom.py "lighteval|math_500"
[2025-05-20 20:59:18,048] [ INFO]: PyTorch version 2.6.0 available. (config.py:54)
INFO 05-20 20:59:21 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 05-20 20:59:21 [__init__.py:239] Automatically detected platform cuda.
╭─────────────────────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────────────────────╮
│ /home/aiscuser/.local/lib/python3.10/site-packages/lighteval/main_custom.py:112 in custom │
│ │
│ 109 │ ) │
│ 110 │ │
│ 111 │ parallelism_manager = ParallelismManager.CUSTOM │
│ ❱ 112 │ model_config = CustomModelConfig(model=model_name, model_definition_file_path=model_ │
│ 113 │ │
│ 114 │ pipeline_params = PipelineParameters( │
│ 115 │ │ launcher_type=parallelism_manager, │
│ │
│ /home/aiscuser/.local/lib/python3.10/site-packages/pydantic/main.py:253 in __init__ │
│ │
│ 250 │ │ """ │
│ 251 │ │ # `__tracebackhide__` tells pytest and some other tools to omit this function fr │
│ 252 │ │ __tracebackhide__ = True │
│ ❱ 253 │ │ validated_self = self.__pydantic_validator__.validate_python(data, self_instance │
│ 254 │ │ if self is not validated_self: │
│ 255 │ │ │ warnings.warn( │
│ 256 │ │ │ │ 'A custom validator is returning a value other than `self`.\n' │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ValidationError: 2 validation errors for CustomModelConfig
model_name
Field required [type=missing, input_value={'model': 'dump', 'model_...ile_path': 'custom2.py'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.11/v/missing
model
Extra inputs are not permitted [type=extra_forbidden, input_value='dump', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/extra_forbiddenExpected behavior
This should run my custom evaluator.
Version info
Ubuntu 22.04, Python 3.10.12
Versions:
$ pip list | egrep "pydantic|lighteval"
lighteval 0.9.2
pydantic 2.11.4
pydantic_core 2.33.2