[BUG] custom model docs don't run: missing imports

[jamesbraza]

Describe the bug

https://github.com/huggingface/lighteval/blob/v0.9.2/docs/source/evaluating-a-custom-model.mdx#creating-a-custom-model did not import ParallelismManager.

https://github.com/huggingface/lighteval/blob/v0.9.2/docs/source/using-the-python-api.mdx uses a nonexistent EnvConfig

To Reproduce

N/a

Expected behavior

Docs to not have missing imports

Version info

lighteval==0.9.2

[clefourrier]

Hi! Good catch, feel free to open a PR for these! :)

[mjpost]

Hi @clefourrier—It's not just that the example doesn't run; the base class, LightevalModel, is now completely (edit: mostly) abstract, and in particular has no __init__(). This means, among other things, that every subclass has to manually implement the following code block:

    def __init__(self, config) -> None:
	self.model_name = config.model_name
        self.model_definition_file_path = config.model_definition_file_path

        self.model_info = ModelInfo(
            model_name=config.model_name,
            model_sha="",
            model_dtype=None,
            model_size="",
        )

        self._tokenizer = AutoTokenizer.from_pretrained("gpt2")

Ideally this functionality would be provided in the base class. I'd be happy to submit a PR for this and the documentation, but I'm writing to check first that HF would be interested in this.

FWIW, here is what's needed for a completely functional implementation that will work with the command-line custom subcommand (expanded from #757). This is much bigger than what's listed in the documentation, which is now out-of-date.

from lighteval.models.abstract_model import LightevalModel, ModelInfo
from transformers import AutoTokenizer

class TestModel(LightevalModel):
    def __init__(self, config) -> None:
	self.model_name = config.model_name
        self.model_definition_file_path = config.model_definition_file_path

        self.model_info = ModelInfo(
            model_name=config.model_name,
            model_sha="",
            model_dtype=None,
            model_size="",
        )

        # you need to provide a dummy tokenizer, and HF has nothing like an identity variant
        self._tokenizer = AutoTokenizer.from_pretrained("gpt2")

    @property
    def tokenizer(self):
        return self._tokenizer

    @property
    def add_special_tokens(self):
        return False

    @property
    def max_length(self) -> int:
        return 32768

    def greedy_until(self, requests, max_tokens=None, stop_sequences=None):
        for request in requests:
            print(vars(request))

    def loglikelihood(self, requests, log=True):
        # Implement loglikelihood computation
        pass

    def loglikelihood_rolling(self, requests):
        # Implement rolling loglikelihood computation
        pass

    def loglikelihood_single_token(self, requests):
        # Implement single token loglikelihood computation
        pass

[clefourrier]

It would be indeed great if you could open a PR for this! cc @NathanHB just for a double look but this lgtm

[clefourrier]

(side note: are you the sacrebleu matt post? - edit: checked your github, seems like you are! I'm a huge fan, loved the lib!!! 🤩 )

[mjpost]

It's me :) I'm using lighteval for the first time and it's clear that the niche that sacrebleu filled (which was at least in part to make it easy to download test sets) has been completely superseded. It's very nice!

I'll prep a PR for comment later in the day.