CI refactor; allow setting args in config

[artemorloff]

might be much more convenient to pass path to config file to run evaluation via lm-eval

PR allows a user to pass --config PATH_TO_YAML_FILE arg that contains the same args as the console evaluation script. With recent updates there are too much args to track them on console. It may be easier to write one yaml file (provide example in configs/ dir) and pass it into the script.

Also PR allows to override this config. If user provides --config PATH param along with other params, they prevail over the same params in config. So, user may define all params in yaml file and change only those that they need right now, others remain as they are in yaml file. To grant that all params from console will be treated with "high priority" added action to each param that just adds arg indicating that some param was explicitly stated by the user from console. Those params won't be changed by config file.

Also config is parsed right after reading to avoid using simple_parse_args_string and make the code more unified and simplified.

[artemorloff]

use case: CUDA_VISIBLE_DEVICES=0 lm_eval --config configs/default_config.yaml

or one can explicitly state: ``CUDA_VISIBLE_DEVICES=0 lm_eval --config configs/default_config.yaml --limit 10 --model hf

[StellaAthena]

This is a really good idea. I haven't looked at the implementation itself but I think that this functionality is going to be hugely valuable. I also strongly agree with the choice to allow both CLI arguments and yaml arguments and having the CLI take precidence over the yaml arguments.

[artemorloff]

@baberabb @StellaAthena i think this would be a great feature for all users - to keep configs in a dir and no need in writing the flags in terminal

[baberabb]

Hi @artemorloff!. This is generally great, especially as the number of arguments which need to pass keep in increasing. One thing: have you considered using a dataclass? Something like:

@dataclass
class EvalConfig:
    model: str = "hf"  
    tasks: Optional[str] = None
    ...
    
    @classmethod
    def from_yaml(cls, path):
        # load and convert yaml
        
    @classmethod
    def from_args(cls, args, yaml_path=None):
        ...

Think this would make the code easier to read, and more maintainable. Happy to sketch it out more if you think it's helpful.

[artemorloff]

@baberabb seems a good idea. will prepare the draft

[artemorloff]

@baberabb added prototype of Eval config class that is used to unify evaluation. Based on BaseModel so that it may later be reworked the way that params are validated and updated inside config class, not inside main.py funcs

[baberabb]

Hi @artemorloff! tysm for working on this. I'll take a closer look this week, but feel free to ping me in case I don't respond by Friday. Couple of initial points (mostly backward compatibility):

1. we don't want to modify the args of evaluate too much and keep it backward compatible as many people use it directly in their training code and libraries.
2. I like the addition of pydantic as well, but need to investigate that it doesn't create any dependency issues (esp. when using in training loop).

[baberabb]

Hi @artemorloff! thanks again for working on this, and sorry for the late review. The logic overall looks pretty great, but made some slight changes:

1. Use dataclass, rather than pydantic. We might want to reconsider this later, but lets not add the dependency just yet.
2. Still pass all the args to simple_evaluate explicitly as we normally do, so that's it easier when calling outside of cli.

Let me know what you think!