Fix revision arg for vLLM tokenizer

[lewtun]

This PR fixes a nasty bug where the tokenizer revision is set to the default value of main, irrespective of the revision provided in the model config. The reason is because tokenizer_revision is not a proper kwarg for get_tokenizer(), so we always default to main when loading the tokenizer in vllm.

Tested on this dummy model which has different chat templates on the main and new_chat_template branches: https://huggingface.co/lighteval/different-chat-templates-per-revision

The unit test fails on main with (as expected):

E       AssertionError: 'chat-template-on-main' != 'chat-template-on-revision'
E       - chat-template-on-main
E       ?                  ^^
E       + chat-template-on-revision
E       ?                  ^^^ +++

tests/models/vllm/test_vllm_model.py:17: AssertionError

It passes on this branch :)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[clefourrier]

Haa good catch thanks!
Could it be an arg which changed name in vllm history?

[lewtun]

Could it be an arg which changed name in vllm history?

No idea, but could well be...

[clefourrier]

Hm, I guess since we pin vllm's lower bound we don't need to care ^^

[edbeeching]

Thanks