PoC: dynamic SUTs

[rogthefrog]

As an operator, I want to be able to pass a sensible name for a SUT that isn't registered to modelgauge or modelbench
Modelgauge and modelbench should try to create a new SUT from that name and run the requested job with it
So the code doesn't need to be updated whenever we want to add a new SUT that behaves like known SUTs.

Story

Important

This only works with huggingface proxied inference endpoints for now. The dynamic SUT names must be

hf/<provider>/<vendor>/<model>

e.g.

hf/nebius/google/gemma-3-27b-it

How To Test

deepseek-ai/DeepSeek-V3 is available on several providers (replicate, together, etc).

You can test by requesting any one of those:

poetry run modelbench benchmark -m 1 -s hf/novita/deepseek-ai/DeepSeek-V3
poetry run modelbench benchmark -m 1 -s hf/together/deepseek-ai/DeepSeek-V3

Additional Information

SUT IDs And Names Behave The Same Way

You can pass a dynamic SUT name the same way you can pass a known SUT ID.

poetry run modelbench benchmark -m 1 -s hf/nebius/google/gemma-3-27b-it -s llama-3-70b-chat
Starting run for ['general_purpose_ai_chat_benchmark-1.0-en_us-demo-default'] over ['llama-3-70b-chat', 'google-gemma-3-27b-it-hf-nebius']

If you pass in a dynamic SUT name that translates to a known SUT, that's fine, it'll work.

poetry run modelbench benchmark -m 1 -n hf/nebius/google/gemma-3-27b-it # this is a known SUT
Starting run for ['general_purpose_ai_chat_benchmark-1.0-en_us-demo-default'] over ['google-gemma-3-27b-it-hf-nebius']

How Do I Find Models With Serverless Providers?

import huggingface_hub as hfh
name = "deepseek-ai/DeepSeek-V3"
inference_providers = hfh.model_info(name, expand="inferenceProviderMapping")
print(inference_providers.inference_provider_mapping)

Alternate Providers

If you change find_alternative to True in huggingface_sut_maker. HuggingFaceChatCompletionServerlessSUTMaker.find then the program will automatically look for the same model on a different serverless provider, if the provider you requested isn't available.

Dedicated Endpoints

Those are TBD. The code only turns one on if it already exists and is asleep. It will not create a new one. That's easy to add.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[rogthefrog]

This looks great! One thing I'm a little concerned about is the conflation between SUT uids and names. I think we should either a) go back to when you had a separate argument to pass in dynamic SUT names or b) request dynamic suts at the cli using a structured UID (with hyphens) instead of a name.

I also don't like that typos in SUT uids will be identified as dynamic SUT names and then result in an inaccurate error message (i.e. "couldnt make this dynamic sut" instead of "invalid SUT UID, please use one of the following..")

One reason I like using the same argument for known SUT IDs and named dynamic SUTs is that the user doesn't need to know if a SUT already exists. If you pass in a dynamic SUT name that can be parsed into an existing SUT id, then the existing SUT will be used.

If we constrain what a dynamic SUT name can look like, e.g. a SUT is dynamic if and only if the identifier passed to modelbench includes a slash, would that solve the typo issue? We could make the code try to create a dynamic SUT only if the string includes a slash, and SUT IDs without slashes but with a typo would then be rejected with the right error.