We have 2 ways to launch a benchmark: yaml config and CLI. For the official submission, we need to define a "ruleset" that locks part of the config (e.g. model, parameter, SLA, etc), and allow the users to work on a subset of the UserConfig (e.g. target_qps, concurrency, endpoints, http client settings etc).
Define and implement the ruleset and integration
We have 2 ways to launch a benchmark: yaml config and CLI. For the official submission, we need to define a "ruleset" that locks part of the config (e.g. model, parameter, SLA, etc), and allow the users to work on a subset of the UserConfig (e.g. target_qps, concurrency, endpoints, http client settings etc).
Define and implement the ruleset and integration