feat: multiple performance configs in a submission config

[tianmu-li]

Each submission config contains accuracy config and a single performance config. When sweeping through concurrencies, requiring an accuracy run for each concurrency value increases runtime, especially when the accuracy dataset takes similar/longer time compared to the performance dataset. Propose to allow multiple performance configs within a submission config.
Before:

  load_pattern:
    type: "concurrency"
    target_concurrency: 512

After:

  load_pattern:
    type: "concurrency"
    target_concurrency: 64,128,256,512

This allows amortization of doing accuracy runs and encourages using a single endpoint config for different concurrency levels.
Several possible considerations:

1. target concurrency values are strictly ascending
2. Requests for a new concurrency are issued after requests for the current concurrency value are all received (not issued).
3. Accuracy run uses the highest concurrency

[arekay-nv]

Thanks @tianmu-li .
The list of concurrency part should be easy to implement. We had a draft version which almost worked, the only change needed was ensuring the reporter starts up with a clean state after each run. There is a major refactor in place for the metrics, so we can revisit this issue after we are done there.
A second concern will be using the max concurrency for the accuracy. As brought up in the meeting, there are cases where the server might drop requests due to network issues. This shows up more significantly with higher concurrency values - so we might end up invalidating the whole run due to low accuracy score while the same setup would have passed with lower concurrency values.

[tianmu-li]

Several ideas about accuracy run failing at high concurrencies:

1. Per-request retries. If a request is dropped, the endpoint client tries to send the same request again, up to a specific max retry count (e.g.: 3). This will likely affect performance during performance runs, so submitters are still be motivated to avoid that as much as possible.
2. Per-run retries. If an accuracy run fails at the highest specified concurrency, the endpoint client retries accuracy run with a lower concurrency until it passes. This will still invalidate performance at failed concurrency values.