[Feature]: Image-Modality Throughput Benchmark

[lk-chen]

🚀 The feature, motivation and pitch

This is a subset of #8385. This issue is intended to track the effort of enabling throughput benchmark for image-modal models.

This is a reasonably large feature, and will span the work among multiple PRs.

Alternatives

Ad-hoc scripts for each model.

Additional context

see #8385

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ywang96]

This is great! I think as part of the effort, it would be great if we can refactor out the dataset generation used for this benchmark if you have then bandwidth.

For the context, today the dataset sampling and generation is done within the benchmark files themselves, but I think we should consolidate the datasets used in these benchmarks into one file (say, benchmark_datasets.py) and import from it instead. This also allows us to plug in custom datasets more easily in the future.