[Community Contributions] examples on distributed inference using 🤗 Accelerate

[sayakpaul]

The inference/distributed directory houses examples on running distributed inference with accelerate:

• Phi2 for language generation
• Stable Diffusion for image generation

The strategy followed there is to load an entire model onto each GPU and sending chunks of a batch through each GPU’s model copy at a time. Synthetic data generation has become an essential toolkit for every ML Engineer. So, it'd be beneficial to extend these examples to include some more use cases:

• Image captioning
• Speech data generation

Some nice to haves:

• Include artifact serialization as done in this
• Keep the artifact serialization code under a thread to not block GPU execution

How can you help?

You could help us contribute an example on any of the above-mentioned use cases or you can come up with your own 🤗 Help us make the art of synthetic data generation scalable, easy, and accessible.

[sayakpaul]

For image recaptioning, I have a quick and dirty PoC here that assumes streaming from webdataset and uses multiprocessing from torch: https://gist.github.com/sayakpaul/dfb26d6562b62ba8c2f613dc912a28a8#file-generate_captions_multigpu-py. Would be nice to make it more efficient and leverage accelerate to make things cleaner and simpler.

[VladOS95-cyber]

Hey @sayakpaul @muellerzr! If it is still actual, I am going to add one more example for Video captioning using LLaVA-NeXT-Video-7B-hf

[sayakpaul]

Of course it's up, that would be so cool! Cc: @a-r-r-o-w