Distributed inference example for llava_next #3179
Conversation
|
Hi @sayakpaul @a-r-r-o-w! This PR is ready for review, please, take a look. |
| indices = np.arange(0, total_frames, total_frames / 8).astype(int) | ||
| video = read_video_pyav(container, indices) | ||
|
|
||
| conversations = [ |
There was a problem hiding this comment.
Maybe we could repurpose this to just yield captions?
There was a problem hiding this comment.
Hey @sayakpaul! What do you mean by repurpose here? The idea of this example was to provide user prompts (Questions) to certain videos and process it. If we want to get just captions, what exactly are we going to do with it?
There was a problem hiding this comment.
Train text-to-video models, for one
There was a problem hiding this comment.
@sayakpaul, please, take a look on recent changes, what do you think?
There was a problem hiding this comment.
Hey @sayakpaul! Do you have any comments on recent changes? I added video dataset loading and splitting it into batches, and then, on every video batch, we distribute prepared prompts to generate answers.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
VladOS95-cyber
force-pushed
the
add-video-capture-example-on-distributed-inference
branch
from
91ee412
to
0a58faa
Compare
Add distributed inference example for LLaVA-NeXT-Video-7B-hf
Before submitting
Pull Request section?
to it if that's the case. Link: [Community Contributions] examples on distributed inference using 🤗 Accelerate #3078
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@sayakpaul @a-r-r-o-w @muellerzr