-
Notifications
You must be signed in to change notification settings - Fork 961
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Distributed inference example for llava_next #3179
base: main
Are you sure you want to change the base?
Distributed inference example for llava_next #3179
Conversation
Hi @sayakpaul @a-r-r-o-w! This PR is ready for review, please, take a look. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this so quickly. Left some comments.
indices = np.arange(0, total_frames, total_frames / 8).astype(int) | ||
video = read_video_pyav(container, indices) | ||
|
||
conversations = [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we could repurpose this to just yield captions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @sayakpaul! What do you mean by repurpose here? The idea of this example was to provide user prompts (Questions) to certain videos and process it. If we want to get just captions, what exactly are we going to do with it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Train text-to-video models, for one
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sayakpaul, please, take a look on recent changes, what do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @sayakpaul! Do you have any comments on recent changes? I added video dataset loading and splitting it into batches, and then, on every video batch, we distribute prepared prompts to generate answers.
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
91ee412
to
0a58faa
Compare
Add distributed inference example for LLaVA-NeXT-Video-7B-hf
Before submitting
Pull Request section?
to it if that's the case. Link: [Community Contributions] examples on distributed inference using 🤗 Accelerate #3078
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@sayakpaul @a-r-r-o-w @muellerzr