-
Notifications
You must be signed in to change notification settings - Fork 26.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add stream
to pipeline parameters
#30487
Comments
Hi @not-lain, thanks for opening a feature request!
Could you elaborate on this a bit e.g. with a code snippet? Is is the streaming feature when generating you wish to be able to use? |
@amyeroberts |
I was thinking about integrating this with only text-generation models, but I think we can do that too with image-to-text models. this is a good resource for that: https://huggingface.co/blog/idefics#getting-started-with-idefics |
Thanks for sharing an example! I'm not sure this is really something we want to add to the pipelines. Pipelines are intended to be simple objects which enable users to get predictions in one line, they're not intended to support all transformers' functionality. In this case, I think it makes sense to leave streaming outside as it enables the user to have full control of the threads and yielding logic. cc @Rocketknight1 @gante for your thoughts |
Yeah, I'm on @amyeroberts's side here - pipelines are (imo) a sort of high-level "on-ramp" API for Once users are streaming output and working with threads/yielding/async/etc. they're probably advanced enough that they don't need the pipelines anyway. |
Personally would love to have streaming support in pipelines - it’s the one missing feature. Currently, streaming is quite difficult to use, but this would make it so much easier. |
FYI: we will be refactoring |
it's ok, I understand. |
generate refactor tracker: #30810 |
Feature request
add option to stream output from pipeline
Motivation
using
tokenizer.apply_chat_template
then other stuff thenmodel.generate
is pretty repetitive and I think it's time to integrate this with pipelines, also it's time to add a streaming pipeline too.Your contribution
I can provide this resource as a reference.
This is a pr I made with the requested feature https://huggingface.co/google/gemma-1.1-2b-it/discussions/14.
another tip I can provide is don't use yield and return in the same function, you should separate them (it's a python problem)
sadly I'm a bit busy lately to open a PR, but if I could find some time I'll try to help out.
The text was updated successfully, but these errors were encountered: