-
Notifications
You must be signed in to change notification settings - Fork 26.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate: assistant should be greedy in assisted decoding #30778
Conversation
@@ -496,6 +496,11 @@ def validate(self, is_init=False): | |||
greedy_wrong_parameter_msg.format(flag_name="top_p", flag_value=self.top_p), | |||
UserWarning, | |||
) | |||
if self.min_p is not None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This check was missing 👼
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for fixing!
Will this result in different outputs for the users? If so, we might want to make sure to run any slow tests.
Question for the future: would we ever want to enable a samplng strategy for the assistant?
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Nope :) The only difference is execution time. (The output is entirely defined by the main model and its generation config, both untouched in this PR)
Perhaps. A potential strategy for improvement is to sample several continuations and try them all as candidates. But that would require significant changes over the current code 👀 |
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
@amyeroberts my answer did not age well (#32867 -- some randomness was lost as a result of this change) |
What does this PR do?
Restores a previous behavior -- forces the assistant in assisted decoding to do greedy decoding.
While providing this answer, I noticed that we dropped a property in the refactor that was present in the original implementation: the assistant should be greedy.
Why? We want the assistant to maximize the number of matches with the original model, to result in the maximum speedup. Remember: the main model always calls the shots, the output text is never changed regardless of the flags we set for the assistant. By forcing the assistant to be greedy, we pick its most likely tokens. If the assistant model has a distribution aligned with the main model, these tokens will also be the main model's most likely tokens, increasing the number of matches.
When running my original benchmarks (https://github.com/gante/huggingface-demos/tree/main/experiments/faster_generation), this resulted in a ~20% speedup @ temperature=0.5 over the current version of the code.