Generate: assistant should be greedy in assisted decoding #30778
Conversation
| @@ -496,6 +496,11 @@ def validate(self, is_init=False): | |||
| greedy_wrong_parameter_msg.format(flag_name="top_p", flag_value=self.top_p), | |||
| UserWarning, | |||
| ) | |||
| if self.min_p is not None: | |||
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Nope :) The only difference is execution time. (The output is entirely defined by the main model and its generation config, both untouched in this PR)
Perhaps. A potential strategy for improvement is to sample several continuations and try them all as candidates. But that would require significant changes over the current code 👀 |
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
@amyeroberts my answer did not age well (#32867 -- some randomness was lost as a result of this change) |
What does this PR do?
Restores a previous behavior -- forces the assistant in assisted decoding to do greedy decoding.
While providing this answer, I noticed that we dropped a property in the refactor that was present in the original implementation: the assistant should be greedy.
Why? We want the assistant to maximize the number of matches with the original model, to result in the maximum speedup. Remember: the main model always calls the shots, the output text is never changed regardless of the flags we set for the assistant. By forcing the assistant to be greedy, we pick its most likely tokens. If the assistant model has a distribution aligned with the main model, these tokens will also be the main model's most likely tokens, increasing the number of matches.
When running my original benchmarks (https://github.com/gante/huggingface-demos/tree/main/experiments/faster_generation), this resulted in a ~20% speedup @ temperature=0.5 over the current version of the code.