Stochastic Sampling for Speculative Decoding example

[mscheong01]

Feature Description

It seems that the speculative decoding example in this repo only utilizes greedy sampling. Are there any plans for supporting stochastic sampling as well? If not so, could I maybe give it a try based on the paper and implementations inside https://github.com/lucidrains/speculative-decoding?

[ggerganov]

Yes, I was thinking about adding stochastic sampling to the speculative example, but haven't gotten to that yet. If you want to give it a try - please go ahead

[BarfingLemurs]

Any numbers? I had thought normal sampling parameters work (70B Q4 + 1.1B Q4 = ~1.3x increase), but don't remember some details.