New stream

[ArthurZucker]

New api:

from tokenizers import Tokenizers
from tokenizers.decoders import DecodeStream
stream = DecodeStream([19567,255, 255]) # init the state with prefill
out = stream.step(tokenizer, 109)
'ั'

and:

from tokenizers import Tokenizers
from tokenizers.decoders import DecodeStream
stream = DecodeStream([19567,255]) # init the state with prefill
out = stream.step(tokenizer, [255,109])  # imagine you had an assitant that generated 2 tokens at the same time
'ั'

Non breaking:

from tokenizers import Tokenizers
from tokenizers.decoders import DecodeStream
stream = DecodeStream(False)
stream.step(tokenizer, 19567)
stream.step(tokenizer, 255)
stream.step(tokenizer, 19567)
out = stream.step(tokenizer, 109)
out
'ั'
tokenizer.encode("อั").ids
[19567, 255, 19567, 109]
tokenizer.decode(tokenizer.encode("อั").ids)
'อั'

This could be somewhat expected, but if you initialize your stream, say with [19567, 255, 19567] first, then you should be able to properly get 'อั' if you step with 109.

We can't go against the fact that [19567, 109] is a "valid" token, so in the context of token generation, we cannot go against this (the first token will always be emitted because it is a valid token). However initializing the stream should still be helpful

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[McPatate]

🔥

[njhill]

Thanks @ArthurZucker!