Skip to content

Bpe.UnknownToken is mutable #7006

Closed
Closed
@stephentoub

Description

@stephentoub

Should it be? Changing it post construction can change the results of tokenizing. If this is supposed to be something created as a singleton and shared, UnknownToken should likely not be allowed to be changed post construction (either only as a ctor argument or at least making the property's setter instead be init).

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions