Skip to content

Cache tokenizer reverse byte map (bytesToString perf) #24

Description

@m96-chan

tokenizer.ts:382-399 rebuilds the unicode-to-byte reverse map from scratch on every call to bytesToString():

private bytesToString(token: string): string {
    const b2u = Tokenizer.getByteToUnicode();
    const u2b = new Map<string, number>();   // rebuilt every call (256 entries)
    for (const [k, v] of b2u) {
      u2b.set(v, k);
    }
    // ...
}

This is called once per token during streaming generation — a hot path. Caching the reverse map as a static field eliminates 256 Map.set operations per decoded token.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions