Pretrained Llama tokenizers don't yield the expected tokenization of "\n"

[JulienVig]

System Info

TypeScript 5.5.4
transformers.js 3.0.2
Node.js v20.170

Environment/Platform

• Website/web-app
• Browser extension
• Server-side (e.g., Node.js, Deno, Bun)
• Desktop app (e.g., Electron)
• Other (e.g., VSCode extension)

Description

Using some pretrained tokenizers doesn't yield the same tokenization of "\n" or " \n" as Tiktokenizer or Xenova's playground.

For example, Xenova/llama-3-tokenizer tokenizes "\n" as [198] and " \n" as [720]
In both playgrounds, selecting Llama 3 in Xenova's playground and meta-llama/Meta-Llama-3-8B in Tiktokenizer, the Llama 3 tokenizer should tokenize "\n" [1734] as and " \n" as [1144, 77]

Similarly for Llama 2, Xenova/llama-tokenizer tokenizes "\n" as [1, 29871, 13] while Xenova's playground yields [1, 320, 29876].

Reproduction

import { AutoTokenizer } from "@huggingface/transformers";

const tokenizer = await AutoTokenizer.from_pretrained("Xenova/llama-3-tokenizer");

const tokens = tokenizer("\n", { return_tensor: false }).input_ids;
console.log(tokens) // prints [198] while [1734] is expected

Similar issue with Xenova/llama-tokenizer.

[xenova]

Hi there 👋 can you reproduce this with the python transformers library?

[JulienVig]

Hi! I actually get consistent results with the python tokenizers:

tokenizer = Tokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
print(tokenizer.encode("\n").ids) # [1, 29871, 13] same as Xenova/llama-tokenizer

tokenizer = Tokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
print(tokenizer.encode("\n").ids) # [128000, 198]  Xenova/llama-3-tokenizer doesn't include special token by default
print(tokenizer.encode("\n", add_special_tokens=False).ids) # [198] same as Xenova/llama-3-tokenizer

So both javascript and python yield different tokenization from the playgrounds. Am I comparing different tokenization settings?

[ingo-m]

I am also wondering what the correct newline token for Llama 3 models is, I opened this issue: huggingface/transformers#35923

My understanding is that in a normal python string ("\n"), the newline gets represented as a single character, which gets tokenized as 198:

from transformers import AutoTokenizer

model_id = "meta-llama/Llama-3.2-1B"

tokenizer = AutoTokenizer.from_pretrained(model_id)

# Normal string

newline_token = tokenizer("\n")["input_ids"]
print(newline_token)
# Single newline in normal string:
# [128000, 198]

Two consecutive newlines get represented as 271:

newline_token = tokenizer("\n\n")["input_ids"]
print(newline_token)
# Two newlines in normal string:
# [128000, 271]

We can also use raw strings in which the "\" doesn't act as an escape character, so r"\n" is represented as the two separate characters "\" and "n". In that case, r"\n" get tokenized as 1734:

newline_token = tokenizer(r"\n")["input_ids"]
print(newline_token)
# Single newline in raw string:
# [128000, 1734]

And double raw string newlines get represented as consecutive 1734:

newline_token = tokenizer(r"\n\n")["input_ids"]
print(newline_token)
# Double newline in raw string:
# [128000, 1734, 1734]

The Llama prompt template contains double newlines [1], so it seems this is relevant, but I don't know which one (198 or 271 or 1734) is the expected newline token.

[1] https://www.llama.com/docs/model-cards-and-prompt-formats/meta-llama-3/#llama-3-instruct

[xenova]

Closing as our implementation matches the Transformers v5 implementation as of Transformers.js v4.

You can also check out our new tokenizers.js library: https://github.com/huggingface/tokenizers.js, which extracts the logic out to a standalone library.