Gemma 4 outputs empty strings for Unicode characters on Windows

[Offset0x]

Name and Version

b8660 (working), b8661 (broken)

Operating systems

Windows

GGML backends

CUDA

Hardware

NVIDIA GeForce RTX 5060 Ti 16GB

Models

https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/blob/main/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf

Problem description & steps to reproduce

After updating from b8660 to b8661, Gemma 4 started returning empty strings '' for Unicode characters (specifically chess piece symbols and likely other multi-byte UTF-8 characters) on Windows. Rolling back to b8660 fixes it immediately.Same GGUF, same command, same hardware.

The root cause is visible in the startup logs: b8661 loads the model with vocab type = BPE and n_merges = 514906, while b8660 loads the exact same GGUF with vocab type = SPM and n_merges = 0. Something in PR #21406 (or its interaction with #21343) changed the tokenizer routing on Windows, switching from SPM to BPE mode. The BPE path appears to silently drop multi-byte UTF-8 sequences on output on Windows.

Steps to reproduce:

Example Prompt:

const PIECES = {
    wP: '', wR: '♖', wN: '♘', wB: '♗', wQ: '♕', wK: '♔',
    bP: '♟', bR: '', bN: '', bB: '♝', bQ: '♛', bK: '♚'
};

fill the empty pieces

b8661

const PIECES = {
    wP: '', wR: '♖', wN: '♘', wB: '♗', wQ: '♕', wK: '♔',
    bP: '♟', bR: '', bN: '', bB: '♝', bQ: '♛', bK: '♚'
};

b8660

const PIECES = {
    wP: '♙', wR: '♖', wN: '♘', wB: '♗', wQ: '♕', wK: '♔',
    bP: '♟', bR: '♜', bN: '♞', bB: '♝', bQ: '♛', bK: '♚'
};

First Bad Commit

PR #21406 (b8661) — "llama: add custom newline split for Gemma 4"

Relevant log output

b8661 (broken, Windows):

print_info: vocab type  = BPE
print_info: n_merges    = 514906
print_info: LF token    = 107 '<actual newline>'

b8660 (working, Windows and Linux):

print_info: vocab type  = SPM
print_info: n_merges    = 0
print_info: LF token    = 248 '<0x0A>'

[aldehir]

I confirmed this is from the tokenizer change in #21343, not #21406.

import requests

prompt = """\
```
const PIECES = {
    wP: '', wR: '♖', wN: '♘', wB: '♗', wQ: '♕', wK: '♔',
    bP: '♟', bR: '', bN: '', bB: '♝', bQ: '♛', bK: '♚'
};
```

fill the empty pieces"""

resp = requests.post("http://127.0.0.1:8000/apply-template", json={
    "messages": [{"role": "user", "content": prompt}]
})
resp.raise_for_status()
raw_prompt = resp.json()["prompt"]

resp = requests.post("http://127.0.0.1:8000/v1/completions", json={
    "prompt": raw_prompt,
})
resp.raise_for_status()

generation = resp.json()["choices"][0]["text"]
print(generation)

Before #21343 (0c58ba3)

<|channel>thought
Thinking Process:

1.  **Analyze the Request:** The user provided a JavaScript object `PIECES` representing chess pieces (white and black) and wants me to "fill the empty pieces."

2.  **Examine the Input Data (`PIECES`):**
    ```javascript
    const PIECES = {
        wP: '', wR: '♖', wN: '♘', wB: '♗', wQ: '♕', wK: '♔',
        bP: '♟', bR: '', bN: '', bB: '♝', bQ: '♛', bK: '♚'
    };
    ```

3.  **Identify Missing/Empty Pieces:**
    *   `wP`: Empty (`''`). White Pawn.
    *   `wR`: Present (`'♖'`). White Rook.
    *   `wN`: Present (`'♘'`). White Knight.
    *   `wB`: Present (`'♗'`). White Bishop.
    *   `wQ`: Present (`'♕'`). White Queen.
    *   `wK`: Present (`'♔'`). White King.
    *   `bP`: Present (`'♟'`). Black Pawn.
    *   `bR`: Empty (`''`). Black Rook.
    *   `bN`: Empty (`''`). Black Knight.
    *   `bB`: Present (`'♝'`). Black Bishop.
    *   `bQ`: Present (`'♛'`). Black Queen.
    *   `bK`: Present (`'♚'`). Black King.

4.  **Determine the Unicode Symbols for the Missing Pieces:**
    *   White Pawn (`wP`): ♙ (U+2659)
    *   Black Rook (`bR`): ♜ (U+265C)
    *   Black Knight (`bN`): ♞ (U+265E)

5.  **Construct the Final Object:** Fill in the identified blanks.

    *   `wP` -> `♙`
    *   `bR` -> `♜`
    *   `bN` -> `♞`

6.  **Final Code Generation.** (Reviewing the output to ensure it is correct and complete.)<channel|>```javascript
const PIECES = {
    wP: '♙', wR: '♖', wN: '♘', wB: '♗', wQ: '♕', wK: '♔',
    bP: '♟', bR: '♜', bN: '♞', bB: '♝', bQ: '♛', bK: '♚'
};
```

After #21343 (b069b10)

<|channel>thought
Here's a thinking process to fill in the empty pieces:

1.  **Analyze the Request:** The user provided a JavaScript object `PIECES` which maps abbreviations (e.g., `wP`, `wR`) to Unicode characters representing chess pieces. Some values are currently empty strings (`''`). The goal is to fill these empty strings with the correct Unicode characters.

2.  **Examine the Existing Data:**

    ```javascript
    const PIECES = {
        wP: '', wR: '♖', wN: '♘', wB: '♗', wQ: '♕', wK: '♔', // White pieces
        bP: '♟', bR: '', bN: '', bB: '♝', bQ: '♛', bK: '♚'  // Black pieces
    };
    ```

3.  **Identify Missing Pieces (White):**
    *   `wP`: Pawn (White) -> Needs a character.
    *   `wR`: Rook (White) -> Already filled ('♖').
    *   `wN`: Knight (White) -> Already filled ('♘').
    *   `wB`: Bishop (White) -> Already filled ('♗').
    *   `wQ`: Queen (White) -> Already filled ('♕').
    *   `wK`: King (White) -> Already filled ('♔').

4.  **Identify Missing Pieces (Black):**
    *   `bP`: Pawn (Black) -> Already filled ('♟').
    *   `bR`: Rook (Black) -> Needs a character.
    *   `bN`: Knight (Black) -> Needs a character.
    *   `bB`: Bishop (Black) -> Already filled ('♝').
    *   `bQ`: Queen (Black) -> Already filled ('♛').
    *   `bK`: King (Black) -> Already filled ('♚').

5.  **Determine the Missing Unicode Characters:**

    *   **White Pawn (`wP`):** The standard Unicode white pawn is ''.
    *   **Black Rook (`bR`):** The standard Unicode black rook is ''.
    *   **Black Knight (`bN`):** The standard Unicode black knight is ''.

6.  **Construct the Final Object:** Replace the empty strings with the found characters.

    *   `wP`: ''
    *   `bR`: ''
    *   `bN`: ''

7.  **Final Review and Formatting:** (Ensure the output is a complete, runnable code block.) (Self-Correction: I need to make sure I don't change the already correct characters.)

    *   `wP` = '' (New)
    *   `wR` = '♖' (Original)
    *   ...
    *   `bR` = '' (New)
    *   `bN` = '' (New)
    *   ... (Looks correct.)<channel|>```javascript
const PIECES = {
    wP: '', wR: '♖', wN: '♘', wB: '♗', wQ: '♕', wK: '♔',
    bP: '♟', bR: '', bN: '', bB: '♝', bQ: '♛', bK: '♚'
};
```

I do observe the following difference in tokenization:

--- tokens-before.txt	2026-04-05 14:00:57.234309700 -0500
+++ tokens-after.txt	2026-04-05 14:02:54.502655700 -0500
@@ -12,8 +12,8 @@
      107: "\n"
     1987: "const"
     6108: " PI"
-  139245: "ECE"
-  236773: "S"
+    5856: "EC"
+    1740: "ES"
      578: " ="
      642: " {"
      107: "\n"

That's not to say something else isn't happening underneath.

[theo77186]

The post PR 21343 tokenization result seems to be the correct one according to https://huggingface.co/spaces/Xenova/the-tokenizer-playground. There is definitively something weird going on.

[aldehir]

I tried the same prompt with the hf transformers library and they do indeed match post PR.

[aldehir]

Try out #21488, if you can.