Isolated reproduction of https://github.com/huggingface/transformers/issues/38071

[willxxy]

System Info

name = "accelerate"
version = "1.12.0"
name = "transformers"
version = "4.57.3"

Python 3.11

Who can help?

@gante @ArthurZucker Related to warning from #38071 for Qwen/Qwen3-Next-80B-A3B-Instruct model

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import pipeline
from tqdm import tqdm
LANGUAGE_MODEL = "Qwen/Qwen3-Next-80B-A3B-Instruct"
def main():

    pipe = pipeline("text-generation", model=LANGUAGE_MODEL, device_map="auto")
    messages = [[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hdi"},
    ],[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hddi"},
    ],[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hasdasi"},
    ],[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hiasdsad"},
    ],[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hiasd"},
    ],[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hiasd"},
    ]]

    BATCH_SIZE = 3
    for out in tqdm(
        pipe(messages, max_new_tokens=4, batch_size=BATCH_SIZE),
        total=len(messages),
        desc="Batched inference",
    ):
        response = out[0]["generated_text"][-1]["content"].strip()
        print(response)

if __name__ == "__main__":
    main()

Adding

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Next-80B-A3B-Instruct", padding = "left")
pipe = pipeline("text-generation", model=LANGUAGE_MODEL, tokenizer=tokenizer, device_map="auto")

Does not make the warning go away.

Expected behavior

(ecg-preprocess) (ecg-encoder) -bash-4.4$ CUDA_VISIBLE_DEVICES=4,5,6,7 uv run src/test.py
The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Loading checkpoint shards: 100%|█████████████████| 41/41 [00:31<00:00,  1.28it/s]
Device set to use cuda:0
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
Batched inference:   0%|                                   | 0/6 [00:00<?, ?it/s])  
Hi!
Hi! It looks
Hello! It seems
Hello! It seems
)
) I'm here
Batched inference: 100%|████████████████████████| 6/6 [00:00<00:00, 82782.32it/s]

[ArthurZucker]

@willxxy this should be "easy" to fix, do you want to open a PR ?

[ChiragSW]

Hi @willxxy there seems to be a common mistake.
while adding this:

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Next-80B-A3B-Instruct", padding = "left")

use this instead:

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Next-80B-A3B-Instruct", padding_side = "left")

please let me know if this fixes the issue.

[Jignesh03-T]

Hi, I would like to work on this issue.
Can I try to reproduce and submit a fix?

[nguyencongtuyenlp]

Hi! I can take this. I’ll open a PR that (1) clarifies the warning to use padding_side="left" and (2) ensures the text-generation pipeline uses left-padding for decoder-only models in batched generation (with a regression test).

[ManasVardhan]

I've opened a PR to fix this: #44021

Root cause: Two issues combine to produce this warning:

1. TextGenerationPipeline doesn't set padding_side='left' for decoder-only models, so the tokenizer's default 'right' padding is used during batch collation — meaning shorter sequences get right-padded, which is incorrect for decoder-only generation.

2. The right-padding detection in generate() uses a heuristic that checks if the last token equals pad_token_id. For Qwen3, pad_token_id == bos_token_id == 151643 (<|endoftext|>), which can cause false positives even in non-padded scenarios.

Fix:

• Set padding_side='left' automatically in TextGenerationPipeline.__init__
• Use attention_mask (when available) for more reliable right-padding detection in generate()

Note: Setting padding='left' (as mentioned in the issue) doesn't work because the correct parameter is padding_side='left' — but even with the correct parameter, the pipeline overrides this during collation, so fixing it at the pipeline level is the proper solution.