[Bug]: Deepseek resoning content is coming as null and the think content is going inside content when using vllm-openai v0.7.2 docker containers

[dbanka]

Your current environment

The output of `python collect_env.py`

INFO 02-16 21:08:24 __init__.py:190] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.31.4
Libc version: glibc-2.35

Python version: 3.12.9 (main, Feb  5 2025, 08:49:00) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.10.230-223.885.amzn2.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 550.144.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               4
On-line CPU(s) list:                  0-3
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 7R32
CPU family:                           23
Model:                                49
Thread(s) per core:                   2
Core(s) per socket:                   2
Socket(s):                            1
Stepping:                             0
BogoMIPS:                             5600.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            64 KiB (2 instances)
L1i cache:                            64 KiB (2 instances)
L2 cache:                             1 MiB (2 instances)
L3 cache:                             8 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-3
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:   Mitigation; safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.1
[pip3] torch==2.5.1
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.48.2
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.7.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-3     0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NCCL_VERSION=2.17.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
NVIDIA_CUDA_END_OF_LIFE=1
CUDA_VERSION=12.1.0
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY```

</details>


### 🐛 Describe the bug

I am running "deepseek-ai/DeepSeek-R1-Distill-Llama-8B" model using docker with parameters:
 --enable-reasoning  --reasoning-parser deepseek_r1   --max-model-len 16800

As per the documentation, all the <think></think> content should go under the key reasoning_content.

When I am testing the deployment using /chat/completions api , the reasoning content is coming as null and the actual reasoning content is going inside the content key.

Request:
{
  "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
  "messages": [
    {"role": "user", "content": "9.11 and 9.8, which is greater?"}
  ]
}


Response:
{
    "id": "chatcmpl-f55ac124e83d4df297cbdd133b28cc5f",
    "object": "chat.completion",
    "created": 1739728395,
    "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "reasoning_content": null,
                "content": "First, I need to compare the two numerical values, 9.11 and 9.8.\n\nTo make the comparison easier, I'll align both numbers by their decimal places. I can rewrite 9.8 as 9.80.\n\nNow, I'll compare the numbers digit by digit. Both have the same whole number part, 9.\n\nNext, I'll look at the tenths place: 1 in 9.11 and 8 in 9.80. Since 8 is greater than 1, 9.80 is greater than 9.11.\n\nAlternatively, I can subtract 9.11 from 9.8:\n9.8 - 9.11 equals 0.69, which is positive. Therefore, 9.8 is greater than 9.11.\n</think>\n\nTo determine which number is greater between **9.11** and **9.8**, follow these steps:\n\n1. **Align the Decimal Places:**\n   - **9.11** has two decimal places.\n   - **9.8** can be written as **9.80** to have two decimal places as well.\n\n2. **Compare the Numbers Digit by Digit:**\n   - **Whole Number Part:** Both numbers have **9** in the whole number part. They are equal so far.\n   - **Tenths Place:** Compare **1** (from 9.11) and **8** (from 9.80).\n     - Since **8 > 1**, **9.80** is greater than **9.11**.\n\n3. **Alternative Method - Subtraction:**\n   - Subtract **9.11** from **9.8**:\n     \\[\n     9.80 - 9.11 = 0.69\n     \\]\n   - Since the result is **positive**, **9.8** is greater than **9.11**.\n\n**Final Answer:**\n\\boxed{9.8}",
                "tool_calls": []
            },
            "logprobs": null,
            "finish_reason": "stop",
            "stop_reason": null
        }
    ],
    "usage": {
        "prompt_tokens": 18,
        "total_tokens": 426,
        "completion_tokens": 408,
        "prompt_tokens_details": null
    },
    "prompt_logprobs": null
}




### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.

[anlongfei]

me too

[majestichou]

Me too.
I use docker image: vllm/vllm-openai:0.7.2 to run DeepSeek-R1-Distill-Llama-70B.

I refer to vllm's official documentation https://docs.vllm.ai/en/stable/features/reasoning_outputs.html and run the following program:

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

# Round 1
messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
response = client.chat.completions.create(model=model, messages=messages)

reasoning_content = response.choices[0].message.reasoning_content
content = response.choices[0].message.content

print("reasoning_content:", reasoning_content)
print("content:", content)

The output is as follows:

reasoning_content: None
content: First, I compare the whole number parts of both numbers. Both 9.11 and 9.8 have the whole number 9.

Since the whole numbers are equal, I move on to compare the decimal parts. In 9.11, the decimal part is 0.11, while in 9.8, it's 0.80 when expressed with two decimal places.

Comparing 0.80 and 0.11, I see that 0.80 is greater than 0.11.

Therefore, 9.8 is greater than 9.11.
</think>

**Question:**
Which is greater: \(9.11\) or \(9.8\)?

**Solution:**
To determine which number is greater between \(9.11\) and \(9.8\), let's compare them step by step.

1. **Compare the Whole Number Part:**
   - Both numbers have the same whole number part: \(9\).

2. **Compare the Decimal Parts:**
   - For \(9.11\), the decimal part is \(0.11\).
   - For \(9.8\), the decimal part is \(0.80\).

3. **Determine Which Decimal is Greater:**
   - \(0.80\) is greater than \(0.11\).

Since the whole number parts are equal and the decimal part of \(9.8\) is greater than that of \(9.11\), we can conclude that:

\[
\boxed{9.8\ >\ 9.11}
\]

Then I use docker image: vllm/vllm-openai:0.7.2 to run DeepSeek-R1-Distill-Qwen-32B, the bug is same!!!

[linpan]

@majestichou reasoning in think tag

[Nietism]

Notice whether there is <think>\\n in the end of the chat_template in your tokenizer_config.json.

8 days ago, there is an upate about the tokenizer_config.json of deepseek-r1 series models
(https://huggingface.co/deepseek-ai/DeepSeek-R1):

Additionally, we have observed that the DeepSeek-R1 series models tend to bypass thinking pattern (i.e., outputting "<think>\n\n</think>") when responding to certain queries, which can adversely affect the model's performance. To ensure that the model engages in thorough reasoning, we recommend enforcing the model to initiate its response with "<think>\n" at the beginning of every output.

If you use the latest version of tokenizer_config.json, you will not get the start think token <think> in the response. And 0.7.2 vllm's implementation about separation of reasoning content and normal content is:

# vllm/entrypoints/openai/reasoning_parsers/deepseek_r1_reasoning_parser.py
    def extract_reasoning_content(
            self, model_output: str, request: ChatCompletionRequest
    ) -> Tuple[Optional[str], Optional[str]]:

        # Check if the model output contains the <think> tokens.
        if (self.think_start_token not in model_output
                or self.think_end_token not in model_output):
            return None, model_output
...

Therefore, you can only get an empty reasoning content and response.choices[0].message.content is all content.

You may go back to the old version tokenizer_config.json, and you should get the right reasoning content and content.
(delete the <think>\\n in the end of the chat_template in your tokenizer_config.json)

PS:
The implementation of vllm seems good now:

vllm/vllm/entrypoints/openai/reasoning_parsers/deepseek_r1_reasoning_parser.py

Lines 132 to 138 in 238dfc8

	else:
	# Add a start token if it's missing to keep compatibility.
	if self.think_start_token not in model_output:
	model_output = f"{self.think_start_token}{model_output}"
	# Use a regex to find the reasoning content
	reasoning_content = self.reasoning_regex.findall(model_output)[0]

[louislepage]

Thanks for the fast help @Nietism !
I just tested this with the older tokenizer revision and it worked like a charm.

The easiest fix for the latest container (at the current time I believe thats v0.7.2) is to use tokenizer from one before the change. For deepseek-ai/DeepSeek-R1-Distill-Qwen-14B that would be --tokenizer-revision 6453600843497f04e63f069049e92bf221d9e000

This returns:

{
  "id": "chatcmpl-xxx",
  "object": "chat.completion",
  "created": 1739788399,
  "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "reasoning_content": null,
        "content": "<think>\nOkay, so I want to find out who won the World Series in 2020. I'm not entirely sure about the exact details, but I think I can figure it out by breaking it down step by step.\n\nFirst, I know that the World Series is the championship series in Major League Baseball (MLB) in the United States. It's played between the winners of the American League (AL) and the National League (NL). The team that wins the World Series is declared the champion of baseball for that year.\n\nI'm trying to remember the teams that were in the 2020 World Series. I think the Los Angeles Dodgers might have been in it because they've been a strong team for a while. But I'm not certain. I also recall hearing something about the Tampa Bay Rays being in the World Series that year. So maybe those were the two teams competing against each other.\n\nNext, I need to recall which team actually won. I think the Dodgers won, but I want to be sure. I remember that the World Series was in a best-of-seven format, so the first team to four wins takes the series. I think the Dodgers won in a short series, maybe only six games. They might have had some strong pitchers and a good lineup.\n\nI'm trying to pinpoint the dates as well. The World Series usually starts in October, and 2020 was during the COVID-19 pandemic, so there might have been some unique circumstances, like the games being held in a neutral location or maybe even in a bubble due to health concerns. I think the games were played at the Globe Life Field in Arlington, Texas, because that's where I remember watching some of the games.\n\nThinking about the highlights, I remember a game where the Dodgers' pitcher, maybe Clayton Kershaw, pitched a lot in the series. But honestly, I'm not too sure about the individual performances. It's more about the team result. \n\nI should also consider if there were any dominant performances by either team. I think the Dodgers had a solid lead in the series, maybe winning four games without the Rays winning any. That would make it a quick sweep, but I'm not certain if that's what happened. \n\nWait, but if the Rays were the underdogs, perhaps they put up a good fight. Maybe they won some games, but in the end, the Dodgers pulled through. I'm pretty confident it's the Dodgers, but I need to make sure of the details before I state that as an answer.\n\nIn summary, I think the Los Angeles Dodgers won the World Series in 2020, defeating the Tampa Bay Rays, and it was held at the Globe Life Field. They probably won the series in six games, but I'm a bit unsure about the exact number of games. Either way, the Dodgers are the champions of that year.\n</think>\n\nThe Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays. The series was held at the Globe Life Field in Arlington, Texas, and the Dodgers triumphed in six games to claim their first championship since 1988.",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 21,
    "total_tokens": 668,
    "completion_tokens": 647,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null
}

And with the reasoning parser enabled, this gives:

{
  "id": "chatcmpl-xxx",
  "object": "chat.completion",
  "created": 1739787402,
  "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "reasoning_content": "\nAlright, I need to figure out who won the World Series in 2020. I know the World Series is the championship series in Major League Baseball, usually played in October, and it’s between the American League and National League champions.\n\nSo, in 2020, things were a bit unusual because of the COVID-19 pandemic. I remember something about a shortened regular season. Let me think, the LA Dodgers and the Texas Rangers were both strong teams that year.\n\nWait, actually, maybe the Dodgers were in the World Series. I think they faced the Tampa Bay Rays. But I'm not entirely sure if that's correct. I should double-check the details in my memory. The Rays were the underdogs, right? They had a lot of young players.\n\nI think the Dodgers won the World Series that year, but I'm not 100% certain. I'm pretty sure it was a sweep or a few games. Maybe they won in six games? I should confirm that. Yeah, I remember Max Muncy was a key player for the Dodgers.\n\nSo, putting it all together, I believe the Los Angeles Dodgers won the 2020 World Series against the Tampa Bay Rays.\n",
        "content": "\n\nThe Los Angeles Dodgers won the 2020 World Series. They defeated the Tampa Bay Rays in a six-game series.",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 21,
    "total_tokens": 303,
    "completion_tokens": 282,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null
}

[majestichou]

Notice whether there is <think>\\n in the end of the chat_template in your tokenizer_config.json.

8 days ago, there is an upate about the tokenizer_config.json of deepseek-r1 series models (https://huggingface.co/deepseek-ai/DeepSeek-R1):

Additionally, we have observed that the DeepSeek-R1 series models tend to bypass thinking pattern (i.e., outputting "\n\n") when responding to certain queries, which can adversely affect the model's performance. To ensure that the model engages in thorough reasoning, we recommend enforcing the model to initiate its response with "\n" at the beginning of every output.

If you use the latest version of tokenizer_config.json, you will not get the start think token <think> in the response. And 0.7.2 vllm's implementation about separation of reasoning content and normal content is:

vllm/entrypoints/openai/reasoning_parsers/deepseek_r1_reasoning_parser.py

def extract_reasoning_content(
        self, model_output: str, request: ChatCompletionRequest
) -> Tuple[Optional[str], Optional[str]]:

    # Check if the model output contains the <think> tokens.
    if (self.think_start_token not in model_output
            or self.think_end_token not in model_output):
        return None, model_output
    else:
        # Use a regex to find the reasoning content
        reasoning_content = self.reasoning_regex.findall(model_output)[0]

        # Remove the reasoning content from the model output
        # Although deepseek's <think> token is always at the
        # beginning of the line, we cannot guarantee that the
        # other models will follow this convention.
        # Therefore, we need to add :start_index.
        start_index = model_output.find(self.think_start_token)
        if start_index != -1:
            end_index = start_index + len(
                f"{self.think_start_token}{reasoning_content}{self.think_end_token}"
            )
            model_output = model_output[:start_index] + \
                            model_output[end_index:]

            if len(model_output) == 0:
                return reasoning_content, None

        return reasoning_content, model_output

Therefore, you can only get an empty reasoning content and response.choices[0].message.content is all content.

You may go back to the old version tokenizer_config.json, and you should get the right reasoning content and content. (delete the <think>\\n in the end of the chat_template in your tokenizer_config.json)

PS: The implementation of vllm seems good now:

vllm/vllm/entrypoints/openai/reasoning_parsers/deepseek_r1_reasoning_parser.py

Lines 132 to 138 in 238dfc8

else:
# Add a start token if it's missing to keep compatibility.
if self.think_start_token not in model_output:
model_output = f"{self.think_start_token}{model_output}"
# Use a regex to find the reasoning content
reasoning_content = self.reasoning_regex.findall(model_output)[0]

Thank you so much!

[hahmad2008]

@majestichou @Nietism I have the same issue,
model: "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
my chat template as following:

"chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='', is_first_sp=true) %}{%- for message in messages %}{%- if message['role'] == 'system' %}{%- if ns.is_first_sp %}{% set ns.system_prompt = ns.system_prompt + message['content'] %}{% set ns.is_first_sp = false %}{%- else %}{% set ns.system_prompt = ns.system_prompt + '\n\n' + message['content'] %}{%- endif %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\n' + '```json' + '\n' + tool['function']['arguments'] + '\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\n' + '```json' + '\n' + tool['function']['arguments'] + '\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{{'<｜Assistant｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}"

[Nietism]

@majestichou @Nietism I have the same issue, model: "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B my chat template as following:

"chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='', is_first_sp=true) %}{%- for message in messages %}{%- if message['role'] == 'system' %}{%- if ns.is_first_sp %}{% set ns.system_prompt = ns.system_prompt + message['content'] %}{% set ns.is_first_sp = false %}{%- else %}{% set ns.system_prompt = ns.system_prompt + '\n\n' + message['content'] %}{%- endif %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\n' + '```json' + '\n' + tool['function']['arguments'] + '\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\n' + '```json' + '\n' + tool['function']['arguments'] + '\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{{'<｜Assistant｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}"

what issues

[hahmad2008]

@Nietism the reasoning_context is null plus sometime the first tag in the content is missing. How did you solve it?

[Nietism]

@Nietism the reasoning_context is null plus sometime the first tag in the content is missing. How did you solve it?

I have no idea if the solution above cannot help you.

1. Debugging
2. Observing the original model output rather than relying on the template

may be helpful.

You can also notice other similar issues
#13125

[hahmad2008]

@Nietism can you share the chat template you used? or the change you did? can't get this revision
--tokenizer-revision 6453600843497f04e63f069049e92bf221d9e000

[majestichou]

"chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}"
@hahmad2008

[Nietism]

@hahmad2008 My vllm version is far behind 0.7.2, so I parse the model outputs manually. You can also fix the codes about reasoning parsers in vllm, it might be helpful.

Maybe modifying the implementation about reasoning content parsing of vllm 0.7.2 could be helpful to you:

This is the implementation of vllm 0.7.2:

# vllm/entrypoints/openai/reasoning_parsers/deepseek_r1_reasoning_parser.py
    def extract_reasoning_content(
            self, model_output: str, request: ChatCompletionRequest
    ) -> Tuple[Optional[str], Optional[str]]:

        # Check if the model output contains the <think> tokens.
        if (self.think_start_token not in model_output
                or self.think_end_token not in model_output):
            return None, model_output
...

self.think_start_token not in model_output leads to an empty reasoning content. You can just replace it with the latest implementation of extract_reasoning_content in the vllm repo. It is compatible with both styles:

• <think>...</think>...
• ...</think>...

vllm/vllm/entrypoints/openai/reasoning_parsers/deepseek_r1_reasoning_parser.py

Lines 123 to 147 in b3942e1

	def extract_reasoning_content(
	self, model_output: str, request: ChatCompletionRequest
	) -> Tuple[Optional[str], Optional[str]]:
	# DeepSeek R1 doesn't generate <think> now.
	# Thus we assume the reasoning content is always at the start.
	# Ref https://huggingface.co/deepseek-ai/DeepSeek-R1/commit/8a58a132790c9935686eb97f042afa8013451c9f
	if self.think_end_token not in model_output:
	return model_output, None
	else:
	# Add a start token if it's missing to keep compatibility.
	if self.think_start_token not in model_output:
	model_output = f"{self.think_start_token}{model_output}"
	# Use a regex to find the reasoning content
	reasoning_content = self.reasoning_regex.findall(model_output)[0]
	end_index = len(
	f"{self.think_start_token}{reasoning_content}{self.think_end_token}"
	)
	final_output = model_output[end_index:]
	if len(final_output) == 0:
	return reasoning_content, None
	return reasoning_content, final_output

If you want to make it suppport streaming also, you should fix the streaming parser. You can refer to #13025

Originally posted by @Nietism in #13125

[saminbatra29]

Hello, instead of using an older version of the chat template, is the issue fixed in vllm so we could use the latest stable version? I was initially using vllm 0.7.1 and noticed the same issue, i.e. reasoning content is in the actual content. I upgraded to vllm 0.7.2 and still have this issue.
I am using DeepSeek-R1-14B-Distill version of Qwen