[Feature]: Guided decoding after thinking is done #18255
Description
🚀 The feature, motivation and pitch
I want to activate Guided Decoding while performing thinking, which is an advantage of the CoT model.
I tried to use the guided decoding function to get a response in json format using Deepseek-R1.
However, when the Guided Decoding function is turned on, the token that breaks the json format is masked, so the token is not generated, and the thinking process is not performed. The quality of the answer decreases due to the inability to think about the strength of the CoT model.
Another problem is that when you want to receive a json response using the with_structure_output method in langchain, when you activate --enable-reasoning in vllm, the json response is filled the reasoning_content field rather than the content field after parsing, causing an exception.
Alternatives
To solve this problem, we request the function of deactivating guided decoding in the thinking process and activating guided decoding only when thinking is terminated and actual answers are generated.
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.