Qwen3 support

[BluePotato321]

Hi, thanks for sharing the great work. Does the code base currently support Qwen3 models? If not, any plans to include it?

[AnselCmy]

Yes, we plan to support Qwen3!

[PaulRoeseler]

Any idea when you would support Qwen3? And can you not just upgrade the vllm version to 0.8.5?

[AnselCmy]

For serving Qwen3, using vllm>=0.8.4 is enough. And for Qwen3, you should carefully parse the output from it, here is an parser example:

def parse_response(text):
        # extract reasoning_content (<think> tag content)
        think_pattern = r'<think>(.*?)</think>'
        think_match = re.search(think_pattern, text, re.DOTALL)
        reasoning_content = think_match.group(1).strip() if think_match else None
        
        # extract tool_calls (<tool_call> tag content)
        tool_pattern = r'<tool_call>(.*?)</tool_call>'
        tool_matches = re.finditer(tool_pattern, text, re.DOTALL)
        tool_calls = []
        for match in tool_matches:
            try:
                # try to parse JSON content
                tool_content = match.group(1).strip()
                tool_json = json.loads(tool_content)
                assert 'name' in tool_json and 'arguments' in tool_json, f"name or arguments not in tool_json: {tool_json}"
                assert isinstance(tool_json['arguments'], dict), f"arguments is not a dict: {tool_json['arguments']}"
                tool_calls.append(tool_json)
            except Exception as e:
                # if JSON parsing fails, add the original string
                # tool_calls.append(tool_content)
                tool_calls.append('error: parse tool call failed')
        
        # remove all tags, get the remaining text
        clean_text = text
        clean_text = re.sub(think_pattern, '', clean_text, flags=re.DOTALL)
        clean_text = re.sub(tool_pattern, '', clean_text, flags=re.DOTALL)
        content = clean_text.strip()
        
        return {
            'reasoning_content': reasoning_content,
            'tool_calls': tool_calls,
            'content': content
        }