baichuan-7b return value of apiserver is garbled

[abc20220327]

I try to use vllm in combination with baichuan-7b, it can run normally, but the return value of apiserver is garbled, I don't know what is possible, can you give me some advice and help

[WoosukKwon]

Hi @abc20220327, thanks for reporting the issue. How did you try the Baichuan model? Currently it is not supported.

[binarrii]

Hi @abc20220327, thanks for reporting the issue. How did you try the Baichuan model? Currently it is not supported.

You can use Vicuna to test Chinese output, then you will see garbled characters!

[binarrii]

Hi @abc20220327, thanks for reporting the issue. How did you try the Baichuan model? Currently it is not supported.

Here is the situation I encountered.

I start the service in the following way:

#!/bin/bash

export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512

nohup python -m vllm.entrypoints.openai.api_server --port 8000 \
             --model "/data/models/vicuna-13b-v1.3" \
             --served-model-name "vicuna-13b-v1.3" \
             --tokenizer "hf-internal-testing/llama-tokenizer" \
             --gpu-memory-utilization 0.35 \
             &> ~/llmapi.out &

Then I request the API and output as follows:

[zhuohan123]

Can you try the latest main branch? Thanks!

[zhuohan123]

I believe this can be fixed by #589

[hmellor]

Closing as this should now be fixed.