Why vllm does not support Chinese input

[929359291]

There is a decode error in Chinese input，is token.decode

[shifan3]

If you use chinese-alpaca/llama, remember their tokenizers are different with the original ones. However in vllm/engine/tokenizer_utils.py it force to use the original llama tokenizer hf-internal-testing/llama-tokenizer. this produce the error. They should allow you to pass a use_fast=False to avoid this behavior, but currently it's not possible.
Before they fix this issue, as a temporary workaround, you can simply replace the tokenizer by
llm.llm_engine.tokenizer = AutoTokenizer.from_pretrained('YOUR CHINESE ALPACA/LLAMA TOKENIZER', use_fast=False)

[929359291]

If you use chinese-alpaca/llama, remember their tokenizers are different with the original ones. However in vllm/engine/tokenizer_utils.py it force to use the original llama tokenizer hf-internal-testing/llama-tokenizer. this produce the error. They should allow you to pass a use_fast=False to avoid this behavior, but currently it's not possible. Before they fix this issue, as a temporary workaround, you can simply replace the tokenizer by llm.llm_engine.tokenizer = AutoTokenizer.from_pretrained('YOUR CHINESE ALPACA/LLAMA TOKENIZER', use_fast=False)

thinks, i try again

[luoyangen]

If you use chinese-alpaca/llama, remember their tokenizers are different with the original ones. However in vllm/engine/tokenizer_utils.py it force to use the original llama tokenizer hf-internal-testing/llama-tokenizer. this produce the error. They should allow you to pass a use_fast=False to avoid this behavior, but currently it's not possible. Before they fix this issue, as a temporary workaround, you can simply replace the tokenizer by llm.llm_engine.tokenizer = AutoTokenizer.from_pretrained('YOUR CHINESE ALPACA/LLAMA TOKENIZER', use_fast=False)

thinks, i try again

Hi, Did you successed by passing use_fast=False?
I tried but get "RuntimeError: CUDA error: device-side assert triggered".

[929359291]

If you use chinese-alpaca/llama, remember their tokenizers are different with the original ones. However in vllm/engine/tokenizer_utils.py it force to use the original llama tokenizer hf-internal-testing/llama-tokenizer. this produce the error. They should allow you to pass a use_fast=False to avoid this behavior, but currently it's not possible. Before they fix this issue, as a temporary workaround, you can simply replace the tokenizer by llm.llm_engine.tokenizer = AutoTokenizer.from_pretrained('YOUR CHINESE ALPACA/LLAMA TOKENIZER', use_fast=False)

thinks, i try again

Hi, Did you successed by passing use_fast=False? I tried but get "RuntimeError: CUDA error: device-side assert triggered".

hi boy, i not try, wait vLLM Development Roadmap #244