v0.3.3 vllm.entrypoints.openai.api_server error

[Uhao-P]

1. GPU：4090 * 4
2. pip install vllm==0.3.3
3. python3 -m vllm.entrypoints.openai.api_server --model /Llama-2-7B-Chat-hf/ --engine-use-ray --host 0.0.0.0 --port 8080 --worker-use-ray --max-num-seqs 64 --tensor-parallel-size 4
4. When I send a request：
5. The following error occurs

[esmeetu]

@Uhao-P Have you tried other models?

[Uhao-P]

Tried, same error

[esmeetu]

@Uhao-P Could you try #3301?

[Uhao-P]

@esmeetu
I tried #3301 ，But the following error occurs

1. git clone https://github.com/esmeetu/vllm.git -b fix-tokenizer
2. cd vllm
3. pip install -e .
4. python3 -m vllm.entrypoints.openai.api_server --model /input2/ --engine-use-ray --host 0.0.0.0 --port 8080 --worker-use-ray --max-num-seqs 64 --tensor-parallel-size 4

[nikitaCodeSave]

Try changing port 8080 to a different port. This port may be reserved on some systems

INFO seems to have automatically assigned you a different port and this may be the problem.