Stuck while inferring with WizardCoder model

[Valdanitooooo]

My code:

	llm_model = LLM(model=model_path)
	...
	...
	sampling_params = SamplingParams(
        temperature=temperature, top_p=top_p, top_k=top_k, max_tokens=max_new_tokens
        ,frequency_penalty=repetition_penalty,ignore_eos=True
    )
    outputs = llm_model.generate([prompt], sampling_params)

My parameters:

{
    "Temperature":0.01,
    "Top-k":70,
    "Top-p":0.85,
    "Max new tokens":4096,
    "Repetition Penalty":1.2
}

Does anyone know where the problem is?

[MasKong]

Same problem. Infer with HF doesn't get stuck but much slower

[Xingxiangrui]

I did not meet this problem on NVIDIA A100 GPU. vLLM is about 25% faster than HF. Please give more information. What is the kind of GPU that you use ?

[MasKong]

I did not meet this problem on NVIDIA A100 GPU. vLLM is about 25% faster than HF. Please give more information. What is the kind of GPU that you use ?

I am using 4 T4. It works well when it comes to one or two cards. But it got stuck in the server with 4 T4.

[Valdanitooooo]

I did not meet this problem on NVIDIA A100 GPU. vLLM is about 25% faster than HF. Please give more information. What is the kind of GPU that you use ?

@Xingxiangrui I am using A100 too, What parameters are you using?

[felipec23]

Same problem here. I'm running inference using an OpenLlama model and calling llm.generate. I have 800 prompts I want to pass to the model, and it always gets stuck at prompt #195.

I created chunks—each of 150 prompts. The first 150 ran successfully, but the second chunk stopped at 45 (195, again).

Any idea of what might be happening? I'm using a T4 in Google Colab.

Update: problem disappears with V100. But it would be nice to know it isn't working in a T4.

[BinhMinhs10]

Same problem. Infer get stuck when run , I'm using a100 and WizardCoder

[hmellor]

Closing this issue as stale as there has been no discussion in the past 3 months.

If you are still experiencing the issue you describe, feel free to re-open this issue.