Queue size increases indefinitely

[QLutz]

System Info

OS version: Linux
Model being used (curl 127.0.0.1:8080/info | jq): TheBloke/Nous-Hermes-2-Mixtral-8x7B-DPO-AWQ
Hardware used (GPUs, how many, on which cloud) (nvidia-smi): 1xL40S
The current version being used: 2.0.4

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Launch TGI using max_total_tokens=max_batch_prefill_tokens=16384; max_input_length=16383; quantize=awq.
After making a few hundred requests, the pod returns empty packets and only a few seconds after the request a
has been made.
Monitoring reveals that tgi_queue_size increases steadily but does not ever go down.

Expected behavior

No stutters.

[Hugoch]

Hey @QLutz , I suspect it may be related to #2099. Can you try to run TGI with --cuda-graphs 0 see if you still see the hang?

[HoKim98]

I had the same problem, and was able to solve it by trying --cuda-graphs 0 method. This obviously caused major performance problems, but it was at least a better option than being broken.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.