higher latency than TGI

[gravitywp]

Is it normal to have higher latency than TGI with a low concurrency, such as 1 or 4?

[WoosukKwon]

Hi @gravitywp, thanks for the question. I don't think it's normal. Could you let us know the model and GPU you are testing it on?

[gravitywp]

@WoosukKwon thanks for the reply. We use the NVIDIA A10 for testing on a fairly long prompt containing 456 tokens.

[hmellor]

Closing this issue as stale as there has been no discussion in the past 3 months.

If you are still experiencing the issue you describe, feel free to re-open this issue.