Exceptions in application logs with larger or larger number of requests

[eero-t]

Setup

These errors originally happened with v0.7 ChatQnA Xeon installation [1], but e.g. updating to TEI services from 1.2-cpu version to latest 1.5-cpu, and and TGI service from 1.4 version to latest 2.2 did help, and they still happen, nearly 1 year later.

[1] https://github.com/opea-project/GenAIExamples/tree/v0.7/ChatQnA/kubernetes/manifests

Use-case

Constantly stress ChatQnA chaqna-xeon-backend-server-svc service endpoint by sending it large[2] number of queries in parallel.

Or do something else that causes responses to slow down: #1936

[2] compared to actual capacity of the service. E.g. 8 queries in parallel for service running on IceLake Xeon.

Actual outcome

Occasionally:

• Process sending it queries gets unexpected EOF error (i.e. service reply ended before specified Content-Length)
• Exception in logs from few of the service pods

For the exception details, see the attachments:

• chaqna-xeon-backend-server-svc
• reranking-svc
• llm-svc

Logs for pods of embedding-svc, redis-vector-db, retriever-svc, tei-embedding-svc, tei-reranking-svc and tgi-svc services did not show any exceptions or other errors.

unexpected EOF error can happen before chaqna-xeon-backend-server-svc replies with the first token, or after it has already provided 100-200 tokens for the reply.

Expected outcome

• Services handle common exceptions gracefully; shortly log the error and tell the caller that now is not a good time to do queries (return e.g. 503 Service Unavailable), instead of spamming log with exception and "crashing" the reply connection

• Have some kind of rate-limiting for chaqna-xeon-backend-server-svc, so that if it gets too many replies before earlier ones have been processed promptly enough, it will start replying 503 pre-emptively[2] (instead of making the situation worse by trying to process all requests although it has no capacity for it currently, and then failing in middle)

Note: Rate-limiting helps when service scale-up is slow (TGI pod may take minutes from startup until it's ready to respond), and once service has been scaled as up as it can go. But it needs to be done so that it does not prevent scale-up, or cause too much fluctuation for it.

(No comment on whether that should be implemented in chaqna-xeon-backend-server-svc itself, or in some load-balancer in front of it.)

[eero-t]

FYI: I have (k8s) readiness probes on TGI and TEI services, because otherwise things fail due to (k8s) service endpoint sending traffic to recently scaled up TGI instances, although TGI is not yet ready to accept requests.

However, sometimes TGI goes to non-ready state some time later (e.g. when there are multiple instances on same Xeon node, and all of them are stressed at the same time). There are no warnings in logs, no crashes. TGI just stops responding, k8s marks them as non-ready, and does not route traffic to them (for a while). This is likely cause at least for some of those exceptions in the services using TGI.

[YuningQiu]

Hello, thanks for bringing up this issue. The latest OPEA v0.8 has been released, and the latest version of ChatQnA has TEI services as 1.5-cpu and TGI service as v2.1.0.

Could you please give a try on v0.8 ChatQnA and let us know if this issue is still bothering you? Thanks a lot!

[xiguiw]

@eero-t
Current test 32 requests with no such issues.

Is this issue still valid at your side?
Could we close it?

[eero-t]

Current test 32 requests with no such issues.

I don't think that is stressing it enough, if you do not see even Timeout issues. I'm seeing these kind of issues every time I stress any of these services, either on CPU or Gaudi. Either test does not use enough parallel requests or token counts are too low (mixing in requests with invalid content may also help seeing exceptions).

Btw. What megaservice_request_pending values ChatQnA is reporting for your test-runs, from its /metrics endpoint? Is it about 32 or smaller?

Is this issue still valid at your side?

Yes, but it's not Xeon, aipc or even ChatQnA specific.

E.g. this ticket shows exceptions seen in DocSum app log: #1481

To re-iterate the point from the original bug description... The expectation is that well written code catches exceptions and provides user a readable description of the caught issue, instead of connection just breaking when backend service fails to return an answer.

[xiguiw]

@eero-t
I have only on machine, so I could not send large number of requests.

Did you deploy on K8S with multiple machines? What's the rough number and the platform (how many machines in the system) of the request you tried?

[eero-t]

@eero-t I have only on machine, so I could not send large number of requests.

Did you deploy on K8S with multiple machines? What's the rough number and the platform (how many machines in the system) of the request you tried?

@xiguiw Sorry for the late reply. Just tried with latest ChatQnA, and single node is enough.

I used changing number (64-256) of parallel (non-identical) requests being constantly sent. vLLM replica count was scaling up and down between 1-2 instances with HPA, but manual scaling should work too.

ChatQnA log with "OK" lines removed is attached, it shows multiple exceptions:
chatqna-exceptions.txt

[xiguiw]

@eero-t I have only on machine, so I could not send large number of requests.
Did you deploy on K8S with multiple machines? What's the rough number and the platform (how many machines in the system) of the request you tried?

@xiguiw Sorry for the late reply. Just tried with latest ChatQnA, and single node is enough.

I used changing number (64-256) of parallel (non-identical) requests being constantly sent. vLLM replica count was scaling up and down between 1-2 instances with HPA, but manual scaling should work too.

ChatQnA log with "OK" lines removed is attached, it shows multiple exceptions: chatqna-exceptions.txt

@eero-t

Thanks for your detail information.
So you send parallel (non-identical) requests, the number of requests varies from 64 to 256.
What's the frequency you send out the request?

I benchmarked vLLM on Xeon GNR, with BKC in August last year.
The batch is < 90 (vLLM only, no retriver/raranking etc.). Cannot find out the numbers. I'll list here if I find out it.

I don't think ChatQnA can process 90 request on single node Xeon.
It's a question how many requests can be processed by Xeon CPU. There should be a limited number.

The problem is what's the expected behavior shoud be if the requests beyond the hardware capabilities.

What's your expectation about this issue?