"max_throughput" mode causes connection timeouts

[tianmu-li]

In max_throughput mode with a lot of connections, a lot of the requests hit "Connection timed out" errors. Attaching error log and .yaml used for testing.

offline_llama3_1b_cnn_perf_unlimited.yaml
debug_client_perf.log

[viraatc]

we attempted to patch this in #168 (merged yesterday),
are you running TOT main?

[tianmu-li]

This is using commit 08476d, which should include #168

[viraatc]

what is the expected throughput? in terms of QPS?

i tested #168 to be fine for >100 qps, lower values were still dropping connections (tested against variable_throughput_server.py)

for lower QPS, lets try increase tcp-keepalive-cnt to really high value like 25:

endpoints/src/inference_endpoint/endpoint_client/http.py

Line 59 in d9a3cc4

TCP_KEEPCNT: int = 5 # 5 failed probes = dead

(socket params might need further tuning)

[tianmu-li]

Expected throughput is ~60. TCP_KEEPCNT=25 makes QPS=60 pass but had to increase TCP_KEEPCNT to 50 to make QPS=15 pass.

[viraatc]

lets see if there is any downside of completely disabling TCP-Keepalive in http-client