How to proper tune the reqwest for high concurrent requests

[gftea]

I use 2 client instance and each client issue 1500 requests concurrently.
I also have increase system max sockets and max files per process.

I have been facing an issue below that after running several loops, the reqwest report below error.
T

Request Error: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(27000), path: "/rpc/v1/transactions/e2255db23c2cfc9dbb300e70fbfdbf81b5f751395a9ba049bea813802e90ba99", query: None, fragment: None }, source: TimedOut }

[seanmonstar]

That answer to the title question is variable, and depends on your workflow and servers. If you provide more information about your configuration, requests and responses, and the servers, we might be able to help better.

[gftea]

The server is 8 core, with http server running 16 workers
The client side I have use pool_idle_time = 30s and request_timeout = 5s
between each loop there is 15s cool down.
the TimedOut from above seems from socket io? I have not looked into the codes yet, but would like first understand this error message meaning

[gftea]

but after I exit the client program and restart again, it would work again for a few loops. I have a feeling that this is something related with reqwest internal resource handling