DNS resolution failure issue with kong version 3.5.0

[mayank-allen]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.5.0

Current Behavior

We are intermittently getting DNS resolution failure. The Client gets this error-
failed the initial dns/balancer resolve for 'classroom-service.classroom.svc.cluster.local' with: failed to create a resolver: failed to set peer name: resource temporarily unavailable
In kong logs we get this kind of errors-
2024/09/26 07:03:28 [error] 1302#0: *1732918 [lua] init.lua:371: execute(): DNS resolution failed: failed to create a resolver: failed to set peer name: resource temporarily unavailable. Tried: ["(short)node-bff.node-bff.svc.cluster.local:(na) - cache-miss","node-bff.node-bff.svc.cluster.local.kong.svc.cluster.local:1 - cache-miss"], client: 10.1.6.200, server: kong, request: "POST /node-bff/api/v1/sr-management/requests/reissue-cards HTTP/1.1", host: "bff-dev.allen-demo.in", referrer: "http://localhost:3000/", request_id: "d4c524226b983517852ed6539f1c7311"

Expected Behavior

We expected that everything will work as in lower version 3.4.0

Steps To Reproduce

Just use version 3.5.0 and may be above (I only tested with 3.5.0) and deploy on your testing environment and let it remain at this version for sometime. In some hours you will start experiencing this issue with DNS not abled to get resolved, requests failing

Anything else?

No response

[StarlightIbuki]

@chobits Could you take a look?

[chobits]

It looks that your operating system resources have exausted then syscall socket's setpeername failed. like openresty/lua-resty-dns#36

Error message reported by this line: https://github.com/openresty/lua-resty-dns/blob/master/lib/resty/dns/resolver.lua#L162

It might be resource limitations: The system may have reached its limit for open sockets or other resources.

Could you check the fds number in your OS/container and the limits like ulimit -a, note we need to know the kong process's limit info.