Perf issue with c-ares DNS resolver

[nicoche]

Hi 👋

Title: Perf issue with c-ares DNS resolver

Description:

The c-ares DNS resolver calls getifaddrs syscall. getifaddrs can be slow or CPU intensive (example #19717). If Envoy has many DNS clusters, the performance hit can be considerable.

Repro steps:

This issue happened for us with the following scenario:

• A server runs latest Envoy version
• The server has 700+ network namespaces
• XDS server tries to push between 500 and 2k DNS clusters onto Envoy

As a result, after connecting to the XDS server, the main thread is saturated initializing c-ares DNS resolvers. For us, it causes high CPU usage and disconnection-reconnection loops to the XDS server, probably because the main thread is busy listing interfaces instead of responding to keep-alives.

Root cause:

Because there are many network namespaces on our server, getifaddrs is CPU intensive.

It seems that c-ares DNS resolver lists the network interfaces of the machine to satisfy filter_unroutable_families https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/network/dns_resolver/cares/v3/cares_dns_resolver.proto#extensions-network-dns-resolver-cares-v3-caresdnsresolverconfig:

(bool) The resolver will query available network interfaces and determine if there are no available interfaces for a given IP family. It will then filter these addresses from the results it presents. e.g., if there are no available IPv4 network interfaces, the resolver will not provide IPv4 addresses.

Suggestion:

No matter the value of filter_unroutable_families, interfaces are listed anyway. I would suggest to:

• Either only list interfaces whenever filter_unroutable_families is true (early return here https://github.com/envoyproxy/envoy/blob/main/source/extensions/network/dns_resolver/cares/dns_impl.cc#L508)
• Or only list interfaces once and re-use the result across different DNS resolvers

Other ideas welcome. I'm happy to provide a PR too

[tyxia]

cced @yanavlasov and @mattklein123 based on cares CODEOWNER

Also cced @yanjunxiang-google worked on this area in the past

[yanjunxiang-google]

@nicoche Can these options you mentioned relieve the issue?

[nicoche]

@yanjunxiang-google I think so.

We built Envoy with a patch that returns {true, true} line 508. It runs since a few days in production and fixed our issue. It would be similar to making sure that getifaddrs is ran only when filter_unroutable_families=true.

I don't know about the other suggestion, but pretty sure it would work too.

If you want to reproduce the issue, I was able to do so locally:

---

Setup:

• 1000 net namespaces and a veth plugged on each one => running getifaddrs takes about 100ms, which is similar to what we witnessed
• 2000 envoy DNS clusters

I can share the scripts to setup that, if you want.

When I use the latest dev image, it takes about 50 seconds to initialize DNS clusters. One of the CPU of the machine is at 100% when I check htop:

$ sudo docker run --network=host -v $(pwd)/envoy.yaml:/envoy.yaml --rm envoyproxy/envoy:dev -c /envoy.yaml
(...)
[2024-06-23 23:32:49.603][1][info][admin] [source/server/admin/admin.cc:65] admin address: 127.0.0.1:9901
[2024-06-23 23:32:49.603][1][info][config] [source/server/configuration_impl.cc:168] loading tracing configuration
[2024-06-23 23:32:49.603][1][info][config] [source/server/configuration_impl.cc:124] loading 0 static secret(s)
[2024-06-23 23:32:49.603][1][info][config] [source/server/configuration_impl.cc:130] loading 2000 cluster(s)
[2024-06-23 23:33:36.851][1][info][config] [source/server/configuration_impl.cc:138] loading 1 listener(s)
[2024-06-23 23:33:36.854][1][info][config] [source/server/configuration_impl.cc:154] loading stats configuration
[2024-06-23 23:33:36.854][1][info][runtime] [source/common/runtime/runtime_impl.cc:625] RTDS has finished initialization
[2024-06-23 23:33:36.854][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:240] cm init: all clusters initialized

When I use the our patch, it loads in 2 seconds:

$ sudo docker run --network=host -v $(pwd)/envoy.yaml:/envoy.yaml --rm xxx/envoy_patched:v1 -c /envoy.yaml
[2024-06-23 23:43:37.880][1][info][admin] [source/server/admin/admin.cc:65] admin address: 127.0.0.1:9901
[2024-06-23 23:43:37.881][1][info][config] [source/server/configuration_impl.cc:168] loading tracing configuration
[2024-06-23 23:43:37.881][1][info][config] [source/server/configuration_impl.cc:124] loading 0 static secret(s)
[2024-06-23 23:43:37.881][1][info][config] [source/server/configuration_impl.cc:130] loading 2000 cluster(s)
[2024-06-23 23:43:39.253][1][info][config] [source/server/configuration_impl.cc:138] loading 1 listener(s)
[2024-06-23 23:43:39.255][1][info][config] [source/server/configuration_impl.cc:154] loading stats configuration
[2024-06-23 23:43:39.256][1][info][runtime] [source/common/runtime/runtime_impl.cc:625] RTDS has finished initialization
[2024-06-23 23:43:39.256][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:240] cm init: all clusters initialized

Setting filter_unroutable_families to true or false does not change anything.

[nicoche]

Hey 👋

Is there anything I can do to help move this forward? I would really love to see a fix in the next release. As mentioned, given a direction, I'm happy to provide a PR. I think that

Only list interfaces whenever filter_unroutable_families is true (early return here https://github.com/envoyproxy/envoy/blob/main/source/extensions/network/dns_resolver/cares/dns_impl.cc#L508)

is the simplest fix.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[nicoche]

Hey,

I'm commenting to prevent this issue from going stale. Is there anything else I can do to help?

[yanjunxiang-google]

HI, @nicoche
Could you please send out a PR for your solution on this?

Thanks for contributing!