Repository navigation
feat(network): attribute proxy sockets to the pod owning their netns - #57899
matthewleese wants to merge 3 commits into
Conversation
In Istio ambient mode, ztunnel calls setns into each enrolled pod's network namespace before bind and connect, so its sockets live in the app pod's namespace but are owned by ztunnel's process. The tracer attributes a connection's container by the owning process, so every such connection is reported as ztunnel, and remote resolution copies that container to the peer. Add network_config.enable_netns_container_attribution (off by default). When enabled, the tracer keeps a workloadmeta-fed map of network namespace to container and, in addProcessInfo, reattributes a socket to the namespace's container when the owner's image is on network_config.netns_container_attribution_owner_images (default: ztunnel), the namespace holds exactly one container, and that container isn't the owner. The host namespace and multi-container pods keep today's behavior. Reattributed connections drop the owner process's service/env/version tags. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
AI review by Codex (OpenAI) - workflow run
The patch is incorrect: the new resolver is missing from the Bazel target, breaking Linux/BPF builds. Review was limited to static inspection; tests were not run.
…l targets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
AI review by Codex (OpenAI) - workflow run
The patch is incorrect: payload enrichment can restore the proxy’s tags, and rewriting the source container also corrupts the PID-to-container fallback mapping. The kernel test does not cover either downstream behavior.
A socket attributed to its network namespace's container is still owned by the proxy process, so anything keyed by PID describes the proxy, not the pod. - Record the owner's container in ConnectionStats.NetNSOriginalContainerID. - Direct sender: skip the PID's service context and process tagger tags for reattributed connections, and build containerForPid from the owner's container. - process-agent: LocalResolver.Resolve reports connections whose laddr container differs from their PID's container; batchConnections skips the same PID enrichment for them and keeps containerForPid on the PID's own container. process_name is kept: it names the owning process, not a service. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Files inventory check summaryFile checks results against ancestor 1e25e036: Results for datadog-agent_7.86.0~devel.git.306.41549ad.pipeline.143647082-1_amd64.deb:No change detected Results for datadog-iot-agent_7.86.0~devel.git.306.41549ad.pipeline.143647082-1_amd64.deb:No change detected |
Static quality checks✅ Please find below the results from static quality gates Successful checksInfo
13 successful checks with minimal change (< 2 KiB)
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 1e25e03 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | dsd_uds_10mb_3k_timestamped_contexts_cpu | % cpu utilization | +0.76 | [+0.49, +1.04] | 1 | Logs |
| ➖ | quality_gate_metrics_logs | memory utilization | +0.66 | [+0.43, +0.89] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_private_action_runner | memory utilization | +0.21 | [+0.08, +0.33] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_idle | memory utilization | +0.14 | [+0.10, +0.18] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | +0.04 | [-0.00, +0.08] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_no_fs_load | memory utilization | +0.03 | [-0.05, +0.10] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.01 | [-0.09, +0.07] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | -0.06 | [-0.10, -0.02] | 1 | Logs bounds checks dashboard |
| ➖ | dsd_uds_10mb_3k_timestamped_contexts_memory | memory utilization | -0.13 | [-0.34, +0.09] | 1 | Logs |
| ➖ | python_openmetrics | % cpu utilization | -0.21 | [-0.89, +0.47] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_logs | % cpu utilization | -1.22 | [-2.09, -0.35] | 1 | Logs bounds checks dashboard |
| ➖ | dsd_uds_client_drop_detector_cpu | % cpu utilization | -2.33 | [-2.83, -1.83] | 1 | Logs |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | python_openmetrics | checks_execution_time | 10/10 | 90.02 ≤ 100 | bounds checks dashboard |
| ✅ | python_openmetrics | cpu_usage | 10/10 | 1330.13 ≤ 1500 | bounds checks dashboard |
| ✅ | python_openmetrics | memory_usage | 10/10 | 4.33GiB ≤ 4.75GiB | bounds checks dashboard |
| ✅ | quality_gate_idle | intake_connections | 10/10 | 4 ≤ 5 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 177.70MiB ≤ 181MiB | bounds checks dashboard |
| ✅ | quality_gate_idle | total_bytes_received | 10/10 | 762.15KiB ≤ 819.20KiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 2 ≤ 5 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 475.04MiB ≤ 542MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | total_bytes_received | 10/10 | 1.14MiB ≤ 1.25MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 17 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 211.13MiB ≤ 228MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_logs | total_bytes_received | 10/10 | 264.12MiB ≤ 292MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 422.77 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 19 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 435.46MiB ≤ 455MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | total_bytes_received | 10/10 | 0.95GiB ≤ 1.04GiB | bounds checks dashboard |
| ✅ | quality_gate_private_action_runner | memory_usage | 10/10 | 74.15MiB ≤ 77MiB | bounds checks dashboard |
| ✅ | quality_gate_security_idle | cpu_usage | 10/10 | 35.25 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_idle | memory_usage | 10/10 | 326.74MiB ≤ 357MiB | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | cpu_usage | 10/10 | 65.63 ≤ 200 | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | memory_usage | 10/10 | 305.43MiB ≤ 337MiB | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | cpu_usage | 10/10 | 29.56 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | memory_usage | 10/10 | 333.39MiB ≤ 348MiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
| } | ||
|
|
||
| id := intern.GetByString(container.ID) | ||
| r.removeLocked(id) |
There was a problem hiding this comment.
I think this would result in us removing the container for a healthy->unhealthy transition, even if the container is still running? Is that intentional?
| } | ||
|
|
||
| func (r *netnsContainerResolver) start(ctx context.Context, wmeta workloadmeta.Component) { | ||
| filter := workloadmeta.NewFilterBuilder(). |
There was a problem hiding this comment.
Should we change this filter to use a single source? To avoid races between different sources emitting different events for the same container?
What does this PR do?
Adds
network_config.enable_netns_container_attributionto system-probe, off by default. When it's on, a connection whose socket was created by an allowlisted proxy inside another pod's network namespace is attributed to the container that owns that namespace, not to the proxy.pkg/network/tracer/netns_containers.go: a map of network namespace inode to container, fed by workloadmeta container events. Each container's namespace is read once from/proc/<pid>/ns/netwhen it starts, and removed when it's deleted, because inodes can be reused. The host's namespace is never mapped.addProcessInfo: after the process cache sets the container, reattribute when the owner's image is onnetwork_config.netns_container_attribution_owner_images(defaultztunnel), the namespace holds exactly one container, and that container isn't the owner. Multi-container pods keep today's behavior. Reattributed connections drop the owner process's service, env, and version tags. This runs before aggregation, because the connection key includes the container.Remote ends need no change: every remote-container resolver copies the peer socket's container, so once the local end is right the remote end follows.
Motivation
In Istio ambient mode,
ztunnelcallssetnsinto each enrolled pod's network namespace beforebindandconnect. Its sockets live in the app pod's namespace, but they're owned byztunnel's process, so CNM attributes all of that traffic toztunnel. A customer running ambient seesztunnelas their largest client and can't see which services talk to each other.RFC: [RFC] CNM Ambient ztunnel Mitigation for NYT
Describe how you validated your changes
-race.TestNetNSContainerAttribution(TracerSuite): the test process enters a new namespace withnetns.WithNS, dials a loopback server there, and asserts the connection is attributed to that namespace's container with process tags cleared, while a control connection in its own namespace keeps the owner's container. Compiled locally; not yet run on a kernel.bpf) and macOS locally. Not yet validated in an ambient cluster.Additional Notes
containerForPidcan map the proxy's PID to one of the reattributed containers. The backend only falls back to it when a connection has no container ID, which for a proxy socket happens only on a process-cache miss.🤖 Generated with Claude Code