Skip to content

feat(network): attribute proxy sockets to the pod owning their netns - #57899

Open
matthewleese wants to merge 3 commits into
mainfrom
mleese/cnm-ambient-netns-attribution
Open

matthewleese wants to merge 3 commits into
mainfrom
mleese/cnm-ambient-netns-attribution

Conversation

@matthewleese

@matthewleese matthewleese commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds network_config.enable_netns_container_attribution to system-probe, off by default. When it's on, a connection whose socket was created by an allowlisted proxy inside another pod's network namespace is attributed to the container that owns that namespace, not to the proxy.

  • pkg/network/tracer/netns_containers.go: a map of network namespace inode to container, fed by workloadmeta container events. Each container's namespace is read once from /proc/<pid>/ns/net when it starts, and removed when it's deleted, because inodes can be reused. The host's namespace is never mapped.
  • addProcessInfo: after the process cache sets the container, reattribute when the owner's image is on network_config.netns_container_attribution_owner_images (default ztunnel), the namespace holds exactly one container, and that container isn't the owner. Multi-container pods keep today's behavior. Reattributed connections drop the owner process's service, env, and version tags. This runs before aggregation, because the connection key includes the container.
  • The network module enables it after the tracer starts; Windows and macOS are no-ops.

Remote ends need no change: every remote-container resolver copies the peer socket's container, so once the local end is right the remote end follows.

Motivation

In Istio ambient mode, ztunnel calls setns into each enrolled pod's network namespace before bind and connect. Its sockets live in the app pod's namespace, but they're owned by ztunnel's process, so CNM attributes all of that traffic to ztunnel. A customer running ambient sees ztunnel as their largest client and can't see which services talk to each other.

RFC: [RFC] CNM Ambient ztunnel Mitigation for NYT

Describe how you validated your changes

  • Unit tests for the resolver: reattribution, own namespace, non-owners, multi-container pods, host and unknown namespaces, delete then inode reuse, stopped and unreadable containers, and concurrent updates under -race.
  • TestNetNSContainerAttribution (TracerSuite): the test process enters a new namespace with netns.WithNS, dials a loopback server there, and asserts the connection is attributed to that namespace's container with process tags cleared, while a control connection in its own namespace keeps the owner's container. Compiled locally; not yet run on a kernel.
  • Built for Linux (bpf) and macOS locally. Not yet validated in an ambient cluster.

Additional Notes

  • Draft: still needs the kernel test run in CI and an ambient lab run.
  • Known limitation: containerForPid can map the proxy's PID to one of the reattributed containers. The backend only falls back to it when a connection has no container ID, which for a proxy socket happens only on a process-cache miss.
  • A follow-up will add a payload field marking reattributed connections, so the backend can label and hide the hops inside a pod.

Note: This PR was created by Claude.

🤖 Generated with Claude Code

In Istio ambient mode, ztunnel calls setns into each enrolled pod's network
namespace before bind and connect, so its sockets live in the app pod's
namespace but are owned by ztunnel's process. The tracer attributes a
connection's container by the owning process, so every such connection is
reported as ztunnel, and remote resolution copies that container to the peer.

Add network_config.enable_netns_container_attribution (off by default). When
enabled, the tracer keeps a workloadmeta-fed map of network namespace to
container and, in addProcessInfo, reattributes a socket to the namespace's
container when the owner's image is on
network_config.netns_container_attribution_owner_images (default: ztunnel),
the namespace holds exactly one container, and that container isn't the
owner. The host namespace and multi-container pods keep today's behavior.
Reattributed connections drop the owner process's service/env/version tags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review by Codex (OpenAI) - workflow run

The patch is incorrect: the new resolver is missing from the Bazel target, breaking Linux/BPF builds. Review was limited to static inspection; tests were not run.

Comment thread pkg/network/tracer/tracer.go
…l targets

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@matthewleese matthewleese added qa/done QA done before merge and regressions are covered by tests qa/rc-required Only for a PR that requires validation on the Release Candidate and removed qa/done QA done before merge and regressions are covered by tests labels Oct 9, 2026
@matthewleese
matthewleese marked this pull request as ready for review October 9, 2026 09:17
@matthewleese
matthewleese requested review from a team as code owners October 9, 2026 09:17
@matthewleese
matthewleese requested review from Janaeq and arbll and removed request for a team October 9, 2026 09:17

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review by Codex (OpenAI) - workflow run

The patch is incorrect: payload enrichment can restore the proxy’s tags, and rewriting the source container also corrupts the PID-to-container fallback mapping. The kernel test does not cover either downstream behavior.

Comment thread pkg/network/tracer/tracer.go
Comment thread pkg/network/tracer/tracer.go
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 81.40%
• Overall Coverage: 58.59% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 41549ad | Docs | Give us feedback!

A socket attributed to its network namespace's container is still owned by
the proxy process, so anything keyed by PID describes the proxy, not the pod.

- Record the owner's container in ConnectionStats.NetNSOriginalContainerID.
- Direct sender: skip the PID's service context and process tagger tags for
  reattributed connections, and build containerForPid from the owner's
  container.
- process-agent: LocalResolver.Resolve reports connections whose laddr
  container differs from their PID's container; batchConnections skips the
  same PID enrichment for them and keeps containerForPid on the PID's own
  container.

process_name is kept: it names the owning process, not a service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added long review PR is complex, plan time to review it and removed medium review PR review might take time labels Oct 9, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Files inventory check summary

File checks results against ancestor 1e25e036:

Results for datadog-agent_7.86.0~devel.git.306.41549ad.pipeline.143647082-1_amd64.deb:

No change detected

Results for datadog-iot-agent_7.86.0~devel.git.306.41549ad.pipeline.143647082-1_amd64.deb:

No change detected

@dd-octo-sts

dd-octo-sts Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor 1e25e03
📊 Static Quality Gates Dashboard
🔗 SQG Job

Successful checks

Info

Quality gate Change Size (prev → curr → max)
✅ agent_deb_amd64 +12.33 KiB (0.00% increase, -0.23% of buffer) 755.264 → 755.276 → 760.520
✅ agent_deb_amd64_fips +28.0 KiB (0.00% increase, -0.52% of buffer) 688.274 → 688.301 → 693.570
✅ agent_msi +8.0 KiB (0.00% increase, -2.17% of buffer) 653.830 → 653.838 → 654.190
✅ agent_rpm_amd64 +12.33 KiB (0.00% increase, -0.23% of buffer) 755.247 → 755.259 → 760.500
✅ agent_rpm_amd64_fips +28.0 KiB (0.00% increase, -0.52% of buffer) 688.258 → 688.285 → 693.550
✅ agent_rpm_arm64 +19.99 KiB (0.00% increase, -0.37% of buffer) 725.785 → 725.804 → 731.100
✅ agent_rpm_arm64_fips +20.0 KiB (0.00% increase, -0.36% of buffer) 664.367 → 664.387 → 669.740
✅ agent_suse_amd64 +12.33 KiB (0.00% increase, -0.23% of buffer) 755.247 → 755.259 → 760.500
✅ agent_suse_amd64_fips +28.0 KiB (0.00% increase, -0.52% of buffer) 688.258 → 688.285 → 693.550
✅ agent_suse_arm64 +19.99 KiB (0.00% increase, -0.37% of buffer) 725.785 → 725.804 → 731.100
✅ agent_suse_arm64_fips +20.0 KiB (0.00% increase, -0.36% of buffer) 664.367 → 664.387 → 669.740
✅ docker_agent_amd64 +12.34 KiB (0.00% increase, -0.23% of buffer) 811.427 → 811.439 → 816.720
✅ docker_agent_arm64 +19.99 KiB (0.00% increase, -0.36% of buffer) 806.591 → 806.610 → 811.970
✅ docker_agent_jmx_amd64 +12.33 KiB (0.00% increase, -0.23% of buffer) 1002.296 → 1002.308 → 1007.590
✅ docker_agent_jmx_arm64 +19.99 KiB (0.00% increase, -0.36% of buffer) 986.240 → 986.260 → 991.620
✅ dogstatsd_deb_arm64 +4.0 KiB (0.01% increase, -0.42% of buffer) 28.278 → 28.282 → 29.200
✅ iot_agent_deb_amd64 +4.0 KiB (0.01% increase, -2.18% of buffer) 47.371 → 47.375 → 47.550
✅ iot_agent_deb_arm64 +4.0 KiB (0.01% increase, -1.02% of buffer) 43.837 → 43.841 → 44.220
✅ iot_agent_rpm_amd64 +4.0 KiB (0.01% increase, -2.19% of buffer) 47.371 → 47.375 → 47.550
✅ iot_agent_suse_amd64 +4.0 KiB (0.01% increase, -2.17% of buffer) 47.370 → 47.374 → 47.550
13 successful checks with minimal change (< 2 KiB)
Quality gate Current Size
✅ agent_heroku_amd64 319.144 MiB
✅ docker_cws_instrumentation_amd64 7.443 MiB
✅ docker_cws_instrumentation_arm64 6.877 MiB
✅ docker_dogstatsd_amd64 39.551 MiB
✅ docker_dogstatsd_arm64 37.629 MiB
✅ docker_host_profiler_amd64 308.721 MiB
✅ docker_host_profiler_arm64 319.864 MiB
✅ dogstatsd_deb_amd64 30.293 MiB
✅ dogstatsd_rpm_amd64 30.293 MiB
✅ dogstatsd_suse_amd64 30.293 MiB
✅ iot_agent_deb_armhf 44.652 MiB
✅ docker_cluster_agent_amd64 211.196 MiB
✅ docker_cluster_agent_arm64 224.224 MiB

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Job ID: fd2afac7-f7ec-4c4b-a753-52cb135730be

Baseline: 1e25e03
Comparison: 41549ad
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
➖ dsd_uds_10mb_3k_timestamped_contexts_cpu % cpu utilization +0.76 [+0.49, +1.04] 1 Logs
➖ quality_gate_metrics_logs memory utilization +0.66 [+0.43, +0.89] 1 Logs bounds checks dashboard
➖ quality_gate_private_action_runner memory utilization +0.21 [+0.08, +0.33] 1 Logs bounds checks dashboard
➖ quality_gate_security_idle memory utilization +0.14 [+0.10, +0.18] 1 Logs bounds checks dashboard
➖ quality_gate_security_mean_fs_load memory utilization +0.04 [-0.00, +0.08] 1 Logs bounds checks dashboard
➖ quality_gate_security_no_fs_load memory utilization +0.03 [-0.05, +0.10] 1 Logs bounds checks dashboard
➖ quality_gate_idle_all_features memory utilization -0.01 [-0.09, +0.07] 1 Logs bounds checks dashboard
➖ quality_gate_idle memory utilization -0.06 [-0.10, -0.02] 1 Logs bounds checks dashboard
➖ dsd_uds_10mb_3k_timestamped_contexts_memory memory utilization -0.13 [-0.34, +0.09] 1 Logs
➖ python_openmetrics % cpu utilization -0.21 [-0.89, +0.47] 1 Logs bounds checks dashboard
➖ quality_gate_logs % cpu utilization -1.22 [-2.09, -0.35] 1 Logs bounds checks dashboard
➖ dsd_uds_client_drop_detector_cpu % cpu utilization -2.33 [-2.83, -1.83] 1 Logs

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed observed_value links
✅ python_openmetrics checks_execution_time 10/10 90.02 ≤ 100 bounds checks dashboard
✅ python_openmetrics cpu_usage 10/10 1330.13 ≤ 1500 bounds checks dashboard
✅ python_openmetrics memory_usage 10/10 4.33GiB ≤ 4.75GiB bounds checks dashboard
✅ quality_gate_idle intake_connections 10/10 4 ≤ 5 bounds checks dashboard
✅ quality_gate_idle memory_usage 10/10 177.70MiB ≤ 181MiB bounds checks dashboard
✅ quality_gate_idle total_bytes_received 10/10 762.15KiB ≤ 819.20KiB bounds checks dashboard
✅ quality_gate_idle_all_features intake_connections 10/10 2 ≤ 5 bounds checks dashboard
✅ quality_gate_idle_all_features memory_usage 10/10 475.04MiB ≤ 542MiB bounds checks dashboard
✅ quality_gate_idle_all_features total_bytes_received 10/10 1.14MiB ≤ 1.25MiB bounds checks dashboard
✅ quality_gate_logs intake_connections 10/10 17 ≤ 40 bounds checks dashboard
✅ quality_gate_logs memory_usage 10/10 211.13MiB ≤ 228MiB bounds checks dashboard
✅ quality_gate_logs missed_bytes 10/10 0B = 0B bounds checks dashboard
✅ quality_gate_logs total_bytes_received 10/10 264.12MiB ≤ 292MiB bounds checks dashboard
✅ quality_gate_metrics_logs cpu_usage 10/10 422.77 ≤ 2000 bounds checks dashboard
✅ quality_gate_metrics_logs intake_connections 10/10 19 ≤ 40 bounds checks dashboard
✅ quality_gate_metrics_logs memory_usage 10/10 435.46MiB ≤ 455MiB bounds checks dashboard
✅ quality_gate_metrics_logs missed_bytes 10/10 0B = 0B bounds checks dashboard
✅ quality_gate_metrics_logs total_bytes_received 10/10 0.95GiB ≤ 1.04GiB bounds checks dashboard
✅ quality_gate_private_action_runner memory_usage 10/10 74.15MiB ≤ 77MiB bounds checks dashboard
✅ quality_gate_security_idle cpu_usage 10/10 35.25 ≤ 100 bounds checks dashboard
✅ quality_gate_security_idle memory_usage 10/10 326.74MiB ≤ 357MiB bounds checks dashboard
✅ quality_gate_security_mean_fs_load cpu_usage 10/10 65.63 ≤ 200 bounds checks dashboard
✅ quality_gate_security_mean_fs_load memory_usage 10/10 305.43MiB ≤ 337MiB bounds checks dashboard
✅ quality_gate_security_no_fs_load cpu_usage 10/10 29.56 ≤ 100 bounds checks dashboard
✅ quality_gate_security_no_fs_load memory_usage 10/10 333.39MiB ≤ 348MiB bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

  • quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.

}

id := intern.GetByString(container.ID)
r.removeLocked(id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would result in us removing the container for a healthy->unhealthy transition, even if the container is still running? Is that intentional?

}

func (r *netnsContainerResolver) start(ctx context.Context, wmeta workloadmeta.Component) {
filter := workloadmeta.NewFilterBuilder().

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we change this filter to use a single source? To avoid races between different sources emitting different events for the same container?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/system-probe internal Identify a non-fork PR long review PR is complex, plan time to review it qa/rc-required Only for a PR that requires validation on the Release Candidate team/agent-security team/cloud-network-monitoring team/ebpf-platform team/fleet-automation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants