fix(emulator): join Atmos's container to the shared network when reuse fails

docs/fixes/2026-08-19-emulator-endpoint-job-container-network-join.md

@@ -0,0 +1,92 @@
+# Fix: Emulator endpoint unreachable from a socket-mounted job container
+
+**Date:** 2026-08-19
+
+## Summary
+
+`atmos terraform test --ci`, run inside a CI job container that talks to Docker only through a
+mounted host socket, started the AWS emulator successfully but handed Terraform/OpenTofu an
+endpoint (`http://172.17.0.1:<port>`) that was not reachable from that same job container. The
+AWS provider's `GetCallerIdentity` call failed with `connect: connection refused` after nine
+retries. Atmos now actively joins its own container to the emulator's dedicated network instead
+of guessing a reachable IP, so the emulator is addressed by DNS alias, not by IP.
+
+## Context
+
+`pkg/emulator/manager.go`'s `endpoint()` prefers a DNS-alias endpoint when Atmos's own container
+can reuse its *existing* network (`container.CurrentContainerNetwork`). A job container started
+with a plain `docker run` (no `--network`) sits on Docker's default bridge network, which is
+correctly excluded from reuse because it doesn't support embedded DNS/aliases. When reuse failed,
+`pkg/emulator/endpoint_host.go`'s `reachableHostForPublishedPorts()` fell back to parsing the
+container's own default gateway out of `/proc/net/route` -- Docker's classic bridge-gateway IP.
+That heuristic assumes the caller and the emulator container share one Linux host's `docker0`
+bridge. Under Docker Desktop for macOS the daemon runs inside a VM; the bridge-gateway address
+there is not where the host's published-port forwarding actually listens for sibling containers,
+so nothing answered -- "connection refused," exactly as reported.
+
+This closes a gap left open by PR #2942 ("Shared per-stack networking for containers, emulators &
+run steps"), which introduced the current-container-network-reuse mechanism but only reused an
+*existing* usable network -- it never made an unusable one (the default bridge) usable.
+
+Reported and reproduced via a disposable copy of an internal application repository's
+`bugs.md` item 8 ("Emulator identity loopback endpoints fail inside GitHub job containers"),
+using a socket-mounted `docker:cli` job container with no `--network` flag -- the same shape as a
+real CI job container that doesn't explicitly join a custom Docker network.
+
+## Changes
+
+- `pkg/container/network.go`: new `NetworkConnector` interface (`ConnectNetwork`) and
+  `networkConnectResult` idempotency helper, alongside the existing `NetworkEnsurer`.
+- `pkg/container/docker.go`, `pkg/container/podman.go`: `ConnectNetwork` implementations via
+  `docker network connect` / `podman network connect`, using a new shared
+  `buildNetworkConnectArgs` helper in `pkg/container/common.go`.
+- `pkg/container/stack_network.go`: `AttachSharedNetwork` now calls
+  `joinCurrentContainerToNetwork` when reuse fails -- best-effort connecting Atmos's own
+  container to the dedicated per-stack network it just ensured for the new container, gated on
+  `PreferCurrentContainerNetwork()` (so host-native/opted-out runs are unaffected). Once that
+  real network attachment exists, every subsequent `CurrentContainerNetwork` call (in this or any
+  later Atmos process in the same container) picks it up automatically -- no changes were needed
+  in `pkg/emulator/manager.go`.
+- `pkg/emulator/endpoint_host.go`: `reachableHostForPublishedPorts()` now prefers
+  `host.docker.internal` (only when it actually resolves, via a 2s-bounded
+  `net.Resolver.LookupHost`) before the raw default-gateway guess, as a secondary hardening for
+  the last-resort fallback path.
+- New tests: `pkg/container/sibling_network_test.go` (`TestSiblingContainerNetworking_Real`, an
+  unstubbed regression test that exercises the real self-detection/join mechanism when actually
+  run inside a container) and `pkg/container/sibling_network_docker_test.go`
+  (`TestSiblingContainerNetworking_Docker`, opt-in via `ATMOS_TEST_SIBLING_CONTAINER=1`, which
+  drives the above by running `go test` itself inside a nested container with no `--network` and
+  the host socket mounted -- the exact reported topology).
+- Unit test coverage added/extended in `pkg/container/{network,common,stack_network}_test.go` and
+  `pkg/container/{docker,podman}_unit_test.go`, and `pkg/emulator/endpoint_host_test.go`.
+
+## Validation
+
+- `go build ./...` -- clean.
+- `go test ./pkg/container/... ./pkg/emulator/...` -- all passing, including the new real-Docker
+  integration test (`TestDockerRuntime_SharedNetworkAliasReachable_Integration`) and the
+  unstubbed self-detection test.
+- `ATMOS_TEST_SIBLING_CONTAINER=1 go test ./pkg/container/ -run TestSiblingContainerNetworking_Docker`
+  -- passes with the fix in place. Verified it's a genuine regression test by temporarily
+  reverting the `joinCurrentContainerToNetwork` call and re-running: fails with
+  `dial tcp: lookup siblingtest-probe on ...: no such host`, confirming the test actually catches
+  the bug rather than passing vacuously.
+- `atmos lint --changed` -- 0 issues.
+- Manual reproduction: disposable copy of the affected application repository, run as a
+  socket-mounted `docker:cli` job container (`docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v
+  "$APP_COPY:/workspace" -e CI=true docker:cli sh -c 'atmos terraform test app -s fixtures
+  --ci'`) with a locally built `atmos` binary. Before the fix: `emulator aws is up at
+  http://172.17.0.1:<port>` followed by `connection refused`. After the fix: `emulator aws is up
+  at http://fixtures-aws:4566` (DNS alias), and the run completes fully --
+  `Success! 1 passed, 0 failed, 0 skipped.` -- with zero `connection refused`/`dial tcp` occurrences
+  across the full ~800-line log, including two real `terraform apply`/`destroy` cycles against
+  the emulator.
+- Addressed CodeRabbit review feedback on cloudposse/atmos#2960: removed an overly broad
+  `"already in"` idempotency substring match (would have masked genuine failures like "port
+  already in use"), bounded the `host.docker.internal` DNS lookup with a timeout, and tightened
+  the corresponding test's timing assertion against the actual configured constant instead of a
+  loose bound.
+
+## Follow-ups
+
+None.

pkg/container/common.go

@@ -433,6 +433,20 @@ func buildStopArgs(containerID string, timeoutSecs int) []string {
 	return []string{"stop", "-t", fmt.Sprintf("%d", timeoutSecs), containerID}
 }
 
+// buildNetworkConnectArgs builds the arguments for a `network connect` operation,
+// attaching containerID to network with each alias registered as a DNS name on
+// it. This function is shared between Docker and Podman runtimes to avoid
+// duplication. Extracted to allow testing the argument building logic without
+// executing commands.
+func buildNetworkConnectArgs(network, containerID string, aliases []string) []string {
+	args := []string{"network", "connect"}
+	for _, alias := range aliases {
+		args = append(args, "--alias", alias)
+	}
+	args = append(args, network, containerID)
+	return args
+}
+
 // buildLogsArgs builds the arguments for a container logs operation.
 // This function is shared between Docker and Podman runtimes to avoid duplication.
 // Extracted to allow testing the argument building logic without executing commands.

pkg/container/common_test.go

@@ -1320,6 +1320,49 @@ func TestBuildStopArgs(t *testing.T) {
 	}
 }
 
+func TestBuildNetworkConnectArgs(t *testing.T) {
+	tests := []struct {
+		name        string
+		network     string
+		containerID string
+		aliases     []string
+		expected    []string
+	}{
+		{
+			name:        "no aliases",
+			network:     "atmos-fixtures",
+			containerID: "abc123",
+			expected:    []string{"network", "connect", "atmos-fixtures", "abc123"},
+		},
+		{
+			name:        "single alias",
+			network:     "atmos-fixtures",
+			containerID: "abc123",
+			aliases:     []string{"fixtures-aws"},
+			expected:    []string{"network", "connect", "--alias", "fixtures-aws", "atmos-fixtures", "abc123"},
+		},
+		{
+			name:        "multiple aliases preserve order",
+			network:     "atmos-fixtures",
+			containerID: "abc123",
+			aliases:     []string{"fixtures-aws", "fixtures-aws-alt"},
+			expected: []string{
+				"network", "connect",
+				"--alias", "fixtures-aws",
+				"--alias", "fixtures-aws-alt",
+				"atmos-fixtures", "abc123",
+			},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := buildNetworkConnectArgs(tt.network, tt.containerID, tt.aliases)
+			assert.Equal(t, tt.expected, result)
+		})
+	}
+}
+
 func TestBuildLogsArgs(t *testing.T) {
 	tests := []struct {
 		name        string

pkg/container/docker.go

@@ -151,6 +151,18 @@ func (d *DockerRuntime) EnsureNetwork(ctx context.Context, name string) error {
 	return networkCreateResult(err, string(output))
 }
 
+// ConnectNetwork attaches an already running container to a user-defined docker
+// network, registering aliases as its DNS names on that network. It implements
+// NetworkConnector so Atmos's own container can join a dedicated stack network
+// it didn't start on.
+func (d *DockerRuntime) ConnectNetwork(ctx context.Context, network, containerID string, aliases []string) error {
+	defer perf.Track(nil, "container.DockerRuntime.ConnectNetwork")()
+
+	cmd := d.command(ctx, buildNetworkConnectArgs(network, containerID, aliases)...)
+	output, err := cmd.CombinedOutput()
+	return networkConnectResult(err, string(output))
+}
+
 // Start starts a container.
 func (d *DockerRuntime) Start(ctx context.Context, containerID string) error {
 	defer perf.Track(nil, "container.DockerRuntime.Start")()

pkg/container/docker_test.go

@@ -667,3 +667,69 @@ func TestDockerRuntime_Exec_Integration(t *testing.T) {
 	err = runtime.Exec(ctx, containerID, []string{"echo", "test"}, execOpts)
 	require.NoError(t, err, "Exec should succeed")
 }
+
+// TestDockerRuntime_SharedNetworkAliasReachable_Integration proves the
+// networking primitive every fix in this package's AttachSharedNetwork /
+// CurrentContainerNetwork machinery ultimately depends on: two sibling
+// containers on the same real, user-defined Docker network can resolve each
+// other by their registered alias and actually connect over TCP. Nothing in
+// the rest of the suite asserted this directly against a real daemon --
+// everything else mocks the runtime, which can't catch a Docker/host
+// networking assumption that turns out to be wrong (as the 172.17.0.1
+// bridge-gateway assumption was).
+func TestDockerRuntime_SharedNetworkAliasReachable_Integration(t *testing.T) {
+	runtime := NewDockerRuntime()
+	ctx := context.Background()
+
+	if err := runtime.Pull(ctx, "alpine:latest"); err != nil {
+		t.Skipf("Docker not available, skipping network alias test: %v", err)
+		return
+	}
+
+	const networkName = "atmos-test-alias-net"
+	require.NoError(t, runtime.EnsureNetwork(ctx, networkName))
+	t.Cleanup(func() {
+		_ = exec.Command("docker", "network", "rm", networkName).Run() // Best-effort test cleanup.
+	})
+
+	// The "server" side: a container that just listens on a TCP port,
+	// registered under the alias its sibling will dial.
+	serverID, err := runtime.Create(ctx, &CreateConfig{
+		Name:    "atmos-test-alias-server",
+		Image:   "alpine:latest",
+		Command: []string{"nc", "-l", "-p", "8080"},
+		Networks: []NetworkAttachment{
+			{Name: networkName, Aliases: []string{"probe-server"}},
+		},
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = runtime.Remove(ctx, serverID, true) })
+	require.NoError(t, runtime.Start(ctx, serverID))
+
+	// The "client" side: a plain sibling container on the same network, with
+	// no special knowledge of the server beyond its alias -- exactly the
+	// relationship between a job container and an emulator container.
+	clientID, err := runtime.Create(ctx, &CreateConfig{
+		Name:            "atmos-test-alias-client",
+		Image:           "alpine:latest",
+		OverrideCommand: true, // sleep infinity.
+		Networks: []NetworkAttachment{
+			{Name: networkName, Aliases: []string{"probe-client"}},
+		},
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = runtime.Remove(ctx, clientID, true) })
+	require.NoError(t, runtime.Start(ctx, clientID))
+
+	// Retry briefly: the server's nc listener may not be bound the instant
+	// its container reports "started".
+	var lastErr error
+	for range 10 {
+		lastErr = runtime.Exec(ctx, clientID, []string{"nc", "-z", "-w", "2", "probe-server", "8080"}, &ExecOptions{})
+		if lastErr == nil {
+			break
+		}
+		time.Sleep(500 * time.Millisecond)
+	}
+	assert.NoError(t, lastErr, "client container must reach the server container by its network alias")
+}

pkg/container/network.go

@@ -22,6 +22,22 @@ type NetworkEnsurer interface {
 	EnsureNetwork(ctx context.Context, name string) error
 }
 
+// NetworkConnector is an optional Runtime capability: it attaches an already
+// running container to a named network. Used to join Atmos's own current
+// container to a stack's dedicated network when CurrentContainerNetwork can't
+// reuse its existing network for DNS-alias resolution (e.g. it's only on
+// Docker's default "bridge", which doesn't support aliases) — without this,
+// Atmos would only be able to guess a reachable address for containers it
+// starts, rather than actually being on the same network as them. Runtimes
+// that don't implement it simply skip the join; callers then fall back to a
+// published-port-based address guess.
+type NetworkConnector interface {
+	// ConnectNetwork attaches containerID to the named network, registering
+	// aliases (if any) as its DNS names on that network. Treats a container
+	// already connected to the network as success.
+	ConnectNetwork(ctx context.Context, network, containerID string, aliases []string) error
+}
+
 // networkCreateResult maps a `network create` invocation to an idempotent result:
 // an "already exists" failure is success, so repeated `up`s are no-ops.
 func networkCreateResult(runErr error, output string) error {
@@ -33,3 +49,18 @@ func networkCreateResult(runErr error, output string) error {
 	}
 	return fmt.Errorf("%w: create network: %w: %s", errUtils.ErrContainerRuntimeOperation, runErr, strings.TrimSpace(output))
 }
+
+// networkConnectResult maps a `network connect` invocation to an idempotent
+// result: a container already attached to the network is success, so a repeat
+// join (e.g. across separate Atmos processes in the same job container) is a
+// no-op rather than an error.
+func networkConnectResult(runErr error, output string) error {
+	if runErr == nil {
+		return nil
+	}
+	lower := strings.ToLower(output)
+	if strings.Contains(lower, "already exists") || strings.Contains(lower, "already connected") {
+		return nil
+	}
+	return fmt.Errorf("%w: connect network: %w: %s", errUtils.ErrContainerRuntimeOperation, runErr, strings.TrimSpace(output))
+}

pkg/container/network_test.go

@@ -26,3 +26,38 @@ func TestNetworkCreateResult(t *testing.T) {
 		assert.Contains(t, got.Error(), "permission denied")
 	})
 }
+
+func TestNetworkConnectResult(t *testing.T) {
+	t.Run("nil error is success", func(t *testing.T) {
+		require.NoError(t, networkConnectResult(nil, ""))
+	})
+
+	t.Run("docker already-exists output is idempotent success", func(t *testing.T) {
+		err := errors.New("exit status 1")
+		// Docker: "Error response from daemon: endpoint with name X already exists in network Y".
+		require.NoError(t, networkConnectResult(err, "Error response from daemon: endpoint with name atmos-job already exists in network atmos-fixtures"))
+	})
+
+	t.Run("podman already-connected output is idempotent success", func(t *testing.T) {
+		err := errors.New("exit status 125")
+		// Podman: "Error: <container> is already connected to network <network>: network is already connected".
+		require.NoError(t, networkConnectResult(err, "Error: abc123 is already connected to network atmos-fixtures: network is already connected"))
+	})
+
+	t.Run("genuine failure propagates", func(t *testing.T) {
+		err := errors.New("exit status 1")
+		got := networkConnectResult(err, "no such container")
+		require.Error(t, got)
+		assert.Contains(t, got.Error(), "no such container")
+	})
+
+	t.Run("already-in-use output is a genuine failure, not idempotent success", func(t *testing.T) {
+		// A too-broad "already in" substring match would misreport this as
+		// success -- it must only match the specific "already exists"/"already
+		// connected" idempotent cases above, not any "already in ..." message.
+		err := errors.New("exit status 125")
+		got := networkConnectResult(err, "Error: port 8080 is already in use")
+		require.Error(t, got)
+		assert.Contains(t, got.Error(), "already in use")
+	})
+}

pkg/container/podman.go

@@ -91,6 +91,18 @@ func (p *PodmanRuntime) EnsureNetwork(ctx context.Context, name string) error {
 	return networkCreateResult(err, string(output))
 }
 
+// ConnectNetwork attaches an already running container to a user-defined podman
+// network, registering aliases as its DNS names on that network. It implements
+// NetworkConnector so Atmos's own container can join a dedicated stack network
+// it didn't start on.
+func (p *PodmanRuntime) ConnectNetwork(ctx context.Context, network, containerID string, aliases []string) error {
+	defer perf.Track(nil, "container.PodmanRuntime.ConnectNetwork")()
+
+	cmd := p.command(ctx, buildNetworkConnectArgs(network, containerID, aliases)...)
+	output, err := cmd.CombinedOutput()
+	return networkConnectResult(err, cleanPodmanOutput(output))
+}
+
 // Create creates a new container.
 func (p *PodmanRuntime) Create(ctx context.Context, config *CreateConfig) (string, error) {
 	defer perf.Track(nil, "container.PodmanRuntime.Create")()