ssh agent sock file lookup failure

[keturn]

What happened?

[17:53:59] debug SSH session created
[17:53:59] debug Forwarding ssh-agent using /tmp/auth-agent805773347/listener.sock
[17:53:59] info Execute SSH server command: sh -c '/usr/local/bin/devpod' agent container setup --setup-info '…' --chown-workspace --inject-git-credentials --debug
[17:53:59] debug Received ping from agent
[17:53:59] error devcontainer up run agent command Process exited with status 1
[17:53:59] error chown ssh agent sock file lookup user user: unknown user 1000:1000
[17:53:59] debug Connection to SSH Server closed

What did you expect to happen instead?

connection to user 0 in a new container

How can we reproduce the bug? (as minimally and precisely as possible)

My devcontainer.json:

{
	"name": "Node.js",
	"image": "docker.io/library/node:lts-alpine"
}

Local Environment:

• DevPod Version: 0.8.3
• Operating System: linux (Ubuntu 25.04)
• ARCH of the OS: AMD64

DevPod Provider:

• Local/remote provider: ssh / podman

Anything else we need to know?

If I explicitly put "remoteUser": "root" in the configuration, it does get a bit further along, but Gateway doesn't succeed in setting up.

I wondered if this was specific to the ssh provider, but using local podman provider fails in the same way.

For comparison, VS Code on its own does seem okay with using this configuration (independent of devpod) with no remoteUser set.

[skevetter]

After merging PR #155, the node image still fails but due to GPG not being installed. To me, this seem like a valid failure. My thoughts are the user defining the devcontainer image is responsible for providing the necessary configuration or a pre-built image that has the minimum requirements for a devcontainer. I think of it as all devcontainers are OCI-compliant containers, but not all oci-compliant containers are devcontainers.

For reference, I used mcr.microsoft.com/devcontainers/javascript-node:dev-trixie and mcr.microsoft.com/devcontainers/base:alpine. Both containers came up successfully.

What are you thoughts? Should DevPod have a hook to install GPG automatically or should the user provide an image/configuration/feature that has this included in the definition.

Error message on VScode for docker.io/library/node:lts-alpine devcontainer

13:47:26.109] stderr> 13:47:26 info stop container gpg-agent: exec: "gpgconf": executable file not found in $PATH
[13:47:26.109] stderr> 13:47:26 info exec: "gpgconf": executable file not found in $PATH
[13:47:26.111] stderr> 13:47:26 error Error tunneling to container: wait: remote command exited without exit status or exit signal
[13:47:26.565] stderr> 13:47:26 error Try enabling Debug mode under Settings to see a more verbose output
[13:47:26.565] stderr> 13:47:26 fatal tunnel to container run in container run gpg agent setup command Process exited with status 1
[13:47:26.568] stderr> kex_exchange_identification: Connection closed by remote host
[13:47:26.568] stderr> Connection closed by UNKNOWN port 65535
[13:47:26.569] > local-server-1> ssh child died, shutting down
[13:47:26.575] Local server exit: 0
[13:47:26.575] Received install output: local-server-1> Running ssh connection command: ssh -v -T -D 60712 -o ConnectTimeout=1800 node-lts-alpine.devpod
local-server-1> Spawned ssh, pid=54253
13:47:25 info sh: gpg: not found
13:47:26 info Reverse forwarding local unix//Users/skevette/.gnupg/S.gpg-agent.extra to remote unix//Users/skevette/.gnupg/S.gpg-agent.extra
13:47:26 info stop container gpg-agent: exec: "gpgconf": executable file not found in $PATH
13:47:26 info exec: "gpgconf": executable file not found in $PATH
13:47:26 error Error tunneling to container: wait: remote command exited without exit status or exit signal

[keturn]

Huh, I had no idea gpg might be a required component. Is that specifically for the ssh provider's "Inject Docker Credentials" functionality? In any case, I agree that's a very separate issue from the sock file lookup error.

[keturn]

The official microsoft.com/devcontainers all have a non-root user with uid 1000, so they're not a great test case for this issue.

I reproduced it with that node:alpine image because it's nice and small and an image someone is likely to actually have Dockerfiles based on, but the Dockerfile I was originally working with when I encountered with was built on docker.io/nvidia/cuda:12.4.1-runtime-ubuntu22.04. In 0.8.6 I still get an error like this:

Execute SSH server command: bash -c '/usr/local/bin/devpod' agent container setup --setup-info '…' --chown-workspace --inject-git-credentials --debug
received ping from agent
Created logger
Start setting up container...
credentials server started
port=15420
incoming client connection
path=/
credentials server started
Chown workspace...
Chown projects...
devcontainer up run agent command Process exited with status 1
user: unknown user 1000:1000

Now, it turns out that something has added a user 1000 to that container (and populated it with the host user info), but the Dockerfile certainly didn't built it that way. inspect shows the container's Config.User is 1000:1000, but the underlying image has no User or Config.User keys in its inspect output at all.

According to the container's CreateCommand info, it was created with --userns=keep-id, which I think must be why it's trying to use 1000. IMHO, this container should have been run with the default userns option of host. The spec is fuzzy around the edges here, but in practice VS Code seems to agree with me, as inspecting the container it created for this config shows Config.User is "" and it was run with --userns="".

We can see devcontainers-cli made that decision in devcontainers/cli#1018

[skevetter]

I had no idea gpg might be a required component

To clarify, this is only required because I have gpg forwarding configured on my host. Assuming gpg forwarding is not set on your host, the remote should not require it.

it turns out that something has added a user 1000

Did devpod create this user?

In an ideal scenario, which user should the container (docker.io/library/node:lts-alpine or docker.io/nvidia/cuda:12.4.1-runtime-ubuntu22.04) be using provided that it launched successfully.

It looks like there was an intentional decision to specially handle podman.

devpod/pkg/driver/docker/docker.go

Lines 302 to 309 in c8210ef

	// In case we're using podman, let's use userns to keep
	// the ID of the user (vscode) inside the container as
	// the same of the external user.
	// This will avoid problems of mismatching chowns on the
	// project files.
	if d.Docker.IsPodman() && os.Getuid() != 0 {
	args = append(args, "--userns", "keep-id")
	}

[skevetter]

I have been thinking about replacing the section with a podman-specific cli flag for userns. It would allow podman users to provide the usersns with the up command. Thoughts?

[skevetter]

This PR #240 should fix (or at least provide flexible options) when working with podman.

[skevetter]

You can now specify userns directly as part of the up command. Let me know if this does not resolve your issue, otherwise I will close this ticket.

devpod/cmd/up.go

Lines 148 to 150 in f81a3a7

	upCmd.Flags().StringVar(&cmd.Userns, "userns", "", "User namespace to use for the container (Podman only; e.g. \"keep-id\", \"host\", or \"auto\")")
	upCmd.Flags().StringSliceVar(&cmd.UidMap, "uidmap", []string{}, "UID mapping for Podman user namespace (Podman only; format: container_id:host_id:amount, e.g. \"0:1000:1\")")
	upCmd.Flags().StringSliceVar(&cmd.GidMap, "gidmap", []string{}, "GID mapping for Podman user namespace (Podman only; format: container_id:host_id:amount, e.g. \"0:1000:1\")")