container pull from ghcr registry DENIED

[StephenKing]

What happened?
DevPod somehow stopped working for me. I've now reset everything and end up with

[12:14:36] debug devcontainer up build image get image build info get image config remotely retrieve image ghcr.io/devcontainers/templates/alpine:3.4.0 GET https://ghcr.io/token?scope=repository%3Adevcontainers%2Ftemplates%2Falpine%3Apull&service=ghcr.io: DENIED: denied

What did you expect to happen instead?

Provisioning to work and IDE to start

How can we reproduce the bug? (as minimally and precisely as possible)

Add the AWS provider

$ devpod provider add skevetter/devpod-provider-aws \
  --name aws-skevetter \
  -o AWS_INSTANCE_PROFILE_ARN=arn:aws:iam::123456789012:instance-profile/foobar \
  -o AWS_DISK_SIZE=100 \
  -o AWS_SECURITY_GROUP_ID=sg-abcdefg \
  -o AWS_SUBNET_ID=subnet-abcdef \
  -o AWS_USE_SESSION_MANAGER=true \
  -o AWS_INSTANCE_TAGS=Name=Creator,Value=steffen

Important

I'm using AWS SSM, maybe this plays a role

Create Workspace

My devcontainer.json:

{
  "image": "ghcr.io/devcontainers/templates/alpine:3.4.0"
}

Options

set to default after rm -r ~/.devpod/

Opening the Woskspace

initial start fails with, probably because SSM not yet ready

[12:12:37] debug done creating devcontainer
[12:12:38] fatal executing agent command injection timeout after 1 attempts

logs-initial.txt

subsequent click on Open does provisioning, but then fails with

[12:14:35] info No containers need to be restarted.
[12:14:36] debug devcontainer up build image get image build info get image config remotely retrieve image ghcr.io/devcontainers/templates/alpine:3.4.0 GET https://ghcr.io/token?scope=repository%3Adevcontainers%2Ftemplates%2Falpine%3Apull&service=ghcr.io: DENIED: denied
[12:14:36] debug failed to rerun as root
[12:14:36] debug error parsing workspace info rerun as root exit status 1
[12:14:36] info No user sessions are running outdated binaries.
[12:14:36] debug SSH session closed
[12:14:36] debug done inject
[12:14:36] debug SSH client closed
[12:14:36] debug tunnel server started, waiting for command completion
[12:14:36] debug done creating devcontainer
[12:14:36] debug done injecting
[12:14:36] debug run provider command
[12:14:36] debug retrying
[12:14:36] debug done executing SSH server helper command
[12:14:36] debug agent injection metrics
[12:14:36] debug done exec
[12:14:37] fatal run agent command failed: Process exited with status 1
error parsing workspace info rerun as root exit status 1

logs-subsequent.txt

Local Environment:

• DevPod Version: v0.9.4
• Operating System: mac
• ARCH of the OS: ARM64

DevPod Provider:

• Cloud Provider: aws
• Custom provider: provide imported provider.yaml config file

Anything else we need to know?

• thanks for your efforts!
• for sure, I can pull the image on the machine manually

[StephenKing]

What I've also tried:

• in the provider config, set false for Inject Docker Credentials and Inject Git Credentials
• move to a public subnet so that I don't need SSM

In those cases, I don't get the DENIED error from above, but also not very charming:

[13:37:04] info creating devcontainer
[13:37:04] debug starting SSH tunnel execution
[13:37:04] debug adding SSH keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
[13:37:04] debug inject and run command
[13:37:04] debug starting agent injection
[13:37:04] debug prefer downloading agent from URL
[13:37:04] debug execute inject script
[13:37:04] debug run provider command
[13:37:05] debug running ssh-add
[13:37:05] debug running ssh-add
[13:37:05] debug running ssh-add
[13:37:05] debug running ssh-add
[13:37:05] debug running ssh-add
[13:37:05] debug attempting to create SSH client
[13:37:24] debug done inject
[13:37:24] debug done injecting
[13:37:24] debug retrying
[13:37:24] debug done exec
[13:37:34] debug agent injection metrics
[13:37:34] debug done executing SSH server helper command
[13:37:34] debug tunnel server started, waiting for command completion
[13:37:34] debug done creating devcontainer
[13:37:35] fatal executing agent command injection timeout after 1 attempts

[StephenKing]

or on subsequent attempts

[13:40:11] info No containers need to be restarted.
[13:40:12] info No user sessions are running outdated binaries.
[13:40:12] info No VM guests are running outdated hypervisor (qemu) binaries on this host.
[13:40:12] info command succeeded on attempt 1
[13:40:13] info + docker version
[13:40:13] info Client: Docker Engine - Community
[13:40:13] info Version:           29.1.5
[13:40:13] info API version:       1.52
[13:40:14] info Go version:        go1.25.6
[13:40:14] info Git commit:        0e6fee6
[13:40:14] info Built:             Fri Jan 16 12:48:47 2026
[13:40:14] info OS/Arch:           linux/amd64
[13:40:14] info Context:           default
[13:40:15] debug devcontainer up runner run container failed to start dev container exit status 127
[13:40:15] debug failed to rerun as root
[13:40:15] debug error parsing workspace info rerun as root exit status 1
[13:40:15] info Server: Docker Engine - Community
[13:40:15] debug SSH session closed
[13:40:15] debug SSH client closed
[13:40:15] debug done inject
[13:40:15] debug tunnel server started, waiting for command completion
[13:40:15] debug done injecting
[13:40:15] debug done creating devcontainer
[13:40:15] debug run provider command
[13:40:15] debug retrying
[13:40:15] debug agent injection metrics
[13:40:15] debug done executing SSH server helper command
[13:40:15] debug done exec
[13:40:16] fatal run agent command failed: Process exited with status 1
error parsing workspace info rerun as root exit status 1

[StephenKing]

same effect of kind of silently failing when I switch to the SSH provider to connect to the same instance

[13:44:24] debug refreshed workspace options
[13:44:24] debug no pro instance available, skipping provider upgrade check
[13:44:24] debug acquire workspace lock
[13:44:24] debug acquired workspace lock
[13:44:24] info creating devcontainer
[13:44:24] debug starting SSH tunnel execution
[13:44:24] debug adding SSH keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
[13:44:24] debug inject and run command
[13:44:24] debug starting agent injection
[13:44:24] debug prefer downloading agent from URL
[13:44:24] debug execute inject script
[13:44:24] debug run provider command
[13:44:24] debug running ssh-add
[13:44:24] debug running ssh-add
[13:44:24] debug running ssh-add
[13:44:24] debug running ssh-add
[13:44:24] debug running ssh-add
[13:44:24] debug attempting to create SSH client
[13:44:38] debug received line after pong
[13:44:39] debug SSH client created
[13:44:40] debug SSH session created
[13:44:40] debug forwarding SSH agent
[13:44:40] debug running agent command in SSH tunnel
[13:44:40] debug starting to decode and write workspace info
[13:44:40] debug decoded workspace info
[13:44:40] debug checking if root privileges are required
[13:44:40] debug error trying to reach docker daemon
[13:44:40] debug re-run as root
[13:44:40] debug starting to decode and write workspace info
[13:44:40] debug decoded workspace info
[13:44:40] debug checking if root privileges are required
[13:44:40] debug creating agent workspace directory
[13:44:40] debug using workspace dir
[13:44:40] debug checking for existing workspace config
[13:44:40] debug checking local folder workspace source
[13:44:40] debug workspace origin not accessible
[13:44:40] debug set content folder to default location
[13:44:40] debug writing workspace info to file
[13:44:40] debug wrote workspace info
[13:44:40] debug processed workspace info
[13:44:40] debug received ping from agent
[13:44:40] debug created logger
[13:44:40] debug credentials server started
 port=16123
[13:44:41] debug incoming client connection
 path=/
[13:44:41] debug credentials server started
[13:44:41] debug create content folder
 path=/home/devpod/.devpod/agent/contexts/default/workspaces/containerlab-ssh/content
[13:44:41] debug using custom docker path docker
[13:44:41] debug devcontainer up runner run container failed to start dev container exit status 127
[13:44:41] debug failed to rerun as root
[13:44:41] debug error parsing workspace info rerun as root exit status 1
[13:44:41] debug SSH session closed
[13:44:41] debug SSH client closed
[13:44:41] debug done inject
[13:44:41] debug tunnel server started, waiting for command completion
[13:44:41] debug done creating devcontainer
[13:44:41] debug done injecting
[13:44:41] debug run provider command
[13:44:41] debug retrying
[13:44:41] debug agent injection metrics
[13:44:41] debug done executing SSH server helper command
[13:44:41] debug done exec
[13:44:42] fatal run agent command failed: Process exited with status 1
error parsing workspace info rerun as root exit status 1

root@ip-10-0-0-224:~# ps aux | grep devpod
root        2734  0.0  0.9 1340424 76220 ?       Ssl  12:39   0:00 /var/lib/toolbox/devpod agent daemon
devpod     14170  9.7  0.1  20268 11512 ?        Ss   12:45   0:00 /usr/lib/systemd/systemd --user
devpod     14171  0.0  0.0  21144  3564 ?        S    12:45   0:00 (sd-pam)
root       14270  0.0  0.0   7076  2232 pts/1    S+   12:45   0:00 grep --color=auto devpod

root@ip-10-0-0-224:~# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

root@ip-10-0-0-224:~# docker ps -a
CONTAINER ID   IMAGE                                          COMMAND                  CREATED         STATUS    PORTS     NAMES
219c7d6fe976   ghcr.io/devcontainers/templates/alpine:3.4.0   "/bin/sh -c 'echo Co…"   2 minutes ago   Created             adoring_brahmagupta
1620891c22fa   ghcr.io/devcontainers/templates/alpine:3.4.0   "/bin/sh -c 'echo Co…"   6 minutes ago   Created             great_jang

[skevetter]

Try using the devcontainer image https://hub.docker.com/r/microsoft/devcontainers-base instead of the template

mcr.microsoft.com/devcontainers/base:alpine

[StephenKing]

Thanks for looking into the issue. But which image do you mean? it lists the same URL image URL

EDIT: got it..

[StephenKing]

okay, that indeed brings me further. But it was just my attempt to reduce my config as much as possible to troubleshoot my problems. I will get back to you. Thanks for the help.

[StephenKing]

I figured it out. The setting Inject Docker Credentials: true (under the provider's Agent settings) breaks GHCR pulls for me. With this set to false (and deletion of the workspace + a new EC2 instance) it works for me!

How is this setting supposed to interact with the DevPod Settings Docker credentials forwarding, which sounds like the same (and I have disabled).

Agent options:

DevPod settings:

[skevetter]

Glad to hear you got it working. Initially, I was hinting that you were using a template and not a devcontainer. I believe that was the source of the first issue.

How is this setting supposed to interact with the DevPod Settings Docker credentials forwarding

As for the credentials, I will have to look into that to see which setting overrides which. How did you have it configured initially?

Here is what I am thinking should happen. Though, if this is the case, I do not know what is the reason for having inject Docker credentials as a provider setting. Maybe this should only be a user customization.

Customation Docker Forwarding	Inject Docker Credentials	Scenario
TRUE	TRUE	True; Docker Credential Forwarding enabled
TRUE	FALSE	True because user customization overrides inject
FALSE	TRUE	False because User customization overrides inject
FALSE	FALSE	False; Docker Credential Forwarding disabled

FWIW, I am working on fixing up the AWS provider now. I can take these new findings into consideration.

[StephenKing]

As for the credentials, I will have to look into that to see which setting overrides which. How did you have it configured initially?

I honestly don't think that I had disabled it before.

As we see in the table, the global setting is completely ignored - so not sure if the provider-specific one could be an empty string and by that use the global setting by default?

On the other hand, I don't know why injecting my credentials breaks it. I've also tried with empty ({}) ~/.docker/config.json, but that had the same effect -or I was just somehow to dump. Should it otherwise work, even though I have this one:

{
	"auths": {},
	"credsStore": "osxkeychain",
	"currentContext": "orbstack"
}

Let me know if I should open a separate issue for the provider options.

[tidux]

I have all credential forwarding options disabled DevPod-wide and I'm still getting the same ghcr.io error.

[skevetter]

Thank you for the information. I am investigating.

Is this issue present in original DevPod? I would like to know if the issue is longstanding or a regression.