Codespaces: "Cannot connect to the Docker daemon" occurs randomly (postCreateCommand)

[mandrasch]

Hi,

I'm trying to get DDEV - an open source Docker-based PHP development toolkit - working reliably in Codespaces. We already had a discussion here, there were some fixes for docker-in-docker in unversal image in the past (🙏 ) and @eiriksm (and others) provided a wait for docker script for the postCreateCommand, which I thought worked.

#!/bin/bash
set -ex

wait_for_docker() {
  while true; do
    docker ps > /dev/null 2>&1 && break
    sleep 1
  done
  echo "Docker is ready."
}

wait_for_docker

# proceed with commands requiring docker

But for some Codespaces launches I still run into

"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?".

Other launches work fine as expected, current ratio is 5 working, 1 error - seems random.

My devcontainer.json:

{
   "image": "mcr.microsoft.com/devcontainers/universal:2",
   "features": {
       "ghcr.io/ddev/ddev/install-ddev:latest": {}
   },
   // ...
   "postCreateCommand": "chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh"
}

Example error output:

2024-05-18 08:23:04.024Z: chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh
2024-05-18 08:23:04.171Z: + wait_for_docker
+ ********
+ docker ps
2024-05-18 08:23:04.666Z: + break
2024-05-18 08:23:04.670Z: + echo 'Docker is ready.'
Docker is ready.
+ ddev debug download-images
2024-05-18 08:23:04.829Z: Downloading ********/docker/compose/releases/download/v2.27.0/docker-compose-linux-x86_64 ... 2024-05-18 08:23:04.839Z: 
2024-05-18 08:23:05.647Z: 
docker-compose 0 B / 60.09 MiB [---------------------------------------]   0.00%2024-05-18 08:23:05.848Z: 
docker-compose 25.86 MiB / 60.09 MiB [==============>------------------]  43.04%2024-05-18 08:23:06.048Z: 
docker-compose 41.45 MiB / 60.09 MiB [======================>----------]  68.99%2024-05-18 08:23:06.235Z: 
docker-compose 60.09 MiB / 60.09 MiB [==============================] 100.00% 0s
Download complete. 
2024-05-18 08:23:20.065Z: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024-05-18 08:23:20.078Z: Failed to PullBaseContainerImages(): exit status 1 
2024-05-18 08:23:20.085Z: {"outcome":"error","message":"Command failed: /bin/sh -c chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh","description":"The postCreateCommand in the devcontainer.json failed.","containerId":"26560cae79c4541c929a677101cf0dea4a9632ceb233c8571c85425a124d2a54"}
2024-05-18 08:23:20.126Z: postCreateCommand failed with exit code 1. Skipping any further user-provided commands.

Full Log: https://gist.github.com/mandrasch/f0ddc7c275c8df68275c88bae5ec9424

My setup: https://github.com/mandrasch/ddev-codespaces-launch-blank
Usage: Browser-based
postCreateCommand: https://github.com/mandrasch/ddev-codespaces-launch-blank/blob/main/.devcontainer/postCreateCommand.sh

Is there a way to debug this? Thanks very much in advance for assistance! 🤗

[rfay]

I thought this had already been dealt with a time or two in previous issues?

[mandrasch]

I thought this had already been dealt with a time or two in previous issues?

Yes, there were fixes in docker-in-docker (universal image), but there is also a new spec for better lifecycle with docker in codespaces mentioned here: #634 (comment)

I'm not sure if this improvement would solve the random docker connection error I saw a few times now or if this is unrelated, therefore I opened up a new issue.

[samruddhikhandale]

Hi 👋

Docker is ready.
2024-05-18 08:23:20.065Z: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

This doesn't look good, the wait_for_docker script thinks the docker started, but apparently that's not the case?

Can you update https://github.com/mandrasch/ddev-codespaces-launch-blank/blob/main/.devcontainer/postCreateCommand.sh as follows and see if this helps? Thanks!

# Docker can take a couple seconds to come up. Wait for it to be ready before
# proceeding with bootstrap.
iterations=10
while ! docker ps &>/dev/null; do
  if [[ $iterations -eq 0 ]]; then
    echo "Timeout waiting for the Docker daemon to start."
    exit 1
  fi

  iterations=$((iterations - 1))
  echo 'Docker is not ready. Waiting 10 seconds and trying again.'
  sleep 10
done

[mandrasch]

Hi @samruddhikhandale, thanks very much for reply! Much appreciated!

I tested the new wait for docker script.

1 attempt out of 11 still failed:
https://gist.github.com/mandrasch/1d293ba7ebe34aef6b8134a4637b29ad

In the failed attempt, docker ps seems to be responding immediately? There are no iterations, but it fails afterwards.

2024-06-06 15:52:38.406Z: + iterations=102024-06-06 15:52:38.419Z: 
+ docker ps
2024-06-06 15:52:38.657Z: + ddev debug download-images2024-06-06 15:52:38.668Z: 

2024-06-06 15:52:53.999Z: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024-06-06 15:52:54.004Z: Failed to PullBaseContainerImages(): exit status 1 

The sucessful attempts seem to run through some iterations before docker ps responds:

2024-06-06 15:53:07.738Z: chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh
2024-06-06 15:53:07.850Z: + iterations=10
+ docker ps
2024-06-06 15:53:19.330Z: + [[ 10 -eq 0 ]]2024-06-06 15:53:19.333Z: 
+ iterations=9
+ echo 'Docker is not ready. Waiting 10 seconds and trying again.'
Docker is not ready. Waiting 10 seconds and trying again.
+ sleep 10
2024-06-06 15:53:29.346Z: + docker ps
2024-06-06 15:53:39.589Z: + [[ 9 -eq 0 ]]
2024-06-06 15:53:39.598Z: + iterations=8
+ echo 'Docker is not ready. Waiting 10 seconds and trying again.'
Docker is not ready. Waiting 10 seconds and trying again.
+ sleep 10
2024-06-06 15:53:49.604Z: + docker ps
2024-06-06 15:53:49.641Z: + ddev debug download-images
2024-06-06 15:53:49.726Z: Downloading ********/docker/compose/releases/download/v2.27.0/docker-compose-linux-x86_64 ... 

Log of successful attempt: https://gist.github.com/mandrasch/8b296552af9c66a1acc4036a10f1154b

[samruddhikhandale]

Thanks for getting back to us. @gauravsaini04 Can you help investigate the issue? It would be good to find out what could be causing the docker to stop after getting started.

[gauravsaini04]

Hi @samruddhikhandale, I was not able to replicate the issue. Tried approx. 20 - 25 times and it would pick up docker almost instantly and doesn't seem to be failing afterwards as well. @mandrasch, let me know if you are still facing the issue with docker daemon stopping randomly.

[colespencer1453]

Potentially unrelated, but I am attempting to start a supabase docker container in a github codespace prebuild.

1. I'm not sure that is possible or proper usage of a prebuild
2. If it is possible I'm not sure which build step it should occur in.

I currently have this as my dev container config. My main goal is to avoid having to wait on docker registry downloads when running supabase start after first creation of a codespace. However, since this should just be a one time build time for the first creation of the the codespace it probably isn't work including in the prebuild

{
  "image": "mcr.microsoft.com/devcontainers/universal:2",
  "features": {
    "ghcr.io/devcontainers-extra/features/supabase-cli:1": {}, // Supabase CLI installed
    "ghcr.io/devcontainers-community/npm-features/prettier:1": {} // Prettier feature
  },
  "onCreateCommand": "npm install && supabase start", // Install dependencies and start frontend after Codespace is created
  "postStartCommand": "supabase start && npm run dev" // Ensure Supabase and frontend run on every start
}

[eric-gitta-moore]

same here.

my devcontainer.json

{
  "name": "Mili Container",
  "image": "mcr.microsoft.com/devcontainers/universal:2",
  "features": {
    "ghcr.io/devcontainers/features/docker-in-docker:2": {}
  },
  "customizations": {
    "vscode": {
      "extensions": [],
      "settings": {
        "editor.formatOnSave": true
      }
    }
  },
  "postStartCommand": "cd /workspaces/mili/ && docker compose up -d"
}

log:

/usr/local/share/pull-git-lfs-artifacts.sh
Fetching git lfs artifacts...
Running the postStartCommand from devcontainer.json...

cd /workspaces/mili/ && docker compose up -d
unable to get image 'grafana/grafana': Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[931 ms] postStartCommand failed with exit code 1. Skipping any further user-provided commands.

{"outcome":"error","message":"Command failed: /bin/sh -c cd /workspaces/mili/ && docker compose up -d","description":"The postStartCommand in the devcontainer.json failed.","containerId":"abed77b7f8770a1ecca83a1d58b887eb49c5988984f8eee4d969bad44c9b4c11"}