Fail without any error code

[SomaGallai]

Minimal .gitlab-ci.yml illustrating the issue

---
default:
  image: docker:latest
  services:
    - docker:23.0.3-dind
stages:
 - build

initial-build-hello:
  stage: build
  script:
    - echo "Empty step to trigger pipeline."

Expected behavior
No public URL, but should echo Empty step to trigger pipeline. and succeed.

Host information
MacOS Ventura 13.3.1
gitlab-ci-local 4.40.1

Additional context
Actual behaviour/output:

$ gitlab-ci-local         
parsing and downloads finished in 31 ms
initial-build-hello               starting docker:latest (build)
initial-build-hello               copied to docker volumes in 504 ms

 FAIL  initial-build-hello              
pipeline finished in 743 ms

[firecow]

Hmm.... I'm seeing a lot more output

parsing and downloads finished in 40 ms
initial-build-hello starting docker:latest (build)
initial-build-hello copied to docker volumes in 746 ms
initial-build-hello started service image: docker:23.0.3-dind with aliases: docker in 415 ms
initial-build-hello > still running...
initial-build-hello > still running...
initial-build-hello > still running...
initial-build-hello service image: docker:23.0.3-dind healthcheck failed
initial-build-hello   Command failed with exit code 124: docker run --rm --name=gcl-wait-for-it-864454-2375 --network gitlab-ci-local-864454 willwill/wait-for-it docker:2375 -t 30
initial-build-hello   wait-for-it.sh: waiting 30 seconds for docker:2375
initial-build-hello   wait-for-it.sh: timeout occurred after waiting 30 seconds for docker:2375
initial-build-hello also see (/home/mjn/workspace/mjn-gitlab-ci-debugging/.gitlab-ci-local/services-output/initial-build-hello/docker:23.0.3-dind-0.log)
initial-build-hello service image: docker:23.0.3-dind healthcheck failed
initial-build-hello   Command failed with exit code 124: docker run --rm --name=gcl-wait-for-it-864454-2376 --network gitlab-ci-local-864454 willwill/wait-for-it docker:2376 -t 30
initial-build-hello   wait-for-it.sh: waiting 30 seconds for docker:2376
initial-build-hello   wait-for-it.sh: timeout occurred after waiting 30 seconds for docker:2376
initial-build-hello also see (/home/mjn/workspace/mjn-gitlab-ci-debugging/.gitlab-ci-local/services-output/initial-build-hello/docker:23.0.3-dind-0.log)
initial-build-hello $ echo "Empty step to trigger pipeline."
initial-build-hello > Empty step to trigger pipeline.
initial-build-hello finished in 33 s

 PASS  initial-build-hello
pipeline finished in 34 s

Let's keep this here, and see if anyone is able to elaborate on the problem, if there is one.

[SomaGallai]

I forgot to mention it was running properly for at least 10+ runs previously and just suddenly started giving me this issue and I wasn't able to get it working once again. I have tried clearing all config and cache in .gitlab-ci-local, removed all docker images and now with the issue .gitlab-ci-local/builds empty.

.gitlab-ci-local/state.yml has the following inside it:

---
pipelineIid: 1

.gitlab-ci-local/output/initial-build-hello.log is completely empty as well

[firecow]

@SomaGallai Can you please rerun it like 20 times in a row.
I'm suspecting a seqmentation fault introduced by Docker Desktop for Mac in latest version.
What Docker Desktop version do you run ?

[SomaGallai]

@firecow I have tried re-running more than 20 times with no success.
I have Docker Desktop 4.19.0 (106363) and Docker Engine v23.0.5

[firecow]

@atli-c Can you try this, you have the exact same setup?

[atli-c]

Host info:
macOS Ventura 13.3.1 (a); Docker Desktop 4.19.0 (on intel cpu); docker engine v23.0.5; gitlab-ci-local v4.40.1
In a new empty directory that is not a git repo. Only with the "Minimal .gitlab-ci.yml illustrating the issue" above.

First run:

Using fallback git commit data
Using fallback git remote data
parsing and downloads finished in 48 ms
initial-build-hello starting docker:latest (build)
initial-build-hello copied to docker volumes in 4.99 s
initial-build-hello > still running...
initial-build-hello pulled docker:23.0.3-dind in 6.01 s
initial-build-hello started service image: docker:23.0.3-dind with aliases: docker in 750 ms
initial-build-hello > still running...
initial-build-hello > still running...
initial-build-hello > still running...
initial-build-hello service image: docker:23.0.3-dind healthcheck failed: docker:2376
initial-build-hello see (/Users/atc/tmp/firecow_gitlab-ci-local_issue861/.gitlab-ci-local/services-output/initial-build-hello/docker:23.0.3-dind-0.log)
initial-build-hello service image: docker:23.0.3-dind healthcheck passed in 36 s
initial-build-hello service image: docker:23.0.3-dind healthcheck failed: docker:2375
initial-build-hello see (/Users/atc/tmp/firecow_gitlab-ci-local_issue861/.gitlab-ci-local/services-output/initial-build-hello/docker:23.0.3-dind-0.log)
initial-build-hello pulled docker:latest in 5.79 s
initial-build-hello $ echo "Empty step to trigger pipeline."
initial-build-hello > Empty step to trigger pipeline.
initial-build-hello finished in 55 s

PASS initial-build-hello
pipeline finished in 56 s

Second run:

Using fallback git commit data
Using fallback git remote data
parsing and downloads finished in 72 ms
initial-build-hello starting docker:latest (build)
initial-build-hello copied to docker volumes in 785 ms
initial-build-hello started service image: docker:23.0.3-dind with aliases: docker in 488 ms
initial-build-hello > still running...
initial-build-hello > still running...
initial-build-hello > still running...
initial-build-hello service image: docker:23.0.3-dind healthcheck failed: docker:2376
initial-build-hello see (/Users/atc/tmp/firecow_gitlab-ci-local_issue861/.gitlab-ci-local/services-output/initial-build-hello/docker:23.0.3-dind-0.log)
initial-build-hello service image: docker:23.0.3-dind healthcheck passed in 31 s
initial-build-hello service image: docker:23.0.3-dind healthcheck failed: docker:2375
initial-build-hello see (/Users/atc/tmp/firecow_gitlab-ci-local_issue861/.gitlab-ci-local/services-output/initial-build-hello/docker:23.0.3-dind-0.log)
initial-build-hello $ echo "Empty step to trigger pipeline."
initial-build-hello > Empty step to trigger pipeline.
initial-build-hello finished in 33 s

PASS initial-build-hello
pipeline finished in 34 s

Contents of .gitlab-ci-local/services-output/initial-build-hello/docker:23.0.3-dind-0.log

stdout

/certs/server/cert.pem: OK
/certs/client/cert.pem: OK

stderr

Certificate request self-signature ok
subject=CN = docker:dind server
Certificate request self-signature ok
subject=CN = docker:dind client
ip: can't find device 'ip_tables'
modprobe: can't change directory to '/lib/modules': No such file or directory
mount: permission denied (are you root?)
Could not mount /sys/kernel/security.
AppArmor detection and --privileged mode might break.
mount: permission denied (are you root?)

[atli-c]

It seems like gitlab CI runs services in privileged mode: https://docs.gitlab.com/ee/ci/services/#security-when-using-services-containers. So I guess gitlab-ci-local should do the same :)

[SomaGallai]

I have setup .gitlab-ci-local-env with the following configuration:

PRIVILEGED=true
ULIMIT=8000:16000
VOLUME=certs:/certs/client
VARIABLE="DOCKER_TLS_CERTDIR=/certs"

But it results in the same fail state with no error for me. But I do have an ARM based MacBook if that might have to something with it

[atli-c]

I just tried again with the same settings in .gitlab-ci-local-env

Using fallback git commit data
Using fallback git remote data
parsing and downloads finished in 50 ms
initial-build-hello starting docker:latest (build)
initial-build-hello copied to docker volumes in 731 ms
initial-build-hello started service image: docker:23.0.3-dind with aliases: docker in 562 ms
initial-build-hello service image: docker:23.0.3-dind healthcheck passed in 5.73 s
initial-build-hello $ echo "Empty step to trigger pipeline."
initial-build-hello > Empty step to trigger pipeline.
initial-build-hello finished in 8.27 s

PASS initial-build-hello
pipeline finished in 8.89 s

Seems to work just fine 🤔
Docker Desktop setting: "file sharing implementation" is set to "VirtioFS"

[SomaGallai]

I am clueless what could be causing this issue and it does not make any sense at all.
I have updated "file sharing implementation" to "VirtioFS", as you suggested, but same results.

[atli-c]

What's the exit-code?

Try running with gitlab-ci-local ; echo "exit code: $?"

[SomaGallai]

@atli-c this is the following output:

$gitlab-ci-local ; echo "exit code: $?"
parsing and downloads finished in 37 ms
initial-build-hello               starting docker:latest (build)
initial-build-hello               copied to docker volumes in 500 ms

 FAIL  initial-build-hello              
pipeline finished in 727 ms
exit code: 0

[simneu]

@SomaGallai weirdly I seem to have the exact same behaviour as you encountered.
My config was working fine for me as I was currently working on it, and suddenly got the same issue as you.
Running on ARM M1 Mac too, Docker Desktop 4.19.0 (106363) too, same .gitlab-ci-local-env too, tried to cleanup everything too ...

I have the same output with your test file:

$gitlab-ci-local ; echo "exit code: $?"
Using fallback git commit data
Using fallback git remote data
parsing and downloads finished in 37 ms
initial-build-hello starting docker:latest (build)
initial-build-hello copied to docker volumes in 485 ms

 FAIL  initial-build-hello
pipeline finished in 658 ms
exit code: 0

It seems to be related to dind, because as soon as I remove those two lines from the .gitlab-ci.yml file, build passes:

  services:
    - docker:23.0.3-dind

Still don't have any solutions for now, will continue investigating on my side.

[simneu]

@SomaGallai I seem to have found a "workaround" (still no idea where it actually comes from, but if you are blocked and have no other way to go around it, you can maybe try this). Warning, this can potentially delete a ton of unused containers, images, cache ... so use it only if you're sure you can loose everything in your Docker:
docker system prune -a

For me, after running this, it started to build again, so there was probably something somewhere in my Docker breaking the dind build (note that I tried rebooting, downgrading Docker Desktop ... but none of those helped).

If you need me to perform some tests and try some stuff to reproduce and try to figure out the actual issue, I'm staying available.