Kaniko build seems to hang on "Retrieving image" in GKE

[yacn]

Actual behavior

When trying to build a container in GKE using Kaniko the run seems to just hang / stop doing anything after the "Retrieving image" log line. The original Dockerfile I tried to use was a multi-stage build, but I can not get even a minimal Dockerfile to build. The issue seems specific to kubernetes/GKE as I could build the original Dockerfile without issue locally with Docker.

Things I've tried:

• using the --registry-mirror flag to pull the library/ images from mirror.gcr.io as well as hosting the images in our own GCR project (thinking being that docker.io was perhaps blocking/limiting us).
• adding resource requests as well with no change in behavior.
• using various kaniko container versions.

Since this happens with every kaniko container I tried there is likely something I'm missing in my pod's YAML or GKE cluster config or something but for the life of me I cannot figure out what.

Expected behavior
The executor should build the container as expected

To Reproduce
Steps to reproduce the behavior:

1. kubectl apply -f kaniko.yml
2. kubectl exec -it kaniko-tst -- sh
3. executor --dockerfile=Dockerfile_min --context=git://<internal_git>/<team>/<repo>.git --git=branch=wip-deployment-debug --no-push --verbosity=trace

Additional Information

• logs:

DEBU[0000] Getting source context from git://<internal_git>/<team>/<repo>.git
DEBU[0000] Getting source from reference refs/heads/wip-deployment-debug
Enumerating objects: 81, done.
Counting objects: 100% (37/37), done.
Compressing objects: 100% (25/25), done.
Total 81 (delta 12), reused 30 (delta 7), pack-reused 44
DEBU[0000] Build context located at /kaniko/buildcontext/
DEBU[0000] Copying file /kaniko/buildcontext/Dockerfile_min to /kaniko/Dockerfile
TRAC[0000] Adding /var/run to initialIgnoreList
DEBU[0000] Skip resolving path /kaniko/Dockerfile
DEBU[0000] Skip resolving path /kaniko/buildcontext/
DEBU[0000] Skip resolving path /cache
DEBU[0000] Skip resolving path
DEBU[0000] Skip resolving path
DEBU[0000] Skip resolving path
DEBU[0000] Skip resolving path
DEBU[0000] Built stage name to index map: map[]
INFO[0000] Retrieving image manifest gcr.io/<project>/alpine:3.14
INFO[0000] Retrieving image gcr.io/<project>/alpine:3.14 from registry gcr.io
E1207 19:05:19.685454      14 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

nothing else logs after that, even leaving it to run for upwards of an hour or overnight

• kaniko.yml

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-tst
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:v1.7.0-debug
    command: ['/busybox/sleep']
    args: ['infinity']
    volumeMounts:
    - name: gce-creds
      mountPath: /secret
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /secret/key.json
    - name: GIT_TOKEN
      valueFrom:
        secretKeyRef:
          name: github-auth-token
          key: token
  restartPolicy: Never
  volumes:
  - name: gce-creds
    secret:
      secretName: jenkins-gce-key

• Dockerfile

FROM gcr.io/<project>/alpine:3.14

RUN echo "foo"

• Build Context
  no build context needed besides Dockerfile
• Kaniko Image (fully qualified with digest)
  happens with every Kaniko container I've tried (latest, debug, v1.7.0-debug, v1.6.0-debug, v1.5.0-debug)
• GKE cluster version
  1.18.20-gke.901

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile