error building image: error building stage: failed to get filesystem from image: stream error: stream ID 3; PROTOCOL_ERROR

[ijaouani]

Actual behavior
kaniko breaks from time to time with this error:

error building image: error building stage: failed to get filesystem from image: stream error: stream ID 3; PROTOCOL_ERROR

I don't have find a way to systematically reproduice it.
Context : I use kaniko with GitLab CI (GitLab Runner).

Expected behavior

Log when it's working:

DEBU[0011] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/rdma false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/perf_event false} {/dev/mqueue false} {/builds false} {/busybox false} {/dev/termination-log false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/var/run/secrets/kubernetes.io/serviceaccount false}] 
DEBU[0011] Not adding /dev because it is ignored        
DEBU[0011] Not adding /etc/hostname because it is ignored 
DEBU[0011] Not adding /etc/resolv.conf because it is ignored 
DEBU[0011] Not adding /proc because it is ignored       
DEBU[0011] Not adding /sys because it is ignored        
DEBU[0013] Not adding /var/run because it is ignored    

Log when it's not working:

DEBU[0010] optimize: cache key for command RUN npm run build ffa994c48d8ca1e707d097d4468b3ab8727b27a1b44d3945dee2b81d9c329d0b 
INFO[0010] Unpacking rootfs as cmd COPY portal/package*.json portal/.npmrc ./ requires it. 
DEBU[0010] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/rdma false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/perf_event false} {/dev/mqueue false} {/builds false} {/busybox false} {/dev/termination-log false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/var/run/secrets/kubernetes.io/serviceaccount false}] 
DEBU[0010] Not adding /dev because it is ignored        
DEBU[0077] Not adding /etc/hostname because it is ignored 
DEBU[0077] Not adding /etc/resolv.conf because it is ignored 
error building image: error building stage: failed to get filesystem from image: stream error: stream ID 3; PROTOCOL_ERROR
ERROR: Job failed: command terminated with exit code 1

To Reproduce

$ echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
$ /kaniko/executor --verbosity=debug --context $CI_PROJECT_DIR/client --dockerfile $CI_PROJECT_DIR/client/$IFCHOR_APP/Dockerfile --destination $CI_REGISTRY_IMAGE/$APP:$TAG

Additional Information
kaniko version : i use the docker image gcr.io/kaniko-project/executor:debug
GitLab Runner version 13.1.1
Runner host inside kubernetes 1.17.7 in Azure Kubernetes Service.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[feifeixiyu]

also see this error from time to time, using jenkins + kubernetes plugin

[mayocream]

Same problem running Jenkins + K8s plugin.

ref:

• x/net/http2: seeing INTERNAL_ERROR golang/go#26338
• stream error: stream ID 19; INTERNAL_ERROR willnorris/imageproxy#156
• storage: multipart: stream error: stream ID 1; INTERNAL_ERROR googleapis/google-cloud-go#987
• [storage] "stream error: stream ID x; INTERNAL_ERROR" googleapis/google-cloud-go#784

[wsszh]

Same problem running tekton...

[wowq]

Any idea for this?

[LPX-E5BD8]

Any idea for this?

try to reset GOPROXY

export GOPROXY=""

[toby1991]

same error

[zhonghuali]

same error

[toby1991]

same error

Seems like docker hub pull frequency issue，when I change the registry to private， this problem not show again!!

[aaron-prindle]

Can anyone in the thread here confirm:

1. If this is still an issue in the latest version of Kaniko, v1.12.1
2. If the suggested workaround of setting GOPROXY="" works to resolve this issue - error building image: error building stage: failed to get filesystem from image: stream error: stream ID 3; PROTOCOL_ERROR #1360 (comment)
3. If this is related to docker throttling as mentioned in the comment above -
   error building image: error building stage: failed to get filesystem from image: stream error: stream ID 3; PROTOCOL_ERROR #1360 (comment)

Trying to understand if this issue is still relevant, if this is a kaniko issue, and how to address the friction here - possibly a fix (depending on root cause), a better error message (if possible) or some documentation