error parsing HTTP 401 response body: unexpected end of JSON input: ""

[ipanova]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

$ podman tag 6dbb9cc54074 namespace/i-have-no-perms-to
$ podman push  namespace/i-have-no-perms-to --log-level=debug
Getting image source signatures

DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/namespace/i-have-no-perms-to/blobs/sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Anamespace%2Fi-have-no-perms-to%3Apull%2Cpush&service=registry.docker.io 
DEBU[0001] HEAD https://registry-1.docker.io/v2/namespace/i-have-no-perms-to/blobs/sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 
DEBU[0001] ... not authorized                           
Error: error copying image to the remote destination: Error trying to reuse blob sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 at destination: Error checking whether a blob sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 exists in docker.io/namespace/i-have-no-perms-to: errors:
denied: requested access to the resource is denied
error parsing HTTP 401 response body: unexpected end of JSON input: ""

Describe the results you received:
error parsing HTTP 401 response body: unexpected end of JSON input: ""

Describe the results you expected:
HEAD request is issued to this endpoint https://registry-1.docker.io/v2/namespace/i-have-no-perms-to/blobs/sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116, which obviously does not contain response body. Client should be able to handle that and finish push operation without parsing errors

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.14.10
Built:        Tue Dec  8 14:37:43 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.26-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 74feb474bea20d593dfb70e1e244893b9eb6b619'
  cpus: 2
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: example.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.6.6-300.fc32.x86_64
  linkmode: dynamic
  memFree: 1871626240
  memTotal: 4118777856
  ociRuntime:
    name: crun
    package: crun-0.17-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.2
  swapFree: 0
  swapTotal: 0
  uptime: 34m 35.25s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/vagrant/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/vagrant/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 15
  runRoot: /run/user/1000/containers
  volumePath: /home/vagrant/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438263
  BuiltTime: Tue Dec  8 14:37:43 2020
  GitCommit: ""
  GoVersion: go1.14.10
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.2.1-1.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

Can you retest with a more recent Podman - 3.1.2, ideally?

…

On Wed, May 12, 2021 at 11:27 AM Ina Panova ***@***.***> wrote: *Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)* /kind bug *Description* $ podman tag 6dbb9cc54074 namespace/i-have-no-perms-to $ podman push namespace/i-have-no-perms-to --log-level=debug Getting image source signatures DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] DEBU[0000] ... will first try using the original manifest unmodified DEBU[0000] Checking /v2/namespace/i-have-no-perms-to/blobs/sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 DEBU[0000] GET https://registry-1.docker.io/v2/ DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Anamespace%2Fi-have-no-perms-to%3Apull%2Cpush&service=registry.docker.io DEBU[0001] HEAD https://registry-1.docker.io/v2/namespace/i-have-no-perms-to/blobs/sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 DEBU[0001] ... not authorized Error: error copying image to the remote destination: Error trying to reuse blob sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 at destination: Error checking whether a blob sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 exists in docker.io/namespace/i-have-no-perms-to: errors: denied: requested access to the resource is denied error parsing HTTP 401 response body: unexpected end of JSON input: "" *Describe the results you received:* error parsing HTTP 401 response body: unexpected end of JSON input: "" *Describe the results you expected:* HEAD request is issued to this endpoint https://registry-1.docker.io/v2/namespace/i-have-no-perms-to/blobs/sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116, which obviously does not contain response body. Client should be able to handle that and finish push operation without parsing errors *Additional information you deem important (e.g. issue happens only occasionally):* *Output of podman version:* $ podman version Version: 2.2.1 API Version: 2.1.0 Go Version: go1.14.10 Built: Tue Dec 8 14:37:43 2020 OS/Arch: linux/amd64 *Output of podman info --debug:* $ podman info --debug host: arch: amd64 buildahVersion: 1.18.0 cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.0.26-1.fc32.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.26, commit: 74feb474bea20d593dfb70e1e244893b9eb6b619' cpus: 2 distribution: distribution: fedora version: "32" eventLogger: journald hostname: example.com idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 5.6.6-300.fc32.x86_64 linkmode: dynamic memFree: 1871626240 memTotal: 4118777856 ociRuntime: name: crun package: crun-0.17-1.fc32.x86_64 path: /usr/bin/crun version: |- crun version 0.17 commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL os: linux remoteSocket: path: /run/user/1000/podman/podman.sock rootless: true slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.1.8-1.fc32.x86_64 version: |- slirp4netns version 1.1.8 commit: d361001f495417b880f20329121e3aa431a8f90f libslirp: 4.3.1 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.4.2 swapFree: 0 swapTotal: 0 uptime: 34m 35.25s registries: search: - registry.fedoraproject.org - registry.access.redhat.com - registry.centos.org - docker.io store: configFile: /home/vagrant/.config/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: fuse-overlayfs-1.4.0-1.fc32.x86_64 Version: |- fusermount3 version: 3.9.1 fuse-overlayfs: version 1.4 FUSE library version 3.9.1 using FUSE kernel interface version 7.31 graphRoot: /home/vagrant/.local/share/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageStore: number: 15 runRoot: /run/user/1000/containers volumePath: /home/vagrant/.local/share/containers/storage/volumes version: APIVersion: 2.1.0 Built: 1607438263 BuiltTime: Tue Dec 8 14:37:43 2020 GitCommit: "" GoVersion: go1.14.10 OsArch: linux/amd64 Version: 2.2.1 *Package info (e.g. output of rpm -q podman or apt list podman):* podman-2.2.1-1.fc32.x86_64 *Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md <https://github.com/containers/podman/blob/master/troubleshooting.md>)* Yes/No *Additional environment details (AWS, VirtualBox, physical, etc.):* — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/containers/podman/issues/10322>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCBY6XYLEM5SKHPQ7S3TNKM7BANCNFSM44Y2VXFQ> .

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[ipanova]

podman-3.3.0-0.5.dev.gited983c9.fc35.x86_64

time="2021-06-14T13:25:42+02:00" level=info msg="podman filtering at log level debug"
time="2021-06-14T13:25:42+02:00" level=debug msg="Called push.PersistentPreRunE(podman --log-level=debug push namespace/i-have-no-perms)"
time="2021-06-14T13:25:42+02:00" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\""
time="2021-06-14T13:25:42+02:00" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2021-06-14T13:25:42+02:00" level=debug msg="Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db"
time="2021-06-14T13:25:42+02:00" level=debug msg="Overriding run root \"/run/containers/storage\" with \"/var/run/containers/storage\" from database"
time="2021-06-14T13:25:42+02:00" level=debug msg="Overriding tmp dir \"/run/libpod\" with \"/var/run/libpod\" from database"
time="2021-06-14T13:25:42+02:00" level=debug msg="Using graph driver overlay"
time="2021-06-14T13:25:42+02:00" level=debug msg="Using graph root /var/lib/containers/storage"
time="2021-06-14T13:25:42+02:00" level=debug msg="Using run root /var/run/containers/storage"
time="2021-06-14T13:25:42+02:00" level=debug msg="Using static dir /var/lib/containers/storage/libpod"
time="2021-06-14T13:25:42+02:00" level=debug msg="Using tmp dir /var/run/libpod"
time="2021-06-14T13:25:42+02:00" level=debug msg="Using volume path /var/lib/containers/storage/volumes"
time="2021-06-14T13:25:42+02:00" level=debug msg="Set libpod namespace to \"\""
time="2021-06-14T13:25:42+02:00" level=debug msg="[graphdriver] trying provided driver \"overlay\""
time="2021-06-14T13:25:42+02:00" level=debug msg="cached value indicated that overlay is supported"
time="2021-06-14T13:25:42+02:00" level=debug msg="cached value indicated that metacopy is being used"
time="2021-06-14T13:25:42+02:00" level=debug msg="cached value indicated that native-diff is not being used"
time="2021-06-14T13:25:42+02:00" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
time="2021-06-14T13:25:42+02:00" level=debug msg="backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true"
time="2021-06-14T13:25:42+02:00" level=debug msg="Initializing event backend journald"
time="2021-06-14T13:25:42+02:00" level=debug msg="configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
time="2021-06-14T13:25:42+02:00" level=debug msg="configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument"
time="2021-06-14T13:25:42+02:00" level=debug msg="Using OCI runtime \"/usr/bin/crun\""
time="2021-06-14T13:25:42+02:00" level=info msg="Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist"
time="2021-06-14T13:25:42+02:00" level=debug msg="Default CNI network name podman is unchangeable"
time="2021-06-14T13:25:42+02:00" level=info msg="Setting parallel job count to 25"
time="2021-06-14T13:25:42+02:00" level=debug msg="Looking up image \"namespace/i-have-no-perms\" in local containers storage"
time="2021-06-14T13:25:42+02:00" level=debug msg="Trying \"namespace/i-have-no-perms\" ..."
time="2021-06-14T13:25:42+02:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2021-06-14T13:25:42+02:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
time="2021-06-14T13:25:42+02:00" level=debug msg="Trying \"localhost/namespace/i-have-no-perms:latest\" ..."
time="2021-06-14T13:25:42+02:00" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@ed210e3e4a5bae1237f1bb44d72a05a2f1e5c6bfe7a7e73da179e2534269c459\""
time="2021-06-14T13:25:42+02:00" level=debug msg="Found image \"namespace/i-have-no-perms\" as \"localhost/namespace/i-have-no-perms:latest\" in local containers storage"
time="2021-06-14T13:25:42+02:00" level=debug msg="Pushing image namespace/i-have-no-perms to namespace/i-have-no-perms"
time="2021-06-14T13:25:42+02:00" level=debug msg="Copying source image [overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@ed210e3e4a5bae1237f1bb44d72a05a2f1e5c6bfe7a7e73da179e2534269c459 to destination image //namespace/i-have-no-perms:latest"
time="2021-06-14T13:25:42+02:00" level=debug msg="No credentials for docker.io found"
time="2021-06-14T13:25:42+02:00" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
time="2021-06-14T13:25:42+02:00" level=debug msg=" Using \"default-docker\" configuration"
time="2021-06-14T13:25:42+02:00" level=debug msg="  Using file:///var/lib/containers/sigstore"
time="2021-06-14T13:25:42+02:00" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io"
time="2021-06-14T13:25:42+02:00" level=debug msg="Using blob info cache at /var/lib/containers/storage/cache/blob-info-cache-v1.boltdb"
time="2021-06-14T13:25:42+02:00" level=debug msg="IsRunningImageAllowed for image containers-storage:[overlay@/var/lib/containers/storage]@ed210e3e4a5bae1237f1bb44d72a05a2f1e5c6bfe7a7e73da179e2534269c459"
time="2021-06-14T13:25:42+02:00" level=debug msg=" Using default policy section"
time="2021-06-14T13:25:42+02:00" level=debug msg=" Requirement 0: allowed"
time="2021-06-14T13:25:42+02:00" level=debug msg="Overall: allowed"
Getting image source signatures
time="2021-06-14T13:25:42+02:00" level=debug msg="Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json]"
time="2021-06-14T13:25:42+02:00" level=debug msg="... will first try using the original manifest unmodified"
time="2021-06-14T13:25:42+02:00" level=debug msg="Checking /v2/namespace/i-have-no-perms/blobs/sha256:dee215ffc666313e1381d3e6e4299a4455503735b8df31c3fa161d2df50860a8"
time="2021-06-14T13:25:42+02:00" level=debug msg="GET https://registry-1.docker.io/v2/"
time="2021-06-14T13:25:43+02:00" level=debug msg="Ping https://registry-1.docker.io/v2/ status 401"
time="2021-06-14T13:25:43+02:00" level=debug msg="GET https://auth.docker.io/token?scope=repository%3Anamespace%2Fi-have-no-perms%3Apull%2Cpush&service=registry.docker.io"
time="2021-06-14T13:25:43+02:00" level=debug msg="HEAD https://registry-1.docker.io/v2/namespace/i-have-no-perms/blobs/sha256:dee215ffc666313e1381d3e6e4299a4455503735b8df31c3fa161d2df50860a8"
time="2021-06-14T13:25:44+02:00" level=debug msg="... not authorized"
time="2021-06-14T13:25:44+02:00" level=debug msg="Looking up image \"namespace/i-have-no-perms\" in local containers storage"
time="2021-06-14T13:25:44+02:00" level=debug msg="Trying \"namespace/i-have-no-perms\" ..."
time="2021-06-14T13:25:44+02:00" level=debug msg="Trying \"localhost/namespace/i-have-no-perms:latest\" ..."
time="2021-06-14T13:25:44+02:00" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@ed210e3e4a5bae1237f1bb44d72a05a2f1e5c6bfe7a7e73da179e2534269c459\""
time="2021-06-14T13:25:44+02:00" level=debug msg="Found image \"namespace/i-have-no-perms\" as \"localhost/namespace/i-have-no-perms:latest\" in local containers storage"
Error: Error trying to reuse blob sha256:dee215ffc666313e1381d3e6e4299a4455503735b8df31c3fa161d2df50860a8 at destination: Error checking whether a blob sha256:dee215ffc666313e1381d3e6e4299a4455503735b8df31c3fa161d2df50860a8 exists in docker.io/namespace/i-have-no-perms: errors:
denied: requested access to the resource is denied
error parsing HTTP 401 response body: unexpected end of JSON input: ""

[mheon]

@vrothberg This one looks like a potential c/image issue, but I think it also ties into the image library error consistency work we want to do - this should probably return some sort of consistent error message indicating we have no permission to push?

[baude]

@vrothberg wdyt?

[vrothberg]

I think this is c/image only. FWIW, the parsing error seems to relate to Docker Hub. Pushing to Quay doesn't yield it.

@mtrmac shall we move the issue over to c/image?

[mtrmac]

I would expect this to run into https://github.com/distribution/distribution/blob/01f589cf8726565aa3c5c053be12873bafedbedc/registry/client/errors.go#L127 , which is also the package where it fails. Something weird is going on.

Sure, move it — if we are going to add a work-around, it will be in c/image.

[mtrmac]

The Immediate Cause

Example, with response dump:

DEBU[0003] GET https://auth.docker.io/token?scope=repository%3Ai-have-no-perms-for-x%2Fbusyboxdoesnoexist%3Apull%2Cpush&service=registry.docker.io 
DEBU[0003] HEAD https://registry-1.docker.io/v2/i-have-no-perms-for-x/busyboxdoesnoexist/blobs/sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b 
DEBU[0004] ... not authorized                           
ERRO[0004] ===RES===
HTTP/1.1 401 Unauthorized
Connection: close
Content-Length: 183
Content-Type: application/json
Date: Wed, 07 Jul 2021 16:46:46 GMT
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=31536000
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:i-have-no-perms-for-x/busyboxdoesnoexist:pull",error="insufficient_scope"


===RES=== 
FATA[0004] Error trying to reuse blob sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b at destination: Error checking whether a blob sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b exists in docker.io/i-have-no-perms-for-x/busyboxdoesnoexist: errors:
denied: requested access to the resource is denied
error parsing HTTP 401 response body: unexpected end of JSON input: "" 

and the presence of WWW-Authenticate means we end up at https://github.com/distribution/distribution/blob/01f589cf8726565aa3c5c053be12873bafedbedc/registry/client/errors.go#L123 , not a few lines below as I have speculated above.

UI Intent

Note the errors:\ndenied: … text; the returned value is an errcode.Errors multi-error list, and this multi-valued reporting seems to be a sort of intentional design of distribution/distribution error reporting; compare containers/podman#10858 (comment) also returning a two-value list.

The counterpart to this is how the Docker caller of that, at the top level (TranslatePullError) just returns the first element (recursively); which is why no-one has probably noticed till now.

API Evolution

On the one hand the current value is clearly not useful and using “the most relevant” error would make sense, OTOH there are already callers checking for various docker/distribution typed errors (for good reason), so stripping data might break some of them.

Well, the most direct way to fix that is probably in the distribution/distribution error parser — and then update to the new repo from docker/distribution to distribution/distribution, which breaks callers’ type checks just the same. Sigh…

• If we are very strict about keeping c/image API, we can’t change any of this. Short-term Podman (libimage?) could recognize errcode.Errors and beautify that, without actually breaking the c/image API.
• Technically, nothing promises callers to return docker/distribution API types. So we can change that, OTOH that’s also pretty certain to silently break some callers that check for those types, starting with our own c/common/pkg/retry. Updating the GitHub.com/containers ecosystem for the new types is the easy part; finding out who else needs to update would be harder.
• Another alternative is to break the callers this time again, and also document the distribution/distribution types. I’m not much of a fan…
• Or break the callers this time again, explicitly document that distribution/distribution values are not an API promise (informative but not ultimately very useful) — and start defining our own error types the callers could rely on. OTOH taking on the full error set would be a non-trivially big project, especially if the intent were to hide the various registry differences.

@vrothberg , others: WDYT?

[vrothberg]

Thanks for the wonderful summary, @mtrmac!

I am in favor of moving to distribution/distribution. I just wanted to browse the repository and noticed that docker/distribution is now forwarding to distribution/distribution. I could swear this was not the case two days ago, where I believe to have seen a "this repository is archived" banner.

Moving over to distribution/distribution and updating the containers ecosystem to me sounds like a good way forward. We should also document that the values are not an API promise. If needed we could selectively add some errors to c/image that would give promises.

[mtrmac]

FWIW #1299 will follow Docker in only reporting the first error from a multi-error list.

[ttys3]

I got lots of this very day with podman:

denied: requested access to the resource is denied
error parsing HTTP 401 response body: unexpected end of JSON input: ""

but I do have permission. retry again it works

the registry server is harbor registry

My colleagues use docker does not have this issue