bug: problems with bearer authentication

[rchincha]

Discussed in #2089

^{Originally posted by nyabla November 29, 2023}
versions I tried on: 1.4.3 and 2.0.0-rc7

my config file: config.json

background: i am trying to run an instance of zot with authentication handled by authentik. i have a sort of proxy in front of the token endpoint of authentik in order to translate the GET token request into a POST request. The clients I tried (docker cli and podman) are both able to obtain a valid token from this endpoint.

problem: zot gives a 500 Internal Server Error response upon receiving a request with a (valid) bearer token ONLY when the service key under bearer is non-empty. if the service key has an empty string ("") then podman/skopeo get confused.

what i tried: making sure that the value of the aud key in the jwt matches the value of service, similarly to the auth.docker.io endpoint.

[nyabla]

logs with podman client and service set to the same as aud in the jwt. zot 1.4.3

{"level":"info","params":{"distSpecVersion":"1.0.1-dev","GoVersion":"go1.19.3","Commit":"v1.4.3-69f0cf6bb4727884af42f38c92858fdb114104de","ReleaseTag":"v1.4.3","BinaryType":"-sync-search-scrub-metrics-lint","AccessControl":null,"Storage":{"RootDirectory":"/tmp/zot","Dedupe":true,"RemoteCache":false,"GC":true,"Commit":false,"GCDelay":3600000000000,"GCInterval":0,"StorageDriver":null,"CacheDriver":null,"SubPaths":null},"HTTP":{"Address":"0.0.0.0","Port":"5000","AllowOrigin":"","TLS":null,"Auth":{"FailDelay":0,"HTPasswd":{"Path":""},"LDAP":null,"Bearer":{"Realm":"http://localhost:8080/token","Service":"zot.local","Cert":"/etc/zot/auth.crt"}},"RawAccessControl":null,"Realm":"","Ratelimit":null},"Log":{"Level":"debug","Output":"","Audit":""},"Extensions":null},"goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:119","time":"2023-11-30T12:10:38.584005867Z","message":"configuration settings"}
{"level":"info","cpus":4,"max. open files":1048576,"listen backlog":"4096","max. inotify watches":"65536","goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:110","time":"2023-11-30T12:10:38.5846984Z","message":"runtime params"}
{"level":"warn","goroutine":1,"caller":"zotregistry.io/zot/pkg/debug/swagger/swagger_disabled.go:22","time":"2023-11-30T12:10:38.590058387Z","message":"skipping enabling swagger because given zot binary doesn't include this feature, please build a binary that does so"}
{"level":"info","module":"http","clientIP":"172.22.0.1:49190","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":283,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.23.1 (github.com/containers/image)"]},"goroutine":51,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:12:20.421826473Z","message":"HTTP API"}
{"level":"error","goroutine":52,"caller":"zotregistry.io/zot/pkg/log/log.go:25","time":"2023-11-30T12:12:21.403685258Z","message":"panic recovered"}
{"level":"info","module":"http","clientIP":"172.22.0.1:43514","method":"GET","path":"/v2/","statusCode":500,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Connection":["close"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.23.1 (github.com/containers/image)"]},"goroutine":52,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:12:21.403825678Z","message":"HTTP API"}

same but with docker cli client

{"level":"info","params":{"distSpecVersion":"1.0.1-dev","GoVersion":"go1.19.3","Commit":"v1.4.3-69f0cf6bb4727884af42f38c92858fdb114104de","ReleaseTag":"v1.4.3","BinaryType":"-sync-search-scrub-metrics-lint","AccessControl":null,"Storage":{"RootDirectory":"/tmp/zot","Dedupe":true,"RemoteCache":false,"GC":true,"Commit":false,"GCDelay":3600000000000,"GCInterval":0,"StorageDriver":null,"CacheDriver":null,"SubPaths":null},"HTTP":{"Address":"0.0.0.0","Port":"5000","AllowOrigin":"","TLS":null,"Auth":{"FailDelay":0,"HTPasswd":{"Path":""},"LDAP":null,"Bearer":{"Realm":"http://localhost:8080/token","Service":"zot.local","Cert":"/etc/zot/auth.crt"}},"RawAccessControl":null,"Realm":"","Ratelimit":null},"Log":{"Level":"debug","Output":"","Audit":""},"Extensions":null},"goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:119","time":"2023-11-30T12:16:46.390232659Z","message":"configuration settings"}
{"level":"info","cpus":4,"max. open files":1048576,"listen backlog":"4096","max. inotify watches":"65536","goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:110","time":"2023-11-30T12:16:46.390424799Z","message":"runtime params"}
{"level":"warn","goroutine":1,"caller":"zotregistry.io/zot/pkg/debug/swagger/swagger_disabled.go:22","time":"2023-11-30T12:16:46.39296521Z","message":"skipping enabling swagger because given zot binary doesn't include this feature, please build a binary that does so"}
{"level":"info","module":"http","clientIP":"172.20.0.1:50076","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":283,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/6.1.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"]},"goroutine":23,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:17:31.927995452Z","message":"HTTP API"}
{"level":"error","goroutine":25,"caller":"zotregistry.io/zot/pkg/log/log.go:25","time":"2023-11-30T12:17:32.711874692Z","message":"panic recovered"}
{"level":"info","module":"http","clientIP":"172.20.0.1:50086","method":"GET","path":"/v2/","statusCode":500,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Connection":["close"],"User-Agent":["docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/6.1.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"]},"goroutine":25,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:17:32.712075996Z","message":"HTTP API"}

docker client with empty service. i havent included a log from podman because it can't handle empty service.

{"level":"info","params":{"distSpecVersion":"1.0.1-dev","GoVersion":"go1.19.3","Commit":"v1.4.3-69f0cf6bb4727884af42f38c92858fdb114104de","ReleaseTag":"v1.4.3","BinaryType":"-sync-search-scrub-metrics-lint","AccessControl":null,"Storage":{"RootDirectory":"/tmp/zot","Dedupe":true,"RemoteCache":false,"GC":true,"Commit":false,"GCDelay":3600000000000,"GCInterval":0,"StorageDriver":null,"CacheDriver":null,"SubPaths":null},"HTTP":{"Address":"0.0.0.0","Port":"5000","AllowOrigin":"","TLS":null,"Auth":{"FailDelay":0,"HTPasswd":{"Path":""},"LDAP":null,"Bearer":{"Realm":"http://localhost:8080/token","Service":"","Cert":"/etc/zot/auth.crt"}},"RawAccessControl":null,"Realm":"","Ratelimit":null},"Log":{"Level":"debug","Output":"","Audit":""},"Extensions":null},"goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:119","time":"2023-11-30T12:20:00.097669041Z","message":"configuration settings"}
{"level":"info","cpus":4,"max. open files":1048576,"listen backlog":"4096","max. inotify watches":"65536","goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:110","time":"2023-11-30T12:20:00.097871564Z","message":"runtime params"}
{"level":"warn","goroutine":1,"caller":"zotregistry.io/zot/pkg/debug/swagger/swagger_disabled.go:22","time":"2023-11-30T12:20:00.101151848Z","message":"skipping enabling swagger because given zot binary doesn't include this feature, please build a binary that does so"}
{"level":"info","module":"http","clientIP":"172.22.0.1:52258","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/6.1.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"]},"goroutine":19,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:20:16.185581632Z","message":"HTTP API"}
{"level":"info","module":"http","clientIP":"172.22.0.1:52262","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/6.1.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"]},"goroutine":11,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:20:16.186862918Z","message":"HTTP API"}

[nyabla]

access claim was missing in jwt. feel free to close

[rchincha]

@peusebiu can you pls check if we could handle/fail this more gracefully.

{"level":"info","module":"http","clientIP":"172.22.0.1:49190","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":283,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.23.1 (github.com/containers/image)"]},"goroutine":51,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:12:20.421826473Z","message":"HTTP API"}
{"level":"error","goroutine":52,"caller":"zotregistry.io/zot/pkg/log/log.go:25","time":"2023-11-30T12:12:21.403685258Z","message":"panic recovered"}

[rchincha]

@nyabla can you give us the exact steps both on authentik and zot so that we can reproduce this panic? Understood that this is an operator error, but still in our opinion best if we handle this.

[eusebiu-constantin-petu-dbk]

@nyabla I tried to reproduce but I can't configure authentik, mainly I can not do the initial setup, it's prompting me to login with user and pass, instead of a prompt with setting up the admin account.

Tried to add a username and password in .env config but same issue.

[rchincha]

Closing this, pls re-open if needed.