On tls issues with the certificate - token service returns empty token without any error or log record

[dee-kryvenko]

Expected behavior and actual behavior:
In my case, certificate I was using seems not to be compatible with current Harbor code (which is a separate issue).

If I were to pull an image:

# crictl --debug pull harbor.example.com/hub.docker.com/jenkins/jenkins:lts
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:harbor.example.com/hub.docker.com/jenkins/jenkins:lts,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
E0303 20:16:54.430697 1755705 remote_image.go:167] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"harbor.example.com/hub.docker.com/jenkins/jenkins:lts\": failed to resolve reference \"harbor.example.com/hub.docker.com/jenkins/jenkins:lts\": failed to authorize: failed to fetch anonymous token: authorization server did not include a token in the response" image="harbor.example.com/hub.docker.com/jenkins/jenkins:lts"
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "harbor.example.com/hub.docker.com/jenkins/jenkins:lts": failed to resolve reference "harbor.example.com/hub.docker.com/jenkins/jenkins:lts": failed to authorize: failed to fetch anonymous token: authorization server did not include a token in the response

With corresponding logs on the Core side:

2024-03-03T20:18:41Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /service/token?scope=repository%3Ahub.docker.com%2Fjenkins%2Fjenkins%3Apull&service=harbor-registry
2024-03-03T20:18:41Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="1fe703fb-ffa5-4fdc-a527-9a25e6361801"]: an unauthorized security context generated for request GET /service/token
2024-03-03T20:18:41Z [DEBUG] [/core/service/token/token.go:37]: URL for token request: /service/token?scope=repository%3Ahub.docker.com%2Fjenkins%2Fjenkins%3Apull&service=harbor-registry
2024-03-03T20:18:41Z [DEBUG] [/core/service/token/creator.go:201]: scopes: [repository:hub.docker.com/jenkins/jenkins:pull]
2024-03-03T20:18:41Z [DEBUG] [/core/service/token/authutils.go:51]: scopes: [repository:hub.docker.com/jenkins/jenkins:pull]
2024/03/03 20:18:41 Model:
2024/03/03 20:18:41 r.r: sub, obj, act
2024/03/03 20:18:41 p.p: sub, obj, act, eft
2024/03/03 20:18:41 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
2024/03/03 20:18:41 m.m: g(r_sub, p_sub) && keyMatch2(r_obj, p_obj) && (r_act == p_act || p_act == '*')
2024/03/03 20:18:41 g.g: _, _
2024/03/03 20:18:41 Policy:
2024/03/03 20:18:41 p: sub, obj, act, eft: [[anonymous /project/2 read allow] [anonymous /project/2/label read allow] [anonymous /project/2/label list allow] [anonymous /project/2/repository list allow] [anonymous /project/2/repository pull allow] [anonymous /project/2/scan read allow] [anonymous /project/2/scanner read allow] [anonymous /project/2/tag list allow] [anonymous /project/2/artifact read allow] [anonymous /project/2/artifact list allow] [anonymous /project/2/artifact-addition read allow] [anonymous /project/2/accessory list allow]]
2024/03/03 20:18:41 g: _, _: []
2024/03/03 20:18:41 Role links for: g
2024/03/03 20:18:41
2024/03/03 20:18:41 Request: anonymous, /project/2/repository, delete ---> false
2024/03/03 20:18:41 Request: anonymous, /project/2/repository, scanner-pull ---> false
2024/03/03 20:18:41 Request: anonymous, /project/2/repository, pull ---> true
2024-03-03T20:18:41Z [DEBUG] [/core/service/token/authutils.go:102]: user: , access: &{repository  hub.docker.com/jenkins/jenkins [pull]}

Steps to reproduce the problem:
I recon there are multiple ways, but in my case I was using a following CA:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-tls-ca
spec:
  isCA: true
  commonName: internal-tls-ca
  secretName: internal-tls-ca
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: self-signed
    kind: ClusterIssuer
    group: cert-manager.io

I'm guessing ECDSA algorithm is not currently supported.

Versions:
Please specify the versions of following systems.

• harbor version: [v2.10.0]
• harbor chart version: [v1.14.0]

Additional context:

CA secret from is given to .core.secretName.

Thread on slack: https://cloud-native.slack.com/archives/CC1E09J6S/p1709497679881159

[MinerYang]

from the debug msg in harbor-core, the user is empty ""

[DEBUG] [/core/service/token/authutils.go:102]: user: , access: &{repository  hub.docker.com/jenkins/jenkins [pull]}

thus the returned token is actually unauthoirzed to your repository. Have your properly save your credential file when using crictl?

harbor.example.com/hub.docker.com/jenkins/jenkins:lts": failed to authorize: failed to fetch anonymous token: authorization server did not include a token in the response

Could you provide your full service token request when pulling?

[dee-kryvenko]

This is a public repository and there is no user. I am trying to pull as anonymous. In slack, I left some more logs on what happens when I try nerdctl login, it is not any less ambiguous error stemming from the same problem.

[dee-kryvenko]

Plus the log literally tells you Request: anonymous, /project/2/repository, pull ---> true

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[dee-kryvenko]

Not stale

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.