Even when explicitly passing https, registry requests to localhost are downgraded to http

[apostasie]

Description

nerdctl login https://127.0.0.1:5000 will always be downgraded to http://127.0.0.1:5000

It seems to be because of the faulty logic in remotes/docker/config/hosts.go that apparently considers "the scheme is ambiguous" as the port is non-standard and the host is localhost.

Whether they fixed it for 2.0 is unclear to me at this point.

Steps to reproduce the issue

1. nerdctl login https://127.0.0.1:5000

Describe the results you received and expected

Seeing: request downgraded to http://127.0.0.1:5000

Should see: not downgraded, the scheme is explicit

What version of nerdctl are you using?

1.7.6 / main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

I believe we should rewrite / cleanup our whole login/logout workflow.
Specifically:

• we should clearly isolate docker stuff in a separate package (auth file manipulation)
• remove duplication
• we keep parsing the address (or extracting the host part of it) - we should instead parse it once into a *url.URL, normalize what has to be, and carry that around instead of dropping the scheme and passing half-complete strings around
• we should also explicitly set the port if ommitted (to simplify hosts.toml resolution - see hosts.toml file resolution does not seem to account for default port being ommitted #3047 for eg).

Overall, the whole Docker approach "downgrade to http if XYZ" has always been a problem since 2014 (stemming from a deeply flawed understanding of TLS security model) and we should clean up the remnants of it so that we have a clear, documented path for it that we can reason about and that makes sense.

[apostasie]

In light of #3052, this ticket should be more about "allowing the user to disable insecure-registry for localhost", which does not seem possible right now.

[Shikachuu]

I believe we should rewrite / cleanup our whole login/logout workflow. Specifically:

* we should clearly isolate docker stuff in a separate package (auth file manipulation)

* remove duplication

* we keep parsing the address (or extracting the host part of it) - we should instead parse it _once_ into a `*url.URL`, normalize what has to be, and carry that around instead of dropping the scheme and passing half-complete strings around

* we should also explicitly set the port if ommitted (to simplify `hosts.toml` resolution - see [hosts.toml file resolution does not seem to account for default port being ommitted #3047](https://github.com/containerd/nerdctl/issues/3047) for eg).

Overall, the whole Docker approach "downgrade to http if XYZ" has always been a problem since 2014 (stemming from a deeply flawed understanding of TLS security model) and we should clean up the remnants of it so that we have a clear, documented path for it that we can reason about and that makes sense.

Let's say we start to use what we already have which is converToHostname it's called in every loginClientSide and GetDefaultAuthConfig which should be fine at least to my understanding.

We already have an URL convert there so it seems like we should to the following:

• parse the schema correctly (without magic string specific up/downgrades)
• parse the ports correctly
• parse the path correctly

After this we should make our own ConfigureHosts() function which we can call on non docker situations (but please correct me if I'm wrong here), since this problem will not only occure in login, but in push too and there are some imgutil refferences too.

(Also it's seems kinda obvious from the comment but I'm willin' to fix/implement all of this. :D)

[apostasie]

@Shikachuu it would definitely be nice to get some help here!

Here is the master ticket for all things impacting auth: #3072 (watch-out, it is a mouthful :D).

About your comment, yep, it could be a nice improvement already to start with that:

• we take the url argument from cobra (note that we allow the user to omit the scheme)
• if it has no scheme, add a "https" scheme (that is the desired behavior - and this is what convertToHostname does)
• now, we can url.Parse it
• then update all downstream methods so that they take the *url.URL instead of a string
• let downstream methods decide how to downgrade / upgrade

I do have a branch with a lot of refactoring on these, but I never finished it and it is now very stale - also, quite busy with other stuff.

If you are interested in having a go at all of this, that would be fantastic! Happy to help you out with review / comments if you need.

Beware though, this is much more complex that you might think (re: hosts.toml files lookup, docker auth cred, etc).

Unrelated fun fact: a long, long time ago, I was friends with the Prezi crew - and I had a wonderful time in Budapest :-).

[apostasie]

Here are some notes from my old stuff. You might want to have a look at the specs for a better understanding of the hosts toml behavior.

"
nerdctl commands involving registry network communication (login, pull, push):

allow a variety of url formats, with or without an explicit scheme or port
will read hosts.toml configuration from up to three locations
will retrieve credentials from docker configuration, again allowing to retrieve from different keys
and might try a variety of final urls dependent on whether the corresponding registry is flagged as insecure (either from command-line or through hosts.toml configuration)
Notes that successful communication but failure to authenticate will NOT trigger a dowgrade of the protocol in the presence of insecure.

Furthermore, nerdctl currently treats localhost hosted registries as insecure by default, although this behavior is controversial and Docker advises against trusting it:

"Local registries, whose IP address falls in the 127.0.0.0/8 range, are automatically marked as insecure as of Docker 1.3.2. It isn't recommended to rely on this, as it may change in the future."

The specification for hosts.toml is here: https://github.com/containerd/containerd/blob/main/docs/hosts.md

The document for docker certificates layout is here: https://docs.docker.com/reference/cli/dockerd/#insecure-registries

"