bug: ECR login fails for cross region registries

[dschaaff]

I upgraded our clusters to the v2 GA this morning. The process went smoothly apart from 1 issue. We use ECR to store helm charts. These ECR hosted charts are in a different region then the k8s cluster where flux’s source controller runs. After updating to v2 we started getting an error

source-controller-5df98bbf86-w44dj manager time="2023-07-05T18:39:11Z" level=info msg="Error logging in to endpoint, trying next endpoint" error="login attempt to https://960048260646.dkr.ecr.us-west-2.amazonaws.com/v2/ failed with status: 400 Bad Request"
source-controller-5df98bbf86-w44dj manager {"level":"error","ts":"2023-07-05T18:39:11.020Z","msg":"failed to login to OCI registry: login attempt to https://960048260646.dkr.ecr.us-west-2.amazonaws.com/v2/ failed with status: 400 Bad Request","name":"flux-system-aggregates-service-preview","namespace":"flux-system","reconciler kind":"HelmChart","annotations":null,"error":"AuthenticationFailed","stacktrace":"github.com/fluxcd/pkg/runtime/events.(*Recorder).AnnotatedEventf\n\tgithub.com/fluxcd/pkg/runtime@v0.40.0/events/recorder.go:137\ngithub.com/fluxcd/pkg/runtime/events.(*Recorder).Eventf\n\tgithub.com/fluxcd/pkg/runtime@v0.40.0/events/recorder.go:114\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.RecordContextualError\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:48\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:193\ngithub.com/fluxcd/source-controller/internal/controller.(*HelmChartReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/internal/controller/helmchart_controller.go:230\ngithub.com/fluxcd/source-controller/internal/controller.(*HelmChartReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/internal/controller/helmchart_controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

If I add the environment variable AWS_REGION=us-west-2 to the source controller then the error goes away and the login succeeds.

This behavior did not occur with cross region registries on version 0.41.2.

[somtochiama]

I think this was introduced by this change here: fluxcd/pkg#560
Taking a look 🙇🏾‍♀️