Feature (What you would like to be added):

Upgrade kustomize to v5.0.0+, and fix the behavior of the eschewed feature dropped due to kubernetes-sigs/kustomize#4731.

Motivation (Why is this needed?):

#748 bumps g/g version from 1.76.0 to 1.86.0, which brought with it unintended changes. Etcd-druid e2e test uses skaffold with kustomize builds, to deploy the infra job which creates/deletes the object storage buckets on the configured providers. The skaffold version used is v1.38.0, which uses an already existing kustomize binary in the PATH if it exists, or else uses the kustomize provided by the kubectl binary present in the PATH.

g/g 1.76.0 used kubectl v1.24.x, which vendors kustomize v4. g/g 1.86.0 uses kubectl v1.28.x, which vendors kustomize v5. Since the make target does not explicitly install kustomize with a specified version as a prerequisite, it forces skaffold to use the kubectl's kustomize version. All these days, make test-e2e in druid faced no issue since the kubectl version installed was 1.24.x, which used kustomize v4. Hence all prow jobs were passing as well. But with this particular PR, kubectl version gets changed to v1.28.x, which uses kustomize v5.

Now, the e2e test infra overlays in druid make use of the eschewed feature mentioned in kubernetes-sigs/kustomize#4731, so that we can only define the env var keys in a file and pull their values from the env vars set in runtime session where the tests are run, such as aws.env and used to populate the infra secret like this. This was possible up till kustomize v4. However with kustomize v5, this functionality has been removed, forcing users to set the env file with the key=value pairs mandatorily.

#748 overcomes this limitation by installing kustomize v4 binary when running e2e tests, thus forcing skaffold to use this binary. But going forward, we need to be able to update to the latest versions of kustomize for various reasons, and will need to find a way to set the fully formed env var pairs which can be used by kustomize v5.

Approach/Hint to the implement solution (optional):

Add new directories hack/e2e-test/infrastructure/overlays/<provider>/common/env for each provider, and ensure they are not checked in, via .gitignore. In the e2e test script which is used to setup and clean up the infra via the skaffold calls, add logic to populate and store a <provider>.env within the abovementioned directory, and have the kustomization secretGenerator use that file instead.