Fix minio tokengen race #7985
Closed
Fix minio tokengen race #7985
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell. + ingester 0%
+ distributor 0%
+ querier 0%
+ querier/queryrange 0.1%
+ iter 0%
+ storage 0%
+ chunkenc 0%
+ logql 0%
+ loki 0% |
jeschkies
requested changes
Dec 21, 2022
| @@ -1,7 +1,8 @@ | |||
| .PHONY: loki-distributed down add-repos update-repos prepare build-latest-image | |||
|
|
|||
| IMAGE_TAG := $(shell ../../../tools/image-tag) | |||
| REGISTRY_PORT ?= $(shell k3d registry list -o json | jq -r '.[] | select(.name == "k3d-grafana") | .portMappings."5000/tcp" | .[0].HostPort') | |||
| EXISTING_REGISTRY_PORT := $(shell k3d registry list -o json | jq -r '.[] | select(.name == "k3d-grafana") | .portMappings."5000/tcp" | .[0].HostPort') | |||
| REGISTRY_PORT ?= $(or $(EXISTING_REGISTRY_PORT),46453) | |||
There was a problem hiding this comment.
That's been an issue on my side 🙂
| @@ -0,0 +1,12 @@ | |||
| # Empty Helm Cluster | |||
|
|
|||
| This is a cluster designed for testing out helm charts. It is meant for running `helm install` against. It provides the required dependencies of the helm chart and nothing else (ie. minio is installed by the helm chart). This differs from the other envs that, while also using the helm chart, use `tanka` to deploy those charts. | |||
There was a problem hiding this comment.
Suggested change
| This is a cluster designed for testing out helm charts. It is meant for running `helm install` against. It provides the required dependencies of the helm chart and nothing else (ie. minio is installed by the helm chart). This differs from the other envs that, while also using the helm chart, use `tanka` to deploy those charts. | |
| This is a cluster designed for testing out Helm charts. It is meant for running `helm install` against. It provides the required dependencies of the Helm chart and nothing else (ie. minio is installed by the Helm chart). This differs from the other envs that, while also using the helm chart, use `tanka` to deploy those charts. |
| }, | ||
|
|
||
| minioNamespace: k.core.v1.namespace.new('minio'), | ||
| minio: helm.template('minio', '../../charts/minio', { |
There was a problem hiding this comment.
Could we have a values file instead? I'd love to document the steps without some automation.
|
./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell. + ingester 0%
+ distributor 0%
+ querier 0%
+ querier/queryrange 0%
+ iter 0%
+ storage 0%
+ chunkenc 0%
+ logql 0%
+ loki 0% |
5 tasks
|
Replacing with #8064, which fixes this problem without removing minio |
mraboosk
pushed a commit
to mraboosk/loki
that referenced
this pull request
Oct 7, 2024
…rafana#8064) **What this PR does / why we need it**: This PR replaces grafana#7985. It solves the same problem, which is the race condition between the minio create bucket job and the enterprise tokengen job, but without removing minio.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Removes MinIO from the loki helm chart. This was meant as a convenience, but there's currently no good way to determine the helm run order of jobs defined by the loki helm chart and those of the subchart. As a result, when using minio, it can often occur that the make buckets job doesn't run before the tokengen, which will cause the whole deployment to fail as the provisioner depends on the admin token generated in the tokengen job.
I think it's reasonable to expect people to bring their own object storage.
Other possible solutions
I'm not sure how helm hook weights work between charts and their dependencies. We may be able to solve this with hook weights, and is worth testing before merging this.
TODO:
Special notes for your reviewer:
This is built on #7984 and includes changes from that PR.
Checklist
CONTRIBUTING.mdguideCHANGELOG.mdupdateddocs/sources/upgrading/_index.md