e2e: refactor metrics test to use NSD and WI #19022
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shoenig
force-pushed
the
e2e-metrics-metrics-metrics
branch
from
1d8b813 to
a00df18
Compare
shoenig
force-pushed
the
e2e-metrics-metrics-metrics
branch
from
a00df18 to
39f2033
Compare
|
Spot check against e2e |
lgfa29
approved these changes
Nov 8, 2023
Comment on lines
+64
to
+69
| http:// { | ||
| {{ $allocID := env "NOMAD_ALLOC_ID" -}} | ||
| {{ range nomadService 1 $allocID "prometheus" }} | ||
| reverse_proxy {{ .Address }}:{{ .Port }} | ||
| {{ end }} | ||
| } |
There was a problem hiding this comment.
Suggested change
| http:// { | |
| {{ $allocID := env "NOMAD_ALLOC_ID" -}} | |
| {{ range nomadService 1 $allocID "prometheus" }} | |
| reverse_proxy {{ .Address }}:{{ .Port }} | |
| {{ end }} | |
| } | |
| http:// { | |
| reverse_proxy {{ range nomadService 1 $allocID "prometheus" }}{{ .Address }}:{{ .Port }} {{end}} | |
| } |
Not that it matters much in this case, but we could let Caddy handle the load balancing. I'm also not too familar with Caddyfiles, so I hope this is correct 😅
There was a problem hiding this comment.
I gave this a try and it failed ... probably not going to spend the time to investigate, so many other test failures to look into 😓
e2e/metrics/input/cpustress.hcl
Outdated
|
|
||
| config { | ||
| command = "stress" | ||
| args = ["--cpu", "1", ] |
There was a problem hiding this comment.
Suggested change
| args = ["--cpu", "1", ] | |
| args = ["--cpu", "1"] |
| } | ||
|
|
||
| task "cpustress" { | ||
| driver = "pledge" |
Comment on lines
+46
to
+47
| # run a private holepunch instance in this group network | ||
| # so prometheus can access the nomad api for service disco |
Yeah for sure, we can look into expanding this especially once we get the windows client back |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR overhauls the metrics e2e suite which has been failing for like a year. It swaps out the use of Consul in favor of Nomad native service discovery and workload identity.
Basically it runs a handful of random jobs which produce metrics, then uses Prometheus to gather those metrics. Caddy is used to expose the Prometheus API to be reachable from the test runner. The little nomad-holepunch thing is used as a side car to enable Prometheus to access the Nomad API using workload identity. It's also used as a service job to represent each Nomad client in the Nomad service registry.