Repository navigation
Conversation
The storage benchmark addressed its actor through the Host header
(<actor>.<atespace>.actors.resources.substrate.ate.dev). The atenet
router no longer routes on Host: it reads the ate-target-actor header
("<atespace>/<actor>"), as the other glutton workers already send.
Without the header, the router could not tell which actor a write was
for and answered every GluttonWriteDisk with 404 "invalid actor
reference", so the benchmark measured no disk writes at all. Send the
header the router expects, and drop the unused Host value.
When ResumeActor failed, the storage benchmark returned without its wait between iterations, and boomer called it again straight away. Once an actor crashed, every later resume failed the same way, so a single crashed actor turned into a tight loop: about 25,000 failed ResumeActor calls in two minutes. That put needless load on ateapi and drowned the real resume latencies in the stats. Wait on every path, as the glutton benchmark does. When ateapi reports the actor as crashed, count it once under CrashCount and stop sending it requests, so a crash shows up as a crash instead of as thousands of failed resumes.
LOCUST_IMAGE_PULL_POLICY now defaults to Always for every image, and setting the variable still overrides it. The default image tag is :latest, so IfNotPresent could start a stale cached image. The storage benchmark had its own copy of goroutineID. It now calls boomerutil.GoroutineID, the shared helper the other boomer user classes already use.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #2057. Until that merges, this PR shows its commits too, so review the last 3 commits.
tests.yamlentry can setstorageClassName. The orchestrator passes it toworkloads/deploy.sh --storage-class-name, which adds theglutton-storagetemplate.tests.yamlentry can list Kubernetes manifest files inadditionalManifests, for anything the test needs that the substrate install does not create. The orchestrator applies them after it deploys substrate and before the workloads, so a manifest can use substrate's CRDs and a StorageClass exists beforeglutton-storageis created. It deletes them in the sweep before each test and in the teardown after it. Relative paths resolve against the directory holdingtests.yaml. Every file is checked before anything is applied, and a missing file fails the test.substrate-benchmark-test-manifestsConfigMap at/etc/orchestrator/test-manifests, next totests.yaml, so a test lists its files astest-manifests/<file>.setup.shbuilds the ConfigMap from the YAML files inbenchmarking/automation/test-manifests/. With no files there, the ConfigMap is empty.durdir_external_volume_exampleentry intests.yamlruns the DurDir load oncsi-nfs-sc, the class--setup-csi=nfscreates. It needs no extra manifest, so there is still one NFS StorageClass.This is the test-specific config Max raised on #2057. It is meant as a starting point for the test API. cc Max Smythe (@maxsmythe) pmandewalkar
Testing: unit tests for the
storageClassNameflag,additionalManifestsvalidation and path resolution, the apply and delete order, and wheremain()applies and deletes the manifests when the test fails or a file is missing.tests.yamlpasses the orchestrator's validation as is and with the example entry uncommented. Thesetup.shConfigMap step was run with notest-manifests/directory, an empty one, and one file, on bash 5.3 and bash 3.2. shellcheck and the boilerplate check pass. Not run end to end on a cluster.