Extending AnomalousStatus to also kill sh steps
#405
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Ineffective. |
jglick
commented
Nov 6, 2024
src/main/java/org/jenkinsci/plugins/workflow/support/steps/ExecutorStepExecution.java
Outdated
Show resolved
Hide resolved
dwnusbaum
approved these changes
Nov 7, 2024
| // Also abort any shell steps running on the same node(s): | ||
| if (!affectedNodes.isEmpty()) { | ||
| StepExecution.applyAll(DurableTaskStep.Execution.class, exec -> { | ||
| if (affectedNodes.contains(exec.node)) { |
There was a problem hiding this comment.
Are there cases (e.g. nodes with multiple executors?) where this could abort steps which are running fine if another step on the same node is having unusual problems? If so, could we check exec.state.cookie or something more precise than just the node name instead?
There was a problem hiding this comment.
In theory perhaps, but this monitor is normally used for cloud nodes with one executor, and it seems unlikely that the agent could be connected and functional on one executor and node block while broken in another one of the same build. (For that matter, it would rarely make any sense to run two concurrent node blocks in the same build on the same agent.)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In many cases when restoring a build into another K8s cluster using a very lossy backup of the filesystem, via EFS Replication (which does not guarantee snapshot semantics), there is some sort of problem with metadata, which prevents
nodeblock retry from recovering automatically. (Prior to jenkinsci/kubernetes-plugin#1617 it did not work even if metadata was perfect.) Sometimes there is a missingprogram.dat, sometimes a corrupted log file, sometimes a missingFlowNode, etc.But in many of these cases (CloudBees-internal reference), the log seems fine and the flow nodes seem fine, yet for reasons I cannot easily follow because
program.datis so opaque, thenodeblock seems to have received anOutcome.abnormalwith the expectedFlowInterrupedExceptionfromExecutorStepDynamicContext.resume; there are also some suppressed exceptions, andAnomalousStatusjust adds to this list, without causing the build to proceed. CallingCpsStepContext.scheduleNextRun()from the script console does not help either. However in most of these cases it does seem to work to abort theshstep running inside: somehow that “wakes up” the program, which then fails thenodeblock in the expected way, letting theretrystep kick in and ultimately letting the build run to completion.