Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup jobs that are canceled #133

Open
benmoss opened this issue Feb 17, 2023 · 0 comments
Open

Cleanup jobs that are canceled #133

benmoss opened this issue Feb 17, 2023 · 0 comments

Comments

@benmoss
Copy link
Contributor

benmoss commented Feb 17, 2023

We sometimes seem jobs that get stuck for some reason (misconfiguration, etc) that causes them to get into a unrecoverable state. If a user hits "cancel" on the Buildkite UI, we should notice this and delete the job from Kubernetes.

Most of the time the agent will see the cancellation and do cleanup, but there are still times when the agent has failed or hasn't started that the job can be orphaned.

We could do this with a background task that periodically fetches builds we are tracking and checks their status on the Buildkite API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant