Track pod state from deployment run

[jpedrick-numeus]

Describe the current behavior

As a cost overrun prevention measure our kubernetes work pool base job template has active_deadline_seconds set. If a pod is killed from under the job, it stays in the Running state forever and takes up slots in the work queue.

Describe the proposed behavior

I think it would make sense to track the pod state for kubenetes work pools, so that the jobs know to either start a new pod and re-run or report 'Failed' with a reason.

Likewise, it would be good if some job metadata (such as the pod name, resource requests, etc) were visible from the prefect UI.

Example Use

They can add a simple layer of protection for cost-overruns by setting active_deadline_seconds.

Additional context

No response

[desertaxle]

Thanks for the enhancement request @jpedrick-numeus! One idea we've had in this area is to have pods send heartbeats back to the Prefect server so that if the heartbeats stop, the server knows if a pod went down. In this case, we'd probably mark the flow run as CRASHED since the underlying infrastructure caused the failure. Does that sounds like it would work for your use case?

Also, where would you expect to see Kubernetes information for a flow run in the Prefect UI?

[jpedrick-numeus]

@desertaxle that would work for me. In my case I only need the pod state to be tracked so that prefect knows to move on to the next job in the Work Queue.

I think the details tab under https:///flow-runs/flow-run/?tab=Details would be perfect.

[jameswu1991]

just to help tie dispersed communication together, i believe these threads are the same issue:

• Flow still in "Running" state when agent is down #7239
• Job State Stuck in Running After Pod Eviction #15945

it seems that using heartbeats to detect crashed pids is an old idea, but unfortunately it caused unhappy memories back in v1. yet, according to the replies in #7239, many members of the community (myself included) would love to see it come back (of course, as an optional non-default feature).

the release of prefect v3 brought back an implementation of heartbeats as a status on the worker (for worker pools only). i would speculate it's not too hard to write a loop service that periodically checks for crashed workers and places their deployments also into crashed state, but the specifics are beyond my expertise.

the ability to distinguish a healthy worker vs a worker that crashed but came back before the heartbeat threshold was reached, may also not be easy to implement. perhaps one could use the unique worker id (auto-generated unique k8s pod name for example) as an incarnation distinguisher.

the other way to (partially) mitigate this issue would be to propagate SIGINT from the parent process to the child process, thereby giving the child process some time to react and gracefully shut down, or at least report back to prefect api that it has crashed. this behavior was first noticed a while ago but it seems to not have been fixed.