Runner records stranded in pending_delete permanently wedge scale set autoscaling

[KT-Doan]

Summary

A runner record can get stranded in pending_delete forever while its provider instance is already gone. Stranded records count toward the scale set's runner count, so once a scale set has stuck records >= min(desired, max_runners) its autoscaler sits in the scale-down branch permanently and never creates a runner again. Jobs then queue indefinitely on a scale set that is actually empty. With min_idle_runners: 0, desired is typically 1 on a quiet set, so two stuck records are enough to wedge any scale set regardless of max_runners. A controller restart clears it (state is rebuilt from the DB), which is how we kept recovering before finding the cause.

Observed repeatedly in a production deployment built from main at afda4e76f1808e8b41d72edd0c17f99a51d84758 with the GCP provider (v0.1.5) and agent mode enabled. Under load we saw multiple new stranded records per day.

The chain, verified in source and controller logs

1. A job completes; the scaleset worker moves the instance to pending_delete (validated write). The instance manager receives the update and acts: consolidateState() force-writes deleting, then calls the provider delete while holding i.mux. A GCE delete takes tens of seconds.

2. The watcher event for that deleting write is received by the manager's updatesLoop, which immediately blocks in handleUpdate() on i.mux. Every later event for the same instance is then dropped by instanceManager.Update()'s 10 s send timeout:

   01:05:20.192 INFO  job completed
   01:05:20.627 INFO  deleting instance in provider
   01:05:30.669 ERROR failed to update instance | timeout while sending update to instance manager
   01:05:40.669 ERROR failed to update instance | timeout while sending update to instance manager
   

3. Meanwhile the agent worker sees the runner report terminated (handleStatusMessage) and calls Runner.SetInstanceToPendingDelete, which uses ForceUpdateInstance: it writes pending_delete over deleting with no transition validation, moving the record backwards mid-delete.

4. The provider delete finishes; the instance is gone. The manager's validated deleted write now fails, because pending_delete -> deleted is not in params.InstanceStatusTransitions:

   01:05:55.550 ERROR consolidating state | setting instance status to deleted: ... invalid instance status transition from pending_delete to deleted
   

5. The blocked handleUpdate from step 2 acquires the mutex and sets the manager's cached state to deleting. consolidateState()'s switch has no case for InstanceDeleting, so every 5 s tick silently does nothing, forever. The manager is alive, so retryFailedRunners skips it; the scaleset worker's consolidation skips deletion-lane records because the provider worker owns them. Nothing ever re-drives the record.

6. runnerCount() is len(w.runners) and counts the stranded record. targetRunners() is min(MinIdleRunners+desired, MaxRunners), so handleAutoScale picks handleScaleDown forever, logging considering runners for removal every 5 s (we counted 240 lines per 10 minutes during one wedge).

Remediation that works on a live wedge

garm-cli runner delete -f <name> moves the record pending_delete -> pending_force_delete -> deleting -> deleted and the set resumes scaling within seconds (verified on two live occurrences). A controller restart also clears it.

Fixes

I have a PR ready with three independent fixes, each with tests; any one of them breaks the chain, and together they remove the corruption, the dead state, and the starvation:

1. Runner.SetInstanceToPendingDelete uses a validated update and treats a refused transition from a deletion-lane status as success (the intent, the runner goes away, is already being fulfilled).
2. consolidateState() treats InstanceDeleting as a resumable delete (deletes are idempotent: providers report a missing instance as success), requeues on provider errors for every non-forced path, and force-writes the final deleted when the row regressed onto the deletion lane behind its back, because the provider resource is confirmed gone at that point.
3. runnerCount() excludes deletion-lane records, and handleScaleDown's deletion-lane case no longer consumes the removal delta.

Two related defects the PR deliberately does not touch, flagging them here instead: a timed-out instanceManager.Update() send discards the update with no requeue, and consolidateState() holds i.mux across provider calls, which is what starves updatesLoop/handleUpdate and causes those drops in the first place. Both are worth fixing but are invasive to the worker's concurrency structure; with the three fixes above the dropped updates become harmless.