WorkerPools resolve their default SandboxConfig and keep configs in use from being deleted

[Zoe Zhao (zoez7)]

Part of #587. This is step 3 of the SandboxConfig management design (linked from #587), after #2332 (WorkerPool spec.sandboxClasses) and #2333 (versioned SandboxConfig and the is-class-default annotation).

This step resolves each WorkerPool's SandboxConfig, records it in status, gates the worker Deployment on it, and keeps a config in use from being deleted. The worker Deployment does not read the resolved config yet: the binaries an actor boots with still come from the SandboxConfig its ActorTemplate names. Pointing workers at the pool's config is the next step.

Changes

WorkerPool status reports the resolved config.

Unpinned: the pool follows the class default.

apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: counter-pool
  namespace: ate-demo
  generation: 1
spec:
  replicas: 2
  workerImage: ateom-gvisor:v1
  sandboxClasses:
  - name: gvisor                  # no configRef
status:
  replicas: 2
  readyReplicas: 2
  selector: ate.dev/worker-pool=counter-pool
  sandboxClasses:
  - name: gvisor
    configRef:
      name: gvisor-default        # newest gvisor config annotated is-class-default
  conditions:
  - type: SandboxConfigResolved
    status: "True"
    reason: Resolved
    message: every sandbox class resolved to a SandboxConfig
    observedGeneration: 1

Pinned: the pool resolves to gvisor-release1 even after a newer default is created.

apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: pinned-pool
  namespace: ate-demo
  generation: 1
spec:
  replicas: 2
  workerImage: ateom-gvisor:v1
  sandboxClasses:
  - name: gvisor
    configRef:
      name: gvisor-release1
status:
  replicas: 2
  readyReplicas: 2
  selector: ate.dev/worker-pool=pinned-pool
  sandboxClasses:
  - name: gvisor
    configRef:
      name: gvisor-release1       # the pinned config
  conditions:
  - type: SandboxConfigResolved
    status: "True"
    reason: Resolved
    message: every sandbox class resolved to a SandboxConfig
    observedGeneration: 1

• With configRef, the pool resolves to that config. It must exist and be of the same class.
• Without configRef, the pool resolves to the newest config of its class annotated sandboxconfig.ate.dev/is-class-default: "true". If two were created in the same second, the name that sorts first wins.
• If no default exists but the pool already uses a config of that class, it keeps that config and stays Resolved. DefaultConfigNotFound is only reported when there is nothing to fall back on: the pool never resolved, or it switched sandbox class.
• The controller re-resolves on every reconcile and whenever a SandboxConfig changes, so a new default takes over within seconds.
• kubectl get workerpool gains two columns: SANDBOXCONFIG, the resolved config, and RESOLVED, the status of the SandboxConfigResolved condition.

An unresolved pool gets no new or changed workers.

• SandboxConfigResolved=False reports one of these reasons:
  • SandboxConfigNotFound
  • SandboxClassMismatch
  • SandboxConfigDeleting
  • DefaultConfigNotFound
• A Warning event is emitted once per distinct failure.
• The controller does not create the pool's Deployment. An existing Deployment keeps running but receives no spec changes, not even a scale, until the pool resolves again.
• status.sandboxClasses keeps the last resolved config while the condition is False. It records what the pool last resolved to; the condition says whether the current spec can be met.

A SandboxConfig in use cannot be deleted.

• When a WorkerPool resolves a SandboxConfig, the WorkerPool controller puts the sandboxconfig.ate.dev/workerpool-protection finalizer on the config before recording it in status.sandboxClasses, so a config is never in use without it. The finalizer is added nowhere else and stays until the config is deleted. A config no pool has ever used carries no finalizer and deletes at once.
• Deleting a config that carries the finalizer marks it for deletion. A new SandboxConfigProtectionReconciler removes the finalizer only once no WorkerPool's status.sandboxClasses names it. It asks a cache index first; when the cache finds no user, it lists WorkerPools from the API server, bypassing the cache, before it removes the finalizer. A config pools stopped using is gone within one reconcile; one that pools use stays Terminating until they move or are deleted.
• A config that is being deleted takes no new pools. A configRef to it reports SandboxConfigDeleting, and it stops being a class default. A pool already on it keeps resolving to it: a pinned pool until its configRef is changed, an unpinned pool until another default exists. A Terminating config therefore only loses users.
• Release is its own reconciler, keyed by SandboxConfig, because WorkerPools carry no finalizer. A deleted pool reconciles as NotFound, so the WorkerPool controller cannot learn which configs it used. The protection reconciler is instead enqueued from WorkerPool events, including deletes, through the old object's status.sandboxClasses. Every SandboxConfig is also reconciled when atecontroller starts, so a config whose last user went away while atecontroller was down is released then.

Supporting changes

• ate-setup delete strips the finalizer from every SandboxConfig that carries it. It runs after the namespace, and with it atecontroller, is gone and the CRDs have been deleted, which removes every WorkerPool. Only atecontroller releases the finalizer, so without this step a config a pool had used would stay Terminating and the sandboxconfigs CRD could not finish deleting. The step is a no-op when the CRD is already gone.
• RBAC (generated role.yaml): atecontroller gains get/list/watch/patch on sandboxconfigs, for resolution, the watch, and the finalizer, and create/patch on events.k8s.io events.
• spec.sandboxClasses[].configRef.name is bounded to 253 characters, and the godoc of the fields Add sandboxClasses with an optional SandboxConfig ref to WorkerPool #2332 added now starts with the serialized field name.
• fake-workersync compares status with DeepEqual, because == no longer compiles now that the status has slice fields.
• Tests: TestMain in the controllers package now runs the protection reconciler and seeds a gvisor class default, so pools without configRef resolve. The shipped-manifest test in pkg/api/v1alpha1 cleans up with a fresh context and asserts the delete; no controller runs there, so the config never gains the finalizer.

Breaking change

A WorkerPool whose class has no resolvable SandboxConfig gets no new or changed workers. An existing Deployment keeps its workers but stops scaling or updating until the pool resolves. Before upgrading, make sure each class your pools use either has a config annotated is-class-default: "true" or is pinned with configRef. The bundled gvisor-default and the micro-VM config from hack/install-microvm-deps.sh are already annotated.

If you roll atecontroller back past this PR, nothing removes the finalizer from configs that carry it anymore. Clear it by hand with kubectl patch sandboxconfig <name> --type=merge -p '{"metadata":{"finalizers":null}}'.

• Tests pass
• Appropriate changes to documentation are included in the PR