Design: Separate reboot coordination and version operator

[dghubble]

Proposal

Simplify the update-operator so that it just performs update coordination (not deploying the agent). Write a separate version-operator which ensures the desired version of the coordinator and agent are running (creating them if not, managing migrations if any in future).

In the desired architecture, there would be three single-purpose components (names are examples):

• container-linux-update-coordinator - deployment which watches node annotations and coordinates reboots as needed to ensure not too many are rebooting at once. (doesn't deploy agent)
• container-linux-update-agent - daemonset which listens for D-Bus signals from update-engine and indicates via annotations that it needs a reboot.
• container-linux-version-operator - (optional) ensures the correct version of the update-coordinator and update-agent are running via a reconciliation loop. Handles migrations, if needed.

A user may choose to deploy just the update-coordinator and update-agent. Or he/she could deploy the container-linux-version-operator to manage those apps on their behalf and contain reconciliation and migration logic.

Problem

In the current design, the update-operator is doing both update coordination and deployment of the agent (conditionally depending on flags) at a hardcoded version. This is problematic

• Users ask for the ability to customize the update-agent daemonset through the update-operator. This expands the scope of the operator greatly and makes it take configuration for the agent it deploys and for coordinating reboots. There is no plan for how migrations will work.
• update-operator is not currently performing a reconciliation loop to ensure the update-agent is running. Its just performing a one-time check on startup.
• update-operator is making assumptions about compatibility between itself and update-agent releases. That complexity should live in a component designed for it.
• Oddity or code smell is apparent from the -manage-agent=true/false option. https://github.com/coreos/container-linux-update-operator/blob/master/cmd/update-operator/main.go#L19
• Conceivably we only need to update one component, not both. Right now we bump both.
• Differs from practices established through experience in other CoreOS and Tectonic operators.

Discussed with @aaronlevy and @euank

[dghubble]

With this proposal, users of this open-source repository would just deploy the two components container-linux-update-coordinator deployment and container-linux-update-agent daemonset. Example manifests would be provided. --manage-agent would be removed. Customization would be possible just by editing the manifests yourself.

container-linux-version-operator would be internal and have a different audience in mind. It aims to automate creation and migrations of those two components and may have Tectonic-specific logic similar to existing internal operators.

I believe this separation will be a benefit to both open-source users and Tectonic users.

[yifan-gu]

LGTM

[derekparker]

LGTM

[dghubble]

The required changes to CLUO have been made. Tectonic migrate to this behavior in coreos/tectonic-installer#1644.