Configurable machine replacement

[Meecr0b]

What would you like to be added (User Story)?

As a operator i would like to be able to configure a time after machines are getting replaced automatically for testing and security reasons.

Detailed Description

Problem Statement:

Regularly replacing machines help in testing application behavior during rolling updates and ensures machines are refreshed periodically, especially important after security incidents.

Proposed Solution:

Implement rolloutBefore.machineExpiry{Minutes,Hours,Days} parameter within the Cluster API (like rolloutBefore.certificatesExpiryDays implemented for KCP), allowing users to specify the maximum time a machine should exist before being automatically replaced.

Benefits:

• Testing Rolling Updates: Simplifies the process of regularly testing how applications behave during rolling updates.
• Security and Compliance: Ensures machines are periodically replaced, reducing the risk of lingering vulnerabilities and ensuring machines are clean post-security incidents.
• Operational Efficiency: Automates a routine maintenance task, reducing manual workload and the potential for human error.

Impact:

• This feature would be highly valuable for IT operations teams managing Kubernetes clusters, particularly those with strict compliance and security requirements.
• It enhances cluster maintenance workflows, contributing to overall system reliability and security.

Anything else you would like to add?

Current workarounds:

• setting spec.rolloutAfter periodically via CronJob for MachineDeployment
• running clusterctl alpha rollout restart machinedeployment/my-md-0 periodically

Label(s) to be applied

/kind feature
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[sbueringer]

/triage accepted

/cc @fabriziopandini @chrischdi

[fabriziopandini]

q: is this about replacing nodes (the node at Kubernetes level) or the entire machine where the node is hosted?

[Meecr0b]

Hi @fabriziopandini it's about machines, i'll update the issue.

[fabriziopandini]

ACK, thanks for the clarification
We need to think a bit about API modeling, but this is a nice feature to have
/help

[k8s-ci-robot]

@fabriziopandini:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

ACK, thanks for the clarification
We need to think a bit about API modeling, but this is a nice feature to have
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dineshba]

I would like to contribute to this feature. If no one is already started, Shall I pick this up? cc: @sbueringer @fabriziopandini @Meecr0b

I can do the initial research and share the api-modeling, high-level code changes for the first review

[sbueringer]

@dineshba Feel free to go ahead

[dineshba]

/assign dineshba

[dineshba]

I would like to share my initial idea and get feedback on the approach

cc: @sbueringer @fabriziopandini (tagging @ykakarap also as he has contributed to MachineDeployment rolloutAfter feature #8216 which is similar to this)

Feature Description

We want to specify a duration in spec after which machines should get replaced. MachineExpiry{Minutes,Hours,Days}. This feature should be available for machines managed by KCP and for machines managed by MachineDeployments/MachineSets.

Related exisiting features

• RolloutAfter in MachineDeployment and KCP.

    // RolloutAfter is a field to indicate a rollout should be performed
	// after the specified time even if no changes have been made to the
	// KubeadmControlPlane.
	// Example: In the YAML the time can be specified in the RFC3339 format.
	// To specify the rolloutAfter target as March 9, 2023, at 9 am UTC
	// use "2023-03-09T09:00:00Z".
    // +optional
	RolloutAfter *metav1.Time `json:"rolloutAfter,omitempty"`

• RolloutBefore.CertificatesExpiryDays in KCP (cert based rollout is only for controlplane machines)

type KubeadmControlPlaneSpec struct {
    // RolloutBefore is a field to indicate a rollout should be performed
	// if the specified criteria is met.
	// +optional
	RolloutBefore *RolloutBefore `json:"rolloutBefore,omitempty"`
}

// RolloutBefore describes when a rollout should be performed on the KCP machines.
type RolloutBefore struct {
  // CertificatesExpiryDays indicates a rollout needs to be performed if the
  // certificates of the machine will expire within the specified days.
  // +optional
  CertificatesExpiryDays *int32 `json:"certificatesExpiryDays,omitempty"`
}

API Change Proposal

Option 1: Specify machine expiry under RolloutBefore

• Add MachineExpiryDays to existing RolloutBefore of KubeadmControlPlaneSpec
• Add RolloutBefore.MachineExpiryDays for MachineDeploymentSpec

In this approach, we are trying to extend the existing API. We want to rollout once it reached the specified duration. Adding it under RolloutBefore struct looks not appropriate. Please suggest your inputs

Option 2: Specify machine expiry under new struct named Rollout (or better name)

type KubeadmControlPlaneSpec struct {
  // Rollout indicates different capabilites to
  // rollout the machine when the specified conditions are met
  Rollout *Rollout `json:"rollout,omitempty"`
}  

type Rollout struct {
  // MachineExpiry indicates the duration after which the machine
  // will be rolled out. If Creation time - Current time  > duration,
  // then rollout and replace the expired machine
  MachineExpiryDays*int`json:"machineexpiry,omitempty"`
}

In this new Rollout struct, we can define MachineExpiryDays. We can add other machine maintenance to this struct in the future.

Code Changes:

• Webhook: Validate if MachineExpiry is greater than or equal to 2 hours
• Controller Logic: When MachineExpiry is specified and if it crossed for the machine's creationTimestamp, then recreate the machine
  • For KCP, return true in https://github.com/kubernetes-sigs/cluster-api/blob/main/util/collections/machine_filters.go#L223 which will rollout the controlplane machines
  • For MachineDeployment, create nil NewMachineSet from https://github.com/kubernetes-sigs/cluster-api/blob/main/internal/controllers/machinedeployment/mdutil/util.go#L420 which will create a new MachineSet

Note: I am new to cluster-api repo and my logic may not be fully correct. I'll add the complete logic, unit tests, manually test it and then finally raise the PR. Please let me know if I am in the right direction regarding the api changes and code changes.

[fabriziopandini]

On my todo list, just but unfortunately not much bandwidth currently 😓