Z1962490 Machine-config-controller not generating new machineconfigs …

…for new Kubeletconfig CR
chinmayi-chandrasekar · Jul 2, 2021 · 6f1a391 · 6f1a391
1 parent 9c0de50
commit 6f1a391
Show file tree

Hide file tree

Showing 2 changed files with 152 additions and 25 deletions.
diff --git a/modules/create-a-containerruntimeconfig-crd.adoc b/modules/create-a-containerruntimeconfig-crd.adoc
@@ -20,15 +20,17 @@ You can modify the following settings by using a `ContainerRuntimeConfig` CR:
 * **Maximum log size**: The `logSizeMax` parameter sets the CRI-O `log_size_max` parameter, which is the maximum size allowed for the container log file. The default is unlimited (`log_size_max = -1`). If set to a positive number, it must be at least 8192 to not be smaller than  the ConMon read buffer. ConMon is a program that
 monitors communications between a container manager (such as Podman or CRI-O) and the OCI runtime (such as runc or crun) for a single container.
 
-You can create multiple `ContainerRuntimeConfig` CRs, as needed, with a limit of ten. For the first `ContainerRuntimeConfig` CR, the MCO creates a machine config appended with `containerruntime`. With each subsequent CR, the controller creates a new `containerruntime` machine config with a numeric suffix. For example, if you have a `containerruntime` machine config with a `-2` suffix, the next `containerruntime` machine config is appended with `-3`.
+You should have one `ContainerRuntimeConfig` CR for each machine config pool with all the config changes you want for that pool. If you are applying the same content to all the pools, you only need one `ContainerRuntimeConfig` CR for all the pools. 
 
-You can also edit an existing `ContainerRuntimeConfig` CRs to edit existing settings or add new settings instead of creating a new CR. It is recommended to create a new CR to modify a different machine config pool, rather than add additional labels to an existing CR.
+You should edit an existing `ContainerRuntimeConfig` CR to modify existing settings or add new settings instead of creating a new CR for each change. It is recommended to create a new `ContainerRuntimeConfig` CR only to modify a different machine config pool, or for changes that are intended to be temporary so that you can revert the changes.
+
+You can create multiple `ContainerRuntimeConfig` CRs, as needed, with a limit of 10 per cluster. For the first `ContainerRuntimeConfig` CR, the MCO creates a machine config appended with `containerruntime`. With each subsequent CR, the controller creates a new `containerruntime` machine config with a numeric suffix. For example, if you have a `containerruntime` machine config with a `-2` suffix, the next `containerruntime` machine config is appended with `-3`.
 
 If you want to delete the machine configs, you should delete them in reverse order to avoid exceeding the limit. For example, you should delete the `containerruntime-3` machine config before deleting the `containerruntime-2` machine config.
 
 [NOTE]
 ====
-If you have a machine config with a `containerruntime-9` suffix, the next machine config is appended with `containerruntime-10` and will fail as it exceeds the limit, even if there are fewer than 10 `containerruntime` machine configs.
+If you have a machine config with a `containerruntime-9` suffix, and you create another `ContainerRuntimeConfig` CR, a new machine config is not created, even if there are fewer than 10 `containerruntime` machine configs.
 ====
 
 .Example showing multiple `ContainerRuntimeConfig` CRs

diff --git a/modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc b/modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc
@@ -8,35 +8,114 @@
 = Creating a KubeletConfig CRD to edit kubelet parameters
 
 The kubelet configuration is currently serialized as an Ignition configuration, so it can be directly edited. However, there is also a new
-`kubelet-config-controller` added to the Machine Config Controller (MCC). This allows you to create a `KubeletConfig` custom resource (CR) to edit the kubelet parameters.
+`kubelet-config-controller` added to the Machine Config Controller (MCC). This allows you to use a `KubeletConfig` custom resource (CR) to edit the kubelet parameters.
+
+You should have one `KubeletConfig` CR for each machine config pool with all the config changes you want for that pool. If you are applying the same content to all the pools, you only need one `KubeletConfig` CR for all the pools. 
+
+You should edit an existing `KubeletConfig` CR to modify existing settings or add new settings instead of creating a new CR for each change. It is recommended to create a new CR only to modify a different machine config pool, or for changes that are intended to be temporary so that you can revert the changes.
+
+You can create multiple `KubeletConfig` CRs, as needed, with a limit of 10 per cluster. For the first `KubeletConfig` CR, the MCO creates a machine config appended with `kubelet`. With each subsequent CR, the controller creates a new `kubelet` machine config with a numeric suffix. For example, if you have a `kubelet` machine config with a `-2` suffix, the next `kubelet` machine config is appended with `-3`.
+
+If you want to delete the machine configs, you should delete them in reverse order to avoid exceeding the limit. For example, you should delete the `kubelet-3` machine config before deleting the `kubelet-2` machine config.
+
+[NOTE]
+====
+If you have a machine config with a `kubelet-9` suffix, and you create another `KubeletConfig` CR, a new machine config is not created, even if there are fewer than 10 `kubelet` machine configs.
+====
+
+.Example `KubeletConfig` CR
+[source,terminal]
+----
+$ oc get kubeletconfig
+----
+
+[source, terminal]
+----
+NAME                AGE
+set-max-pods        15m
+----
+
+.Example showing a `KubeletConfig` machine config
+[source,terminal]
+----
+$ oc get mc | grep kubelet
+----
+
+[source, terminal]
+----
+...
+99-worker-generated-kubelet-1                  b5c5119de007945b6fe6fb215db3b8e2ceb12511   3.2.0             26m
+...
+----
+
+The following procedure is an example to show how to configure the maximum number of pods per node on the worker nodes.
+
+.Prerequisites
+
+. Obtain the label associated with the static `MachineConfigPool` CR for the type of node you want to configure.
+Perform one of the following steps:
+
+.. View the machine config pool:
++
+[source,terminal]
+----
+$ oc describe machineconfigpool <name>
+----
++
+For example:
++
+[source,terminal]
+----
+$ oc describe machineconfigpool worker
+----
++
+.Example output
+[source,yaml]
+----
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfigPool
+metadata:
+  creationTimestamp: 2019-02-08T14:52:39Z
+  generation: 1
+  labels:
+    custom-kubelet: set-max-pods <1>
+----
+<1> If a label has been added it appears under `labels`.
+
+.. If the label is not present, add a key/value pair:
++
+[source,terminal]
+----
+$ oc label machineconfigpool worker custom-kubelet=set-max-pods
+----
 
 .Procedure
 
-. Run:
+. View the available machine configuration objects that you can select:
 +
 [source,terminal]
 ----
 $ oc get machineconfig
 ----
 +
-This provides a list of the available machine configuration objects you can select. By default, the two kubelet-related configs are `01-master-kubelet` and `01-worker-kubelet`.
+By default, the two kubelet-related configs are `01-master-kubelet` and `01-worker-kubelet`.
 
-. To check the current value of max pods per node, run:
+. Check the current value for the maximum pods per node:
 +
 [source,terminal]
 ----
-# oc describe node <node-ip> | grep Allocatable -A6
+$ oc describe node <node_name>
 ----
 +
-Look for `value: pods: <value>`.
-+
 For example:
 +
 [source,terminal]
 ----
-# oc describe node ip-172-31-128-158.us-east-2.compute.internal | grep Allocatable -A6
+$ oc describe node ci-ln-5grqprb-f76d1-ncnqq-worker-a-mdv94
 ----
 +
+Look for `value: pods: <value>` in the `Allocatable` stanza:
++
 .Example output
 [source,terminal]
 ----
@@ -49,7 +128,7 @@ Allocatable:
  pods:                        250
 ----
 
-. To set the max pods per node on the worker nodes, create a custom resource file that contains the kubelet configuration. For example, `change-maxPods-cr.yaml`:
+. Set the maximum pods per node on the worker nodes by creating a custom resource file that contains the kubelet configuration:
 +
 [source,yaml]
 ----
@@ -60,13 +139,17 @@ metadata:
 spec:
   machineConfigPoolSelector:
     matchLabels:
-      custom-kubelet: large-pods
+      custom-kubelet: set-max-pods <1>
   kubeletConfig:
-    maxPods: 500
+    maxPods: 500 <2>
 ----
+<1> Enter the label from the machine config pool.
+<2> Add the kubelet configuration. In this example, use `maxPods` to set the maximum pods per node.
 +
-The rate at which the kubelet talks to the API server depends on queries per second (QPS) and burst values. The default values, `50` for `kubeAPIQPS` and `100` for `kubeAPIBurst`, are good enough if there are limited pods running on each node. Updating the kubelet QPS and burst rates is recommended if there are enough CPU and memory resources on the node:
-+
+[NOTE]
+====
+The rate at which the kubelet talks to the API server depends on queries per second (QPS) and burst values. The default values, `50` for `kubeAPIQPS` and `100` for `kubeAPIBurst`, are sufficient if there are limited pods running on each node. It is recommended to update the kubelet QPS and burst rates if there are enough CPU and memory resources on the node.
+
 [source,yaml]
 ----
 apiVersion: machineconfiguration.openshift.io/v1
@@ -76,43 +159,85 @@ metadata:
 spec:
   machineConfigPoolSelector:
     matchLabels:
-      custom-kubelet: large-pods
+      custom-kubelet: set-max-pods
   kubeletConfig:
     maxPods: <pod_count>
     kubeAPIBurst: <burst_rate>
     kubeAPIQPS: <QPS>
 ----
-
-.. Run:
+====
+.. Create the `KubeletConfig` object:
 +
 [source,terminal]
 ----
 $ oc create -f change-maxPods-cr.yaml
 ----
 
-.. Run:
+.. Verify that the `KubeletConfig` object is created:
 +
 [source,terminal]
 ----
 $ oc get kubeletconfig
 ----
 +
-This should return `set-max-pods`.
+.Example output
+[source, terminal]
+----
+NAME                AGE
+set-max-pods        15m
+----
 +
 Depending on the number of worker nodes in the cluster, wait for the worker nodes to be rebooted one by one. For a cluster with 3 worker nodes, this could take about 10 to 15 minutes.
 
-. Check for `maxPods` changing for the worker nodes:
+. Verify that the changes are applied to the node:
+
+.. Check on a worker node that the `maxPods` value changed:
 +
 [source,terminal]
 ----
-$ oc describe node
+$ oc describe node <node_name>
 ----
 
-.. Verify the change by running:
+.. Locate the `Allocatable` stanza:
++
+[source,terminal]
+----
+ ...
+Allocatable:
+  attachable-volumes-gce-pd:  127
+  cpu:                        3500m
+  ephemeral-storage:          123201474766
+  hugepages-1Gi:              0
+  hugepages-2Mi:              0
+  memory:                     14225400Ki
+  pods:                       500 <1>
+ ...
+----
+<1> In this example, the `pods` parameter should report the value you set in the `KubletConfig` object.
+
+. Verify the change in the `KubeletConfig` object:
 +
 [source,terminal]
 ----
 $ oc get kubeletconfigs set-max-pods -o yaml
 ----
 +
-This should show a status of `True` and `type:Success`
+This should show a status of `True` and `type:Success`:
++
+[source,yaml]
+----
+ ...
+spec:
+  kubeletConfig:
+    maxPods: 500
+  machineConfigPoolSelector:
+    matchLabels:
+      custom-kubelet: set-max-pods
+status:
+  conditions:
+  - lastTransitionTime: "2021-06-30T17:04:07Z"
+    message: Success
+    status: "True"
+    type: Success
+ ...
+----