-
Notifications
You must be signed in to change notification settings - Fork 75
Actions and Handling Special Cases
This article will cover use cases around actions that will be considered special cases and have different behaviors than default. Action types include Rightsizing, Moves, Scaling.
Turbonomic will have out-of-the-box behavior treatment for resizing workloads based on the following scenarios:
- Side-cars and resizing automation
- Operator-controlled workloads and resizing automation
- System namespace workloads and generating actions
Side-cars are typically injected, and the limits/requests are controlled by another workload controller, and not the parent of the pod that the side car was injected into. Therefore an action to update a parent controller would fail because the side-car spec is not there. To handle this, Turbonomic will automatically discover side-car container specs, create a group, and set a default policy to have resize actions to recommend only. This allows the resizing of other container specs that are not side cars to execute.
- Group name: "Injected Sidecars/All ContainerSpecs" (one per cluster)
- Fefault policy name: "Injected Sidecars/All ContainerSpecs Resize Recommend Only [k8s cluster name]"
Starting in 8.10.6 KubeTurbo will auto-create Container Spec groups for workloads controlled by an Operator by default and create a new default policy that sets those resize actions to recommend only so they cannot be automated. Additionally if someone tries to create a policy to automate those resize actions we prevent/block that policy from being created.
- Group name: "Operator Controlled ContainerSpecs" (one per cluster)
- Default policy name: "Operator Controlled ContainerSpecs Resize Recommend Only [k8s cluster name]"
If you want to allow specific workloads that are controlled by an Operator to generate resize actions, you can add them to the configmap
. The following example shows how to modify the kubeturbo
Custom Resource, called kubeturbo-release
in this example, which is part of the Operator Hub / Operator deployment method. You can use one or both of two methods to exclude workloads
- Workload names =
operatorControlledWorkloadsPatterns
- Namespace names =
operatorControlledNamespacePatterns
kubectl edit kubeturbo kubeturbo-release
spec:
exclusionDetectors:
operatorControlledWorkloadsPatterns:
- turbon.*
- testing-.*
operatorControlledNamespacePatterns:
- turbonomic
- gke-.*
After applying the updates, to verify they have be applied correctly there are two ways to confirm this.
First check the configmap
to verify the changes were applied, see example below:
kubectl get configmap turbo-config-kubeturbo-release -o yaml
Check for the section turbo-autoreload.config
to see the updates applied above
apiVersion: v1
data:
turbo-autoreload.config: |-
{
...
},
"exclusionDetectors": {
"operatorControlledWorkloadsPatterns": ["turbon.*","testing-.*"],
"operatorControlledNamespacePatterns": ["turbonomic","gke-.*"]
}
}
Second check the kubeturbo logs to ensure that it has picked up the changes correctly, look for the following entries in the logs:
kubectl logs kubeturbo-release-7cb4b886c5-glv9b | grep exclusion
Output will look similar to the below:
I1122 14:43:07.603989 dynamic_config.go:119] Operator controlled workload exclusion set to: []
I1122 14:43:07.604013 dynamic_config.go:125] Operator controlled namespace exclusion set to: [turbonomic]
I1122 19:14:52.314970 dynamic_config.go:119] Operator controlled workload exclusion set to: [turbon.* testing-.*]
I1122 19:14:52.315096 dynamic_config.go:125] Operator controlled namespace exclusion set to: [turbonomic gke-.*]
Starting in 8.10.6 there is a default behavior for workloads discovered in a set of well-known system namespaces will automatically have resize actions disabled. There will be a new setting in the KubeTurbo configmap
to create a new group and policy to disable resize actions for Container Spec workloads in the "System namespaces" such as those starting with kube-.*
, openshift-.*
, cattle.*
:
- Group name: "System Namespaced ContainerSpecs" (one per cluster)
- Fefault policy name: "System Namespaced ContainerSpecs Resize Disabled [k8s cluster name]"
To apply this configuration to the configmap
, the following example shows how to modify the kubeturbo
Custom Resource, called kubeturbo-release
in this example, which is part of the Operator Hub / Operator deployment method. You will identify Namespace name patterns, which when detected, will place all the discovered workloads in these namespaces into a group that will disable resize actions from generating.
kubectl edit kubeturbo kubeturbo-release
spec:
systemWorkloadDetectors:
namespacePatterns:
- kube-.*
- openshift-.*
- cattle.*
After applying the updates, to verify they have be applied correctly there are two ways to confirm this.
First check the configmap
to verify the changes were applied, see example below:
kubectl get configmap turbo-config-kubeturbo-release -o yaml
Check for the section turbo-autoreload.config
to see the updates applied above
apiVersion: v1
data:
turbo-autoreload.config: |-
{
...
},
"systemWorkloadDetectors": {
"namespacePatterns": ["kube-.*","openshift-.*","cattle.*"]
}
}
Second check the kubeturbo logs to ensure that it has picked up the changes correctly, look for the following entries in the logs:
kubectl logs kubeturbo-release-7cb4b886c5-glv9b | grep -i "namespace det"
Output will look similar to the below:
I1122 14:43:07.603966 dynamic_config.go:113] System Namespace detectors set to: [kube-.*]
I1122 19:14:52.314883 dynamic_config.go:113] System Namespace detectors set to: [kube-.* openshift-.* cattle.*]
As of 8.10.1 we define a default min of 1 and max of 1000 setting for all discovered MachineSet NodePools in an OpenShift cluster only. This is defined in the configMap
as shown below:
data:
turbo-autoreload.config: |-
{
"nodePoolSize": {
"min": 1,
"max": 1000
You can change the default values to any values you like in each KubeTurbo deployment by updating the KubeTurbo deployment:
kubectl edit kubeturbo kubeturbo-release
spec:
nodePoolSize:
max: 100
min: 2
After applying the updates, verify they have be applied correctly by checking the configmap
to verify the changes were applied, see example below:
kubectl get configmap turbo-config-kubeturbo-release -o yaml
data:
turbo-autoreload.config: |-
{
"nodePoolSize": {
"min": 2,
"max": 100
We will be expanding this to be for each discovered NodePool outside of OpenShift and MachineSets in OpenShift and have a policy setting in the UI that this can be configured in.
Introduction
Kubeturbo Use Cases
Kubeturbo Deployment
Kubeturbo Config Details and Custom Configurations
Actions and how to leverage them
- Overview
-
Resizing or Vertical Scaling of Containerized Workloads
a. DeploymentConfigs with manual triggers in OpenShift Environments - Node Provision and Suspend (Cluster Scaling)
- SLO Horizontal Scaling
- Turbonomic Pod Moves (continuous rescheduling)
-
Pod move action technical details
a. Red Hat Openshift Environments
b. Pods with PVs
IBM Cloud Pak for Data & Kubeturbo:Evaluation Edition
Troubleshooting
- Startup and Connectivity Issues
- KubeTurbo Health Notification
- Logging: kubeturbo log collection and configuration options
- Startup or Validation Issues
- Stitching Issues
- Data Collection Issues
- Collect data for investigating Kubernetes deployment issue
- Changes to Cluster Role Names and Cluster Role Binding Names