Skip to content

Latest commit

 

History

History
232 lines (194 loc) · 11.1 KB

KustomizeBestPractices.md

File metadata and controls

232 lines (194 loc) · 11.1 KB
Raw

Kustomize Best Practices

How to utilize kustomize processing directives to minimize yaml errors and simplify yaml resources.

1. Identify the resources that encompass a kustomize target

In many cases there are resources along what is below:

  • CustomResourceDefinition
  • ClusterRole, ClusterRoleBinding or Role, RoleBinding
  • ConfigMap
  • Deployment
  • Service
  • ServiceAccount
  • VirtualService

This collection of resources should by grouped under a particular parent folder (eg tf-job-operator, etc), within a subfolder 'base'.

1a. Resource file naming

Resources should be organized by kind, where the resource is in a file that is the lower-case hyphenized form of the Resource kind. For example: a Deployment would go in a file named deployment.yaml. A ClusterRoleBinding would go in a file called cluster-role-binding.yaml. If there are multiple resources within a kustomize target (eg more than one deployment), you may want to maintain a single resource per file and add a prefix|suffix of the resource name to the filename. For example the file name would be <kind>-<name>.yaml. See below for an example.

example: /manifests/profiles

profiles
└── base
    ├── README.md
    ├── cluster-role-binding.yaml
    ├── crd.yaml
    ├── deployment.yaml
    ├── kustomization.yaml
    ├── role-binding.yaml
    ├── role.yaml
    ├── service-account.yaml
    └── service.yaml

1b. Removing common attributes across resources

There are often repeated attributes across resources: labels, namespace, or perhaps a common prefix used for each resource. You can move name prefixes into the kustomization.yaml file and then make adjustments within each resource; removing the prefix from its name. Additionaly you can move labels and their selectors into the kustomization.yaml. Yo can move the namespace into the kustomization.yaml. All of these will be added back into the resource by running kustomize build.

example: /manifests/profiles/base/kustomization.yaml. Contains namespace, nameprefix, commonLabels.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- crd.yaml
- service-account.yaml
- cluster-role-binding.yaml
- role.yaml
- role-binding.yaml
- service.yaml
- deployment.yaml
namespace: kubeflow
namePrefix: profiles-
commonLabels:
  kustomize.component: profiles
images:
  - name: gcr.io/kubeflow-images-public/profile-controller
    newName: gcr.io/kubeflow-images-public/profile-controller
    newTag: v20190228-v0.4.0-rc.1-192-g1a802656-dirty-f95773

The original deployment in profiles looked like:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    kustomize.component: profiles
  name: profiles-deployment
  namespace: kubeflow
spec:
  selector:
    matchLabels:
      kustomize.component: profiles
  template:
    metadata:
      labels:
        kustomize.component: profiles
    spec:
      containers:
      - command:
        - /manager
        image: gcr.io/kubeflow-images-public/profile-controller:v20190228-v0.4.0-rc.1-192-g1a802656-dirty-f95773
        imagePullPolicy: Always
        name: manager
      serviceAccountName: profiles-controller-service-account

Moving labels, namespace and the nameprefix 'profiles-' to kustomization.yaml reduces deployment.yaml to

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment
spec:
  template:
    spec:
      containers:
      - name: manager
        command:
        - /manager
        image: gcr.io/kubeflow-images-public/profile-controller:v20190228-v0.4.0-rc.1-192-g1a802656-dirty-f95773
        imagePullPolicy: Always
      serviceAccountName: controller-service-account

Note: A kustomize target should always 'build', so you should add what's needed to allow a kustomize build to succeed (and for unittests to work). Defining a namespace in kustomization.yaml is required to run kustomize build, even though there is a namespace override in the parent kustomization.yaml generated by kfctl under /manifests/profiles. This generated kustomization.yaml provides overrides using values from app.yaml and will appear within the manifest cache after running kfctl generate....

1c. Parameters

There are 3 types of kustomize parameters that can be leveraged when building a kustomize target. The vars, patchesStrategicMerge and patchesJson6209 are described in detail in the kustomization.yaml file.

  1. vars
    Kustomize vars should be used whenever a simple text replacement is needed. The text must be a string, for example you cannot use kustomize vars for deployment.spec.replicas which takes a int. You will need to use a Json Patch for this. Kustomize vars require a section within the kustomization.yaml under vars: that identifies the variable name, its object reference and optionally its field reference in the object. This uniquely identifies the source of the variable. A configuration yaml may also be needed to identify those resources referencing the variable. In many cases the variable will reference a ConfigMap that is generated in the kustomization.yaml file using values contained in a params.env file. The configuration yaml will then list resources and their json paths where the variable is used eg $(varname). See ambassador for an example of using a var named ambassadorServiceType that is generated from a configMapGenerator which reads this value from params.env. In order to use ambassadorServiceType, you need to list which resource uses it in params.yaml and finally, actually use the variable.
  2. patchesStrategicMerge
    This contains a list of resources which have one or more changes from the base resource. Only the changes and enough information to uniquely describe the resource should be in this resource file. This should not be used if you're changing an array in the base resource. For this you should use a Json Patch.
  3. patchesJson6902
    Json Patches are the most versatile, having the ability to add, replace or remove array entries in resources (eg deployment/spec/template/spec/containers/env/[-+]) as well as other sections that may be parts of the schema or integers. [Ambassador]'s kustomization.yaml also contains an example of a json patch that replaces an integer using the patch found in deployment-amabassador-patch.yaml in the ambassador base directory.

1d. Overlays

Certain resources or resource modifications can be further grouped by a particular concept that cuts across kustomize base targets such as platform type, an Istio service, basic-auth, etc. There are a number of examples where base targets are changed by specific overlays and can be used as a reference when adding a new target or overlay. One example of an overlay is under profiles debug overlay. In this case, there is a need to debug the profile-controller using dlv/GoLand. This requires replacing the deployment command with '/go/bin/dlv' and adding args required by dlv. It also requires replacing the image with one that has been modified and matches with the debugger source code.

- Invoking overlays

Overlays can be used directly by running kustomize build at the overlay subdirectory. In most cases overlays are invoked by kfctl by specifying the overlay to call within the app.yaml. An example of invoking the debug overlay is shown below:

apiVersion: kfdef.apps.kubeflow.org/v1alpha1
kind: KfDef
metadata:
  creationTimestamp: null
  labels:
    app.kubernetes.io/name: experiments
  name: experiments
  namespace: experiments
spec:
  appdir: /Users/kdkasrav/experiments
  componentParams:
    profiles:
    - name: overlay
      value: debug
    - name: project
  components:
  - profiles
  manifestsRepo: /Users/kdkasrav/experiments/.cache/manifests/pull/31/head
  packageManager: kustomize@pull/31
  packages:
  - profiles
  project: constant-cubist-173123
  repo: /Users/kdkasrav/experiments/.cache/kubeflow/pull/3108/head/kubeflow
  useBasicAuth: false
  useIstio: false
  version: pull/3108
status: {}

Calling kfctl generate all will produce the profiles resources with a modified Deployment that allows remotely attach to dlv running the profile-controller .

- Invoking multiple overlays

Using profiles as the use case

profiles
├── base
│   └── kustomization.yaml
└── overlays
    ├── debug
    │   └── kustomization.yaml
    └── devices
        └── kustomization.yaml

Normally kustomize provides the ability to overlay a 'base' set of resources with changes that are merged into the base from resources that are located under an overlays directory. Kustomize doesn't provide for an easy way to combine more than one overlay for example debug and devices. However kustomize has several feature requests to allow this and combining overlays is possible if one were to create a kustomization.yaml under profiles that included base and the kustomization directives in debug and devices. This requires some path corrections and some constraints within each kustomization but allows 'mixins' to be used in the manner described above. In order to 'mixin' debug and devices the app.yaml would add both overlays in the profiles componentParams as shown below

  componentParams:
    profiles:
    - name: overlay
      value: debug
    - name: overlay
      value: devices

The kustomization.yaml generated in .cache/manifests/master/profiles/ looks like

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- base
commonLabels:
  app.kubernetes.io/name: experiments
configMapGenerator:
- env: overlays/debug/params.env
  name: parameters
configurations:
- overlays/debug/params.yaml
generatorOptions:
  disableNameSuffixHash: true
namespace: experiments
patchesStrategicMerge:
- overlays/debug/deployment.yaml
- overlays/devices/deployment.yaml
vars:
- fieldref:
    fieldPath: data.project
  name: project
  objref:
    apiVersion: v1
    kind: ConfigMap
    name: parameters
- fieldref:
    fieldPath: data.namespace
  name: namespace
  objref:
    apiVersion: v1
    kind: ConfigMap
    name: parameters