[Feature][Design] KRM KCL Spec & User Story

[Peefy]

Introduction

KCL is specifically used for configuration writing and policy validation in cloud-native and Kubernetes scenarios. However, it is obviously not enough to use only one language for Kubernetes configuration management. KCL needs to be integrated with more scenarios and specifications to make it easier and more standardized for users to use KCL in their specific scenarios.

KCL was initially used for modeling, and Kubernetes naturally provided the KRM model. We can reuse existing resources and standardize them in KCL's registry using tools such as KCL OpenAPI and KPM. For example, users can reuse existing resources and abstract their own models (building models such as containers, services, computing, storage, and networks) and reuse them.

Overview

This issue is mainly used to clarify user stories and unified specifications of various KCL scenario models mainly for KRM. (👀 TBD: Perhaps we can give it a better name instead of KRM KCL Spec...)

Pain Points

• YAML and template bloat.
• Customized needs: Helm charts conditional configuration, validating, editing, and overlaying such as network labels and annotations, multiple tenants, and multiple environments.
• KCL language has hands-on costs and lacks overall product/process integration. (Abstraction + Deploy + ...)

See Appendix for more

User Story

User Story I (Day1 Deployment)

As SRE/Ops (who), ... I want (what), because (why) ...

I hope to achieve fast and reliable application deployment and updates through deployment automation, improving deployment efficiency, and reducing error rates. Specifically, I hope to have a simple and easy-to-use automation deployment tool that can help me complete the following tasks:

• Building: I hope that the application's build process can be triggered automatically after the code is submitted to the Version Control System (VCS), and generate deployable program packages.
• Testing: I hope that after the program package is generated, it can automatically perform various tests such as unit testing, integration testing, and functional testing to ensure program quality.
• Deployment: I hope that the program package can be quickly and reliably deployed to target servers or cloud platforms through automation tools, reducing manual operations and error rates.
• Environments: I hope that I can easily manage the differences between different environments and clusters, such as the development environment, testing environment, and production environment, to ensure deployment consistency and reliability through automation tools.
• Monitoring: I hope that I can monitor various indicators during the deployment process in real-time through automation tools, and quickly roll back to the previous version to deal with exceptions in a timely manner.

I hope this automation deployment tool can support various application programming languages and frameworks, as well as multiple deployment targets such as virtual machines, containers, and cloud platforms. Additionally, I hope this tool can be integrated with other operations and maintenance tools to achieve a comprehensive automation operations and maintenance chain.

User Story II (Day2 Operation)

As SRE/Ops (who), ... I want (what), because (why) ...

Here are some examples of Day2 operations for SRE:

• Monitor system health: Continuously monitor system performance, availability, and functionality to detect any issues and promptly resolve them.
• Incident response: Investigating and resolving incidents as they occur, including identifying root causes and implementing mitigating actions.
• Capacity planning: Analyzing system resource usage and forecasting future demand to ensure that sufficient resources are available to meet business needs.
• Security management: Monitoring and managing security risks to ensure that systems and data are protected from threats.
• Change management: Ensuring that any changes to the system are fully tested and evaluated before they are implemented, to minimize the risk of service disruption. (e.g., Canary, Blue Green, Horizontal, Vertical, etc.).
  • operate on multi-clusters with good experience, just like a single cluster
  • apply progressive changes to workloads(like 1%->5%->10%->50%->100%), cluster by cluster.
  • apply progressive rollback, cluster by cluster.
  • for batched apps, apply changes to one batch at the same time. (optional)
  • for batched apps, control the workflow of each batch, like pause, resume, stop, etc. (optional)
  • for each app, control the workflow of each step, like pause, resume, stop, etc. (optional)
  • for each app, control the workflow of serving ingress traffic(like 1%->5%->10%->50%->100%) (optional)
  • continuously detect and manage configuration drifts
  • Automation: Developing and implementing automated processes to reduce the risk of human error and improve efficiency.
• Training and development: Providing training and development opportunities to team members to ensure that they have the skills and knowledge required to perform their roles effectively.

User Story III

As application developers (who), ... I want (what), because (why) ...

The most important concern for application developers might be to devote maximum their attention to the functional-related development, and to shorter the development-validation cycles, which bring about continuous progressive product enhancement.

User Story IV

As platform developers (who), ... I want (what), because (why) ...

• Code: install necessary functional module as needed
• Change: Change the code, push, and merge the code.
• Ship: With a faster way to ship(within about 3 steps), for most cloud native resources (such as deployment, services, etc.) and cloud resources (such as storage), I do not need to write code or only write a small amount of code.

Solution

• By using KCL and VCS/CI/CD integration to solve automation process problems.
• Designing KCL models and KCL characteristics to address efficiency and stability issues such as deployment, environment construction, and monitoring configuration.

Design

KRM KCL Spec

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  annotations: 
    krm.kcl.dev/version: 0.0.1
    krm.kcl.dev/type: abstraction # abstraction, mutation, validation
    documentation: >-
        KRM KCL Documents
# EDIT THE SOURCE!
# This should be your KCL code which preloads the `ResourceList` to `option("resource_list").
spec:
  # code source
  params:
     param1: param_value1
     param2: param_value2
  source: |
    # Add annotations
    [resource | {if resource.kind == "Deployment": metadata.annotations: {"managed-by" = "helmfile-kcl"}} for resource in option("resource_list").items]
  # source: path/to/file
  # source: oci://
  # source: git://
  # source: https://

The above specifications are the same for any cloud-native tool and K8s controller that integrates KCL (e.g., Helm KCL Plugin, Kustomize KCL Plugin). The usage of the specifications can be mainly divided into three categories: mutation, validation, and abstraction. Through KRM-KCL, we can generate YAML using KCL, and perform mutation or validation based on the stock of YAML.

KRM KCL Function Spec

A KRM KCL function contains as follows

where:

• input items: The input list of KRM resources to operate on.
• output items: The output list obtained from adding, removing, or modifying items in the input.
• functionConfig: An optional meta resource containing the arguments to this invocation of the function.
• results: An optional meta resource emitted by the function for observability and debugging purposes.

The spec definition (WIP: v1alpha1)

schema KCLRun:
    apiVersion: "krm.kcl.dev/v1alpha1" | "krm.kcl.dev/v1beta1" | "krm.kcl.dev/v1" = "krm.kcl.dev/v1alpha1" # The resource API version
    kind: "KCLRun" = "KCLRun" # The resource type
    spec: KCLRunSpec # The resource spec
    status?: KCLRunStatus # The resource status

schema KCLRunSpec:
    source: str # KCL code source, which can come from local files, git repositories, or OCI source code
    params: {str:} # Params: Input parameters of the model.

schema KCLRunStatus:
    message: str  # Diagnostic message for running KCL.
    severity: "error" | "warning" | "info"  # Diagnostic type for running KCL.
    resourceRef?: ResourceRef  # Which KRM and KCL execution encountered problems?

schema ResourceRef:
    apiVersion: str
    kind: str
    name: str
    namespace: str

Examples

• Mutation

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: set-annotations
  metadata:
    annotations: 
      krm.kcl.dev/version: 0.0.1
      krm.kcl.dev/type: mutation
      documentation: >-
        Add or change annotations
spec:
  params:
    toAdd: addValue
  source: oci://ghcr.io/kcl-lang/set-annotation

• Validation

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: https-only
  metadata:
    annotations: 
      krm.kcl.dev/version: 0.0.1
      krm.kcl.dev/type: validation
      documentation: >-
        Requires Ingress resources to be HTTPS only.  Ingress resources must
        include the `kubernetes.io/ingress.allow-http` annotation, set to `false`.
        By default a valid TLS {} configuration is required, this can be made
        optional by setting the `tlsOptional` parameter to `true`.
        More info: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
spec:
  source: oci://ghcr.io/kcl-lang/https-only

• Abstraction

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: web-service
  metadata:
    annotations: 
      krm.kcl.dev/version: 0.0.1
      krm.kcl.dev/type: abstraction
      documentation: >-
        Web service application abstraction
spec:
  params:
    name: app
    containers:
      ngnix:
        image: ngnix
        ports:
          containerPort: 80
    labels:
      name: app
  source: oci://ghcr.io/kcl-lang/web-service

The output is

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    name: app
spec:
  selector:
    matchLabels:
      name: app
  template:
    metadata:
      labels:
        name: app
    spec:
      containers:
        - name: ngnix
          image: ngnix
          ports:
            containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: app
  labels:
    name: app
spec:
  selector:
    name: app
  ports:
    - port: 80

Components

• Arch

• Can be used both on the client and at runtime.

How to (WIP)

Use IDE products and CLI tools such as KPM as a unified interface for user interaction. At the basic feature level, CLI/IDE/Dashboard is equivalent

Take CLI as an example

Run

• kubectl kcl apply config.yaml
• kustomize fn config.yaml
• kpt fn
• helmfile kcl template
• kusion apply
• ...

Code

• kpm init
• kpm push

Doc

• kpm doc

Check

Lint

• kpm lint

Test

• kpm test

Import

Convert various data/DSL configurations and schemas of inventory into KCL

• kpm import
  • Data: JSON/YAML/TOML/...
  • Schema: Protobuf/Terraform/JsonSchema/...

KRM KCL Implementation

• https://github.com/kcl-lang/krm-kcl

KRM KCL Examples

• Mutation

https://github.com/kcl-lang/krm-kcl/tree/main/examples/mutation

• Validation

https://github.com/kcl-lang/krm-kcl/tree/main/examples/validation

• Abstraction

https://github.com/kcl-lang/krm-kcl/tree/main/examples/abstraction

Further More

Automation & GitOps

Moreover, the things defined in the above specifications can still be used in Dev-CI-CD and other operation and maintenance automation processes in combination with IDE and other products, such as overall integration: Gitlab (Dev) - Jenkins (CI) - Harbor (Registry) - KRM-KCL Spec (Config) - ArgoCD (CD) - Kubernetes (Runtime).

Config

• Use KRM directly (K8s Raw manifests, Kustomize, Helm Chart, ...)
• Use KCL Directly (Without Abstraction)
• Use KCL Directly (With Abstraction)
• KRM + KCL with KRM KCL Spec
• KPM/KCLOpenAPI import existing data/schema to KCL

Regardless of the configuration method used, they are all friendly to the operation and maintenance automation process.

Data/Schema Migration

• Schema: [Feature][Track] KPM Import Tool #526
• Data: API: support generate KCL configuration and schema from json/yaml #530

Appendix

Real feedback

• Overall Product Integration

• Unhappy Path Usage

• I do not like yaml

• Multiple Environments and Namespaces

• Conditional Expressions in Helm values.yaml

Reference

• The Kubernetes Resource Model: https://docs.google.com/document/d/1RmHXdLhNbyOWPW_AtnnowaRfGejw-qlKQIuLKQWlwzs/edit#
• KRM Function: https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/api-conventions/functions-spec.md#krm-functions-specification
• KPT: https://github.com/GoogleContainerTools/kpt
• KPT KCL SDK Issue: [Feature Request] Support functions written by KCL and its SDKs for KPT  kptdev/kpt#3671
• kpt-functions-catalog: https://github.com/GoogleContainerTools/kpt-functions-catalog
• A KPT go function catalog: https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/master/functions/go/set-namespace
• Kustomize: https://github.com/kubernetes-sigs/kustomize
• Build Custom Kustomize Plugin With Containerized KRM function: https://thirumurthi.hashnode.dev/build-custom-kustomize-plugin-with-containerized-krm-function
• Helm: https://github.com/helm/helm
• Helm Schema Generation Plugin: https://github.com/holgerjh/helm-schema
• Helm Diff Plugin: https://github.com/databus23/helm-diff
• Helm S3 Plugin: https://github.com/hypnoglow/helm-s3
• Validating Helm Chart Values with JSON Schemas: https://www.arthurkoziel.com/validate-helm-chart-values-with-json-schemas/
• Score Helm Chart: https://github.com/score-spec/score-helm-charts
• Helmfile: https://github.com/helmfile/helmfile
• Bridgekeeper: https://github.com/MaibornWolff/bridgekeeper
• OPA Gatekeeper: https://github.com/open-policy-agent/gatekeeper
• Crossplane: https://github.com/crossplane/crossplane
• keptn: https://github.com/keptn/keptn
• Kubernetes Resource Model (KRM): https://github.com/kubernetes/design-proposals-archive/blob/main/architecture/resource-management.md
• ArgoCD: https://github.com/argoproj/argo-cd
• Terragrunt: https://terragrunt.gruntwork.io/
• Grafana Helm charts: https://github.com/grafana/helm-charts
• Grafana Jsonnet libs: https://github.com/grafana/jsonnet-libs
• Grafana Tanka based on Jsonnet: https://github.com/grafana/tanka
• Grafana Tanka Helm integration: https://tanka.dev/helm#helm-support
• Prometheus Kubernetes Manifests based on Jsonnet: https://github.com/prometheus-operator/kube-prometheus
• https://www.reddit.com/r/kubernetes/comments/118bmwu/helm_or_kustomize_for_my_situation/
• https://www.reddit.com/r/kubernetes/comments/119jolm/separate_repository_for_helm_charts_vs_integrated/
• https://www.reddit.com/r/kubernetes/comments/xx0e6h/why_we_use_cue_and_not_helm/
• Why we use cue not helm: https://cloudplane.org/blog/why-cue
• External Secret: https://external-secrets.io/v0.7.2/
• Kubescape: https://github.com/kubescape/kubescape
• KPT Starlark function: https://github.com/GoogleContainerTools/kpt-functions-catalog/blob/master/functions/go/starlark/starlark/config.go
• https://github.com/FairwindsOps/polaris
• Lingon Comparision: https://github.com/volvo-cars/lingon/blob/main/docs/comparison.md
• KCL Operator: https://github.com/KusionStack/kcl-operator
• Kubebuilder: https://github.com/kubernetes-sigs/kubebuilder
• Operator SDK tool: https://sdk.operatorframework.io/docs/building-operators/golang/quickstart/
• How to: KusionStack end-user use cases and examples: https://github.com/KusionStack/examples/issues/6
• How to: A typical workflow from an app developer's perspective: https://github.com/KusionStack/examples/issues/7
• How to: A typical workflow from a platform developer's perspective: https://github.com/KusionStack/examples/issues/8
• Cloud Development - Day1/Day2: https://blog.brainboard.co/infography-cloud-development-day-1-day-2-9b5636a42e6b

[Peefy]

@ekkoful Hi, here. I hope we can participate in improving this content together. ❤️ Ultimately, we use a unified specification to support client and runtime KCL integration tools.

[ekkoful]

@ekkoful Hi, here. I hope we can participate in improving this content together. heart Ultimately, we use a unified specification to support client and runtime KCL integration tools.

Yeah. Good job. It's a good arch pic for developer.

[ekkoful]

I plan to implement the KCLRun kind resource for kubernetes resource. There is the basic design. cc: @Peefy

Note: The KCLRun kind maybe update the kubernetes resource, like pod, deployment...

[Peefy]

I plan to implement the KCLRun kind resource for kubernetes resource. There is the basic design. cc: @Peefy

Note: The KCLRun kind maybe update the kubernetes resource, like pod, deployment...

OK, no problem!

[Peefy]

I plan to implement the KCLRun kind resource for kubernetes resource. There is the basic design. cc: @Peefy Note: The KCLRun kind maybe update the kubernetes resource, like pod, deployment...

OK, no problem!

What needs to be added is:

• KCLRun Spec defines how to abstract, mutate, and validate this resource
• The KCL Operator obtains the resources to be operated through Kubernetes Web hook and then calls the krm-kcl component for operation. For example, a classic example is the injection of network sidecars, such as istio