Skip to content

Commit

Permalink
New Kapitan Reference setup (#141)
Browse files Browse the repository at this point in the history
* working

* Enable dynamic load for generator classes
  • Loading branch information
ademariag authored Aug 27, 2023
1 parent 370049f commit 3a5b21b
Show file tree
Hide file tree
Showing 589 changed files with 6,728 additions and 17,146 deletions.
5 changes: 4 additions & 1 deletion .github/workflows/integration-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,8 @@ jobs:
with:
cluster_name: kind

- name: Check connectivity to the cluster
run: ./compiled/tutorial/scripts/kubectl get pods

- name: Run chart-testing (install)
run: ./compiled/tutorial/scripts/apply.sh
run: ./compiled/tutorial/scripts/apply
18 changes: 15 additions & 3 deletions .kapitan
Original file line number Diff line number Diff line change
@@ -1,4 +1,16 @@
version: 0.31
version: 0.32
compile:
prune: true
embed-refs: true
prune: true
embed-refs: true
fetch: true
yaml-dump-null-as-empty: true
compose-node-name: true
refs-path: ./system/refs
jinja2-filters: ./system/lib/jinja2_filters.py
search-paths:
- .
- ./system/
- ./system/lib
- ./system/generators
refs:
refs-path: ./system/refs
10 changes: 5 additions & 5 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,26 +1,26 @@
exclude: ^compiled/|^components/charts/
exclude: ^compiled/|^system/sources/
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
rev: v4.4.0
hooks:
- id: trailing-whitespace
- id: check-docstring-first
- id: check-json
- id: check-yaml
exclude: |
(?x)(
^Docs/|
^system/sources/|
)
- id: debug-statements
- id: name-tests-test
- id: requirements-txt-fixer
- id: check-merge-conflict
- repo: https://github.com/psf/black
rev: 22.8.0
rev: 23.7.0
hooks:
- id: black
- repo: https://github.com/PyCQA/isort
rev: 5.10.1
rev: 5.12.0
hooks:
- id: isort
args: ["--profile", "black"]
151 changes: 60 additions & 91 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,23 +11,30 @@ $ git clone git@github.com:kapicorp/kapitan-reference.git kapitan-templates
$ cd kapitan-templates

$ ./kapitan compile
Compiled postgres-proxy (1.51s)
Compiled tesoro (1.70s)
Compiled echo-server (1.64s)
Compiled mysql (1.67s)
Compiled gke-pvm-killer (1.17s)
Compiled prod-sockshop (4.74s)
Compiled dev-sockshop (4.74s)
Compiled tutorial (1.68s)
Compiled global (0.76s)
Compiled examples (2.60s)
Compiled pritunl (2.03s)
Compiled sock-shop (4.36s)
Rendered inventory (3.45s)
Compiled pritunl (0.23s)
Compiled vault (0.27s)
Compiled examples (0.28s)
Compiled gke-pvm-killer (0.10s)
Compiled mysql (0.10s)
Compiled postgres-proxy (0.11s)
Compiled sock-shop (0.23s)
Compiled echo-server (0.11s)
Compiled global (0.09s)
Compiled guestbook-argocd (0.12s)
Compiled tutorial (0.15s)
Compiled kapicorp-project-123 (0.09s)
Compiled kapicorp-terraform-admin (0.10s)
Compiled tesoro (0.13s)
Compiled dev-sockshop (0.24s)
Compiled prod-sockshop (0.27s)
Compiled argocd (0.99s)
Compiled github-actions (6.99s)
```

## Slow walk-through
## Generators documentation (IN PROGRESS)

[Manifest Generator Documentation](components/generators/kubernetes/README.md)
[generators.kapitan.dev](https://generators.kapitan.dev/)

### Tools

Expand All @@ -37,95 +44,57 @@ For now, you can see that the [`./kapitan`](kapitan) file is a wrapper script th

*Note*: For speed, if kapitan is already installed, it will prefer the non-docker version.

| Script | Description |
| ------ | ----------- |
| Script | Description |
|-----------|----------------------------------|
| ./kapitan | Wrapper script to invoke kapitan |
| [generate_sa_secrets.sh](templates/scripts/generate_sa_secrets.sh) | Templated script to automatically inject service accounts into refs |
| [import_kubernetes_clusters](scripts/import_kubernetes_clusters) | Helper scripts that looks for GKE cluster and automatically imports them into the inventory |


### Libraries

This repo already packs some important libraries that you will want to have when working with kapitan.

| Name | Description | Inventory file |
| ---- | ----------- | -------------- |
| [kube-libsonnet](https://github.com/bitnami-labs/kube-libsonnet) | bitnami-labs kube library | [kube.yml](inventory/classes/kapitan/kube.yml) |
| [kubernetes-generator](components/generators/kubernetes) | [Synthace](www.synthace.com) manifests generator | [generators/kubernetes.yml](inventory/classes/kapitan/generators/kubernetes.yml)|
| [ingresses-generator](components/generators/ingresses) | [Synthace](www.synthace.com) ingresses generator | [generators/ingresses.yml](inventory/classes/kapitan/generators/ingresses.yml)|
| [utils](lib/utils.libsonnet) | helpful utilites ||
| [kap](lib/kap.libsonnet) | Kapitan boilerplate in one file ||
|kgenlib| Kapitan Generators SKD | [kgenlib.yml](inventory/classes/kapitan/kgenlib.yml)


Kapitan allows you to manage external dependencies like the above libraries.
For instance, in the [spinnaker.yml](inventory/classes/kapitan/spinnaker.yml) file, the "dependencies" directive tells Kapitan where to find the library.

To update them, run:
This repo enables fetching by default through the ``.kapitan` file, which only fetches non existing dependencies.

```shell script
./kapitan compile --fetch
Dependency lib/kube.libjsonnet : already exists. Ignoring
./kapitan compiledd
Compiled tesoro (1.70s)
Compiled echo-server (1.64s)
Compiled mysql (1.67s)
Compiled gke-pvm-killer (1.17s)
Compiled prod-sockshop (4.74s)
Compiled dev-sockshop (4.74s)
Compiled tutorial (1.68s)
Compiled global (0.76s)
Compiled examples (2.60s)
Compiled pritunl (2.03s)
Compiled sock-shop (4.36s)
```
version: 0.32
compile:
prune: true
embed-refs: true
fetch: true
```

## Generators

As explained in the blog post [Keep your ship together with Kapitan](https://medium.com/kapitan-blog/keep-your-ship-together-with-kapitan-d82d441cc3e7). generators are a
powerful idea to simplify the management your setup.

We will release initially generators for kubernetes manifests, terraform and spinnaker pipelines.

For now, only the `manifests` and `ingresses` generators are available

### Manifests generator

The `manifests` generator allows you to quickly generate Kubernetes manifests from a much simpler yaml configuration.

The aim for this approach is to allow you to cover the vast majority of the needs you will have for your components.
More complex scenarios can also be achieved by expanding the library, or implementing your own template.

### Examples
To help you get started, please look at the following examples:

| source | description | output |
| ------ | ----------- | ------ |
|[mysql](inventory/classes/components/mysql.yml)| Example MySQL statefulset | [manifests](compiled/mysql/manifests)|
|[echo-server](inventory/classes/components/echo-server.yml)| Example using [echo-server](https://github.com/jmalloc/echo-server) | [manifests](compiled/echo-server/manifests)|
|[gke-pvm-killer](inventory/classes/components/gke-pvm-killer.yml)| Example using [estafette-gke-preemptible-killer](https://github.com/estafette/estafette-gke-preemptible-killer)| [manifests](compiled/gke-pvm-killer/manifests)|
|[postgres-proxy](inventory/classes/components/postgres-proxy.yml)| Example using [cloud-sql-proxy](https://github.com/GoogleCloudPlatform/cloudsql-proxy) to connect to a Cloud SQL Postgres instance| [manifests](compiled/postgres-proxy/manifests)|
|[logstash](inventory/classes/components/logstash.yml)| Example of [Logstash](https://www.elastic.co/logstash) configuration | [manifests](compiled/examples/manifests)
|[tesoro](inventory/classes/components/kapicorp/tesoro.yml)| Example of [tesoro](https://github.com/kapicorp/tesoro) configuration | [manifests](compiled/tesoro/manifests)
|[pritunl](inventory/classes/components/pritunl/pritunl.yml)| Example of [pritunl](https://pritunl.com/) configuration | [manifests](compiled/pritunl/manifests)



Please find the generated manifests in the [compiled](compiled) folder


### Ingresses generator

The `ingresses` generator adds to the `manifests` generator the ability to easily define ingress resources.

### Examples
To help you get started, please look at the following examples:

| source | description | output |
| ------ | ----------- | ------ |
|[echo-server](inventory/classes/components/echo-server.yml)| Defining ingress paths using [echo-server](https://github.com/jmalloc/echo-server) | [manifests](compiled/echo-server/manifests)|

[Documentation](components/generators/kubernetes/README.md)

### Request or submit your examples
We have used this generator extensively, and we know it covers the majority of the use cases.
If you want a specific example, please let us know (or submit your PR)
To update them from the upstream version, force fetch by running:

By adding more example we will be able to stress test the library to make sure we really satisfy all the most common use cases.
```shell script
./kapitan compile --force-fetch
Dependency https://github.com/kapicorp/generators.git: saved to system/lib
Dependency https://github.com/kapicorp/generators.git: saved to system/generators/kubernetes
Dependency https://github.com/kapicorp/generators.git: saved to system/generators/terraform
Dependency argo-cd: saved to system/sources/charts/argo-cd/argo-cd/3.32.0/v2.2.3
Rendered inventory (3.45s)
Compiled vault (0.27s)
Compiled pritunl (0.27s)
Compiled examples (0.32s)
Compiled gke-pvm-killer (0.10s)
Compiled mysql (0.10s)
Compiled postgres-proxy (0.10s)
Compiled sock-shop (0.23s)
Compiled echo-server (0.11s)
Compiled global (0.09s)
Compiled tutorial (0.14s)
Compiled guestbook-argocd (0.11s)
Compiled kapicorp-project-123 (0.09s)
Compiled kapicorp-terraform-admin (0.09s)
Compiled tesoro (0.13s)
Compiled dev-sockshop (0.24s)
Compiled prod-sockshop (0.27s)
Compiled argocd (0.97s)
Compiled github-actions (7.13s)
```
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
|||
| --- | --- |
| **Target** | argocd |
| **Project** | `not defined`|
| **Project** | `argocd`|
| **Cluster** | 'Not defined' |
| **Namespace** | `argocd` |

Expand Down
12 changes: 6 additions & 6 deletions compiled/argocd/manifests/argo-cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3507,7 +3507,7 @@ spec:
apiVersion: v1
kind: Service
metadata:
annotations: null
annotations:
labels:
app: redis-ha
chart: redis-ha-4.12.17
Expand Down Expand Up @@ -3539,7 +3539,7 @@ spec:
apiVersion: v1
kind: Service
metadata:
annotations: null
annotations:
labels:
app: redis-ha
chart: redis-ha-4.12.17
Expand Down Expand Up @@ -3812,7 +3812,7 @@ spec:
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: argo-cd-redis-ha-haproxy
tolerations: null
tolerations:
volumes:
- configMap:
name: argo-cd-redis-ha-configmap
Expand Down Expand Up @@ -4538,7 +4538,7 @@ spec:
name: data
- mountPath: /health
name: health
- args: null
- args:
env:
- name: REDIS_ADDR
value: redis://localhost:6379
Expand All @@ -4563,7 +4563,7 @@ spec:
periodSeconds: 15
timeoutSeconds: 3
resources: {}
volumeMounts: null
volumeMounts:
initContainers:
- args:
- /readonly-config/init.sh
Expand Down Expand Up @@ -4604,7 +4604,7 @@ spec:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
annotations: null
annotations:
name: data
spec:
accessModes:
Expand Down
57 changes: 57 additions & 0 deletions compiled/argocd/scripts/bash.include
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
set -o nounset -o pipefail -o noclobber -o errexit

TARGET_NAME="argocd"
TARGET_PATH="argocd"
GCP_PROJECT_ID="argocd"
TARGET="argocd"
TARGET_RELATIVE_PATH_FROM_KAPITAN_BASEDIR=compiled/${TARGET_PATH}

SCRIPT_RELATIVE_PATH=${BASH_SOURCE[1]}
SCRIPT_RELATIVE_DIR=$(dirname ${SCRIPT_RELATIVE_PATH})
SCRIPT_ABS_PATH=$(cd "${SCRIPT_RELATIVE_DIR}"; pwd)

KAPITAN_ABSOLUTE_BASEDIR=${SCRIPT_ABS_PATH%${TARGET_RELATIVE_PATH_FROM_KAPITAN_BASEDIR}/*}
SCRIPT_RELATIVE_DIR_FROM_KAPITAN_BASEDIR=$(dirname ${SCRIPT_ABS_PATH#${KAPITAN_ABSOLUTE_BASEDIR}/})

GIT_ABSOLUTE_BASEDIR=$(cd ${KAPITAN_ABSOLUTE_BASEDIR}; git rev-parse --show-toplevel)
KAPITAN_BASEDIR_RELATIVE_PATH_FROM_GIT_BASEDIR=$(realpath ${KAPITAN_ABSOLUTE_BASEDIR} --relative-to=${GIT_ABSOLUTE_BASEDIR})
KAPITAN_BASEDIR_RELATIVE_PATH_FROM_SCRIPT_DIR=$(realpath ${KAPITAN_ABSOLUTE_BASEDIR} --relative-to=${SCRIPT_ABS_PATH})
KAPITAN_BASEDIR_RELATIVE_PATH_FROM_PWD=$(realpath ${KAPITAN_ABSOLUTE_BASEDIR} --relative-to=$(pwd))

BASH_INCLUDE_RELATIVE_PATH=${BASH_SOURCE[0]}
BASH_INCLUDE_RELATIVE_DIR=$(dirname ${BASH_INCLUDE_RELATIVE_PATH})

KAPITAN_TEMPLATES_DIRNAME=${KAPITAN_BASEDIR_RELATIVE_PATH_FROM_GIT_BASEDIR%%/*}

# Legacy
ROOT=${KAPITAN_ABSOLUTE_BASEDIR}
DIR=${SCRIPT_RELATIVE_DIR}

source ${SCRIPT_RELATIVE_DIR}/gcloud.include
source ${SCRIPT_RELATIVE_DIR}/kapitan.include

KUBECTL_CONTEXT="${TARGET}"
KUBECTL_SCRIPT="${SCRIPT_RELATIVE_DIR}/kubectl"
KUBECTL_COMMAND="${KUBECTL_BINARY} --context ${KUBECTL_CONTEXT}"

function in_docker() {
grep -sq 'docker\|lxc' /proc/1/cgroup
}

function check_installed() {
CMD=$1
if ! $(which ${CMD} > /dev/null); then
error "${CMD} not installed. Exiting..."
fi
}

# Only GNU xargs supports --no-run-if-empty
XARGS="xargs --no-run-if-empty"
if ! echo | $XARGS 2>/dev/null; then
# Looks like we have BSD xargs, use -x instead
XARGS="xargs"
fi


SCRIPT_TMP_DIR="$(mktemp -d /tmp/kapitan.XXXXXXXXXX)"
trap "rm -fr '${SCRIPT_TMP_DIR}'" INT TERM EXIT
Loading

0 comments on commit 3a5b21b

Please sign in to comment.