The following labs will provide examples to put into practice concepts and procedures learned in this chapter. We will use Minikube on docker (--driver=docker). You can use other Kubernetes implementations because we are going to use Kubernetes standar commands, available on any moder Kubernetes version (1.20+). We have decided to use Minikube on Docker because this version allows us to fix the IP address used for the Minikube Host (in this case executed as a container). The lab will give you a good idea of CI/CD practices with the modern GitOps model by creating a fully functional CI/CD environment using GitLab and ArgoCD. We will separate the labs in different sections to help you follow the required steps.
I will use my Linux desktop, but I will give you equivalent Powershell commands in case you use Microsoft Windows as your base operating system, although sometimes it is even easier to open a WSL2 console on your Windows host and execute commands like openssl or base64.
IMPORTANT NOTE: Please stop Docker Desktop before creating your Minikube environemnt. Both can run at the same time but they will consume lot of hardware resources and you should choose the right Kubernetes context for your environment.
Ensure that you have downloaded the content of this book’s GitHub repository in https://github.com/PacktPublishing/Docker-for-Developers-Handbook.git. For this chapter’s labs we will use the content of Chapter13 directory.
You can use one of the following Kubernetes Desktop environments:
- Docker Desktop (NetworkPolicy resources are not available at the time of writting this book in this platform)
- Rancher Desktop (supports NetworkPolicy resources)
- Minikube
- KinD
NOTE: Open a PowerShell console with administrator privileges to use Minikube.
We will first move inside our Chapter13 labs directory, located inside our local Github repository. You will see here the following content:
Chapter13 tree .
.
├── ArgoCD
│ ├── Applications
│ │ ├── minikube-simplestlab.deployed
│ │ └── minikube-simplestlab.yaml
│ ├── argo-cd-5.43.5.tgz
│ ├── create_values.sh
│ ├── minikube_install_values.template.yaml
│ └── minikube_install_values.yaml
├── GitLab
│ ├── create_values.sh
│ ├── gitlab-7.2.4.tgz
│ ├── minikube_install_values.template.yaml
│ └── minikube_install_values.yaml
├── Readme.md
└── Simplestlab
├── Code
│ ├── simplestapp
│ │ ├── dbconfig.json
│ │ ├── Dockerfile
│ │ ├── package.json
│ │ ├── README.md
│ │ ├── reset.html
│ │ ├── simplestapp.html
│ │ └── simplestapp.js
│ ├── simplestdb
│ │ ├── docker-entrypoint-initdb.d
│ │ │ └── init-demo.sh
│ │ ├── docker-entrypoint.sh
│ │ ├── Dockerfile
│ │ └── Dockerfile.scratch
│ └── simplestlb
│ ├── Dockerfile
│ ├── entrypoint.sh
│ └── nginx.conf
├── HelmCharts
│ ├── simplestlab
│ │ ├── Chart.lock
│ │ ├── charts
│ │ │ ├── simplestlab-app-1.0.0.tgz
│ │ │ ├── simplestlab-db-1.0.0.tgz
│ │ │ └── simplestlab-lb-1.0.0.tgz
│ │ ├── Chart.yaml
│ │ ├── templates
│ │ │ ├── _helpers.tpl
│ │ │ ├── ingress.yaml
│ │ │ └── NOTES.txt
│ │ └── values.yaml
│ ├── simplestlab-app
│ │ ├── Chart.yaml
│ │ ├── templates
│ │ │ ├── deployment.yaml
│ │ │ ├── _helpers.tpl
│ │ │ ├── hpa.yaml
│ │ │ ├── NOTES.txt
│ │ │ ├── secret.yaml
│ │ │ ├── serviceaccount.yaml
│ │ │ ├── service.yaml
│ │ │ └── tests
│ │ │ └── test-connection.yaml
│ │ └── values.yaml
│ ├── simplestlab-db
│ │ ├── Chart.yaml
│ │ ├── templates
│ │ │ ├── dbcredentials.secret.yaml
│ │ │ ├── _helpers.tpl
│ │ │ ├── initdb.secret.yaml
│ │ │ ├── NOTES.txt
│ │ │ ├── serviceaccount.yaml
│ │ │ ├── service.yaml
│ │ │ ├── statefulset.yaml
│ │ │ └── tests
│ │ │ └── test-connection.yaml
│ │ └── values.yaml
│ └── simplestlab-lb
│ ├── Chart.yaml
│ ├── templates
│ │ ├── configmap.yaml
│ │ ├── daemonset.yaml
│ │ ├── _helpers.tpl
│ │ ├── NOTES.txt
│ │ ├── serviceaccount.yaml
│ │ ├── service.yaml
│ │ └── tests
│ │ └── test-connection.yaml
│ └── values.yaml
└── Values
└── simplestlab
└── values.yaml
We have prepared for you 3 main directories:
- ArgoCD - Contains the installation of ArgoCD component and the Application resource we will use to deploy our SimplesLab application.
- GitLab - Contains the installation of GitLab component.
- SimplestLab - This directory contains all the code, Helm Charts and values used for deploying a SimplestLab application instance.
We will need the following tools on our environment:
- Minikube (and Docker if you follow my steps using a fixed IP for the lab).
- Git
- ArgoCD CLI - To create and modify the integration of ArgoCD inside our Kubernetes cluster.
- OpenSSL o Certutil on MS Windows - To decode some base64 strings in case you don't have base64.
- Helm - To deploy your Helm Charts.
- Kubectl - To connect to our Kubernetes cluster.
- Base64 - For decoding some strings.
To help you follow the paths used we will create a simplified prompt for the lab and prepare a bin directory for the binary tools.
- Microsfot Windows PowerShell:
PS C:\Users\frjaraur\Documents\...\Chapter13> function prompt {"Chapter13$ "}
Chapter13$ md $HOME\bin
Directory: C:\Users\frjaraur
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 8/18/2023 8:07 PM bin
Chapter13$ $env:PATH += ";$HOME\bin"
Chapter13$ $MINIKUBEIP = minikube ip
- Linux/MacOS:
export PS1="Chapter13$ "
mkdir $HOME/bin
export PATH=$PATH:$HOME/bin
Chapter13$ export MINIKUBEIP=$(minikube ip)
NOTE: We can recover the Minikube cluster IP by just issuing minikube ip. Ensure you have the MINIKUBEIP defined before executing the installation of GitLab and ArgoCD. We will use this value for the ingress.
- Linux:
If you are using Linux, you can use your package manager to install Git.
- Windows:
Chapter13$ winget install --id Git.Git -e --source winget
Found Git [Git.Git] Version 2.41.0.3
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Downloading https://github.com/git-for-windows/git/releases/download/v2.41.0.windows.3/Git-2.41.0.3-64-bit.exe
██████████████████████████████ 58.3 MB / 58.3 MB
Successfully verified installer hash
Starting package install...
Successfully installed
- Linux:
If you are using Linux, you can use the kubectl binary directly.
Chapter13$ curl -L "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" -o $HOME/bin/kubectl
Chapter13$ chmod 755 $HOME/bin/kubectl
- Windows:
Chapter13$ Invoke-WebRequest -URI https://dl.k8s.io/release/v1.28.0/bin/windows/amd64/kubectl.exe -OutFile $HOME\bin\kubectl.exe -UseBasicParsing
- Linux:
If you are using Linux, you can use the helm binary directly. Download your desired version from Download your desired version, unpackage it and move the helm binary to your $HOME/bin directory.
- Windows:
Chapter13$ Invoke-WebRequest -URI https://get.helm.sh/helm-v3.12.2-windows-amd64.zip -OutFile $env:TEMP\helm.zip -UseBasicParsing
Chapter13$ Expand-Archive $env:TEMP\helm.zip -DestinationPath $env:TEMP\helm_install
Chapter13$ mv $env:TEMP\helm_install\windows-amd64\helm.exe $HOME\bin\helm.exe
Chapter13$ helm version
version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.18.8"}
- Linux:
If you are using Linux, you can use your package manager to install Git.
- Linux
Chapter13$ curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
Chapter13$ sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
Chapter13$ rm argocd-linux-amd64
- Windows:
$version = (Invoke-RestMethod https://api.github.com/repos/argoproj/argo-cd/releases/latest).tag_name
Replace $version in the command below with the version of Argo CD you would like to download:
$url = "https://github.com/argoproj/argo-cd/releases/download/" + $version + "/argocd-windows-amd64.exe"
$output = "argocd.exe"
Invoke-WebRequest -Uri $url -OutFile $HOME/bin/$output
For a better experince, use at least 8GB of RAM and 4 vCPUs.
IMPORTANT NOTE: We will integrate the our GitLab Registry inside the Minikube platform as an insecure registry. This way we will not need to manage certificates in our Kubernetes cluster for pulling images for our deployed GitLab platform.
Chapter13$ minikube start --driver=docker --memory=8gb --cpus=4 --addons=ingress,metrics-server --cni=calico --insecure-registry="172.16.0.0/16" --static-ip=172.31.255.254
....
....
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
Configuring Calico (Container Networking Interface) ...
Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
▪ Using image registry.k8s.io/metrics-server/metrics-server:v0.6.4
....
....
Verifying ingress addon...
Enabled addons: storage-provisioner, default-storageclass, metrics-server, ingress
Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
We can easily verify our environment:
Chapter13$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane 3d1h v1.27.3
And now we can create the namespace for the SimplestLab application:
Chapter13$ kubectl create namespace simplestlab
namespace/simplestlab created
Because you have cloned the Labs from Packt GitHub repositoy, your application directories already include a .git folder. Copy the content of SimplestLab folder to a new worspace location on the Chapter13 folder.
- Linux:
Chapter13$ cp -R SimplestLab Simplestlab_WORKSPACE
- Windows:
Copy-Item SimplestLab -Recurse -Destination Simplestlab_WORKSPACE
NOTE: Remove all the hiden .git folders in Simplestlab_WORKSPACE folder and subfolders if any everytime you start with the lab.
How we will use these folder:
-
We will only use simplestapp code, hence Code\simplestapp will be synchronized (pushed) in GitLab.
-
All the HelmCharts subfolders will be needed because we prepared an umbrella chart (simplestlab) and the CI/CD proces will validate and recreate its dependencies as needed.
-
During the labs we will just modify the values file included inside Values folder. This way, we can manage the SimplestLab deployment parameters.
IMPORTANT NOTE: For now on we will use Simplestlab_WORKSPACE instead of the original Simplestlab folder, although you may read Simplestlab. You have copied the complete content without the .git files to be able to use all the GitLab respositories for managing all code changes. You can use Git commands to initialize the git local repository.
We have prepared for you a script that generates a valid values YAML file for the installation of GitLab. Ensure you have MINIKUBEIP variable set.
NOTE: If you prefer, you can manually create your own minikube_install_values.yaml file manually using the template minikube_install_values.template.yaml as example (change MINIKUBEIP with your value).
- Linux
Chapter13$ export MINIKUBEIP=$(minikube ip)
Chapter13$ echo $MINIKUBEIP
172.31.255.254
Chapter13$ cd GitLab
Chapter13/GitLab$ ls
create_values.sh gitlab-7.2.4.tgz minikube_install_values.template.yaml minikube_install_values.yaml
Chapter13/GitLab$ bash create_values.sh
We can now proceed to install GitLab:
Chapter13/GitLab$ helm upgrade --install gitlab gitlab-7.2.4.tgz --timeout 600s --namespace gitlab --create-namespace --values minikube_install_values.yaml
Release "gitlab" does not exist. Installing it now.
NAME: gitlab
LAST DEPLOYED: Fri Aug 18 21:03:17 2023
NAMESPACE: gitlab
STATUS: deployed
REVISION: 1
NOTES:
=== NOTICE
The minimum required version of PostgreSQL is now 13. See https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/doc/installation/upgrade.md for more details.
=== WARNING
Automatic TLS certificate generation with cert-manager is disabled.
One or more of the components does not have a TLS certificate Secret configured.
As a result, Self-signed certificates were generated for these components.
You may retrieve the CA root for these certificates from the `gitlab-wildcard-tls-ca` secret, via the following command. It can then be imported to a web browser or system store.
kubectl get secret gitlab-wildcard-tls-ca -ojsonpath='{.data.cfssl_ca}' | base64 --decode > gitlab.172.31.255.254.nip.io.ca.pem
If you do not wish to use self-signed certificates, please set the following properties:
- global.ingress.tls.secretName
OR all of:
- global.ingress.tls.enabled (set to `true`)
- gitlab.webservice.ingress.tls.secretName
- registry.ingress.tls.secretName
- minio.ingress.tls.secretName
- gitlab.kas.ingress.tls.secretName
=== WARNING
Automatic TLS certificate generation with cert-manager is disabled and no TLS certificates were provided. Self-signed certificates were generated that do not work with gitlab-runner. Please either disable gitlab-runner by setting `gitlab-runner.install=false` or provide valid certificates.
=== NOTICE
You've installed GitLab Runner without the ability to use 'docker in docker'.
The GitLab Runner chart (gitlab/gitlab-runner) is deployed without the `privileged` flag by default for security purposes. This can be changed by setting `gitlab-runner.runners.privileged` to `true`. Before doing so, please read the GitLab Runner chart's documentation on why we
chose not to enable this by default. See https://docs.gitlab.com/runner/install/kubernetes.html#running-docker-in-docker-containers-with-gitlab-runners
Help us improve the installation experience, let us know how we did with a 1 minute survey:https://gitlab.fra1.qualtrics.com/jfe/form/SV_6kVqZANThUQ1bZb?installation=helm&release=16-2
NOTE: We will use the GitLab generated CA during the ArgoCD deployment. You have to retrieve the CA root for these certificates from the
gitlab-wildcard-tls-casecret, via the following command. It can then be imported to a web browser or system store.kubectl get secret gitlab-wildcard-tls-ca -ojsonpath='{.data.cfssl_ca}' | base64 --decode > /tmp/gitlab.172.31.255.254.nip.io.ca.pem
We will now retrieve the initial root password:
- Linux:
Chapter13$ kubectl get secret -n gitlab gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' |base64 -d
pA0jOC2GiQUxKRnN1tv5jurhvwYas0XeVs6PWXuwZ0kOCYVCqI4grn17h4J6h4iK
- Windows:
Chapter13$ $GitLabRootb64Pass=kubectl get secret -n gitlab gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}'
Chapter13$ echo $GitLabRootb64Pass
eVk5ckdtb1NtZGdyMldjRk5IcVR5bHVjVktWQlpOWjM1aVUzR1pkaHVLNGNsanB1bHBmOGkxNDVHZGhKOUNuYw==
Chapter13$ echo $GitLabRootb64Pass >$env:TEMP\GitLabRootb64Pass.encoded.txt
Chapter13$ gc $env:TEMP\GitLabRootb64Pass.encoded.txt
eVk5ckdtb1NtZGdyMldjRk5IcVR5bHVjVktWQlpOWjM1aVUzR1pkaHVLNGNsanB1bHBmOGkxNDVHZGhKOUNuYw==
Chapter13$ certutil -f -decode $env:TEMP\GitLabRootb64Pass.encoded.txt $env:TEMP\GitLabRootb64Pass.decoded.txt
Input Length = 182
Output Length = 64
CertUtil: -decode command completed successfully.
Chapter13$ gc $env:TEMP\GitLabRootb64Pass.decoded.txt
pA0jOC2GiQUxKRnN1tv5jurhvwYas0XeVs6PWXuwZ0kOCYVCqI4grn17h4J6h4iK
And now we can verify the Ingress resources available:
Chapter13$ kubectl get ingress -n gitlab
NAME CLASS HOSTS ADDRESS PORTS AGE
gitlab-kas nginx kas.172.31.255.254.nip.io 172.31.255.254 80, 443 19m
gitlab-minio nginx minio.172.31.255.254.nip.io 172.31.255.254 80, 443 19m
gitlab-registry nginx registry.172.31.255.254.nip.io 172.31.255.254 80, 443 19m
gitlab-webservice-default nginx gitlab.172.31.255.254.nip.io 172.31.255.254 80, 443 19m
We can now access GitLab using our browser and openning https://gitlab.172.31.255.254.nip.io using root user and its associated password.
This workaround will help us to avoid certificate issues when Minikube's Kubernetes try to pull images from the GitLab registry:
Chapter13$ minikube ssh
docker@minikube:~$ sudo mkdir /etc/docker/certs.d/registry.172.31.255.254.nip.io
docker@minikube:~$ openssl s_client -connect registry.172.31.255.254.nip.io:443 -showcerts </dev/null 2>/dev/null | openssl x509 -outform PEM |sudo tee /etc/docker/certs.d/registry.172.31.255.254.nip.io/registry.172.31.255.254.nip.io.crt
docker@minikube:~$ ls /etc/docker/certs.d/registry.172.31.255.254.nip.io/
registry.172.31.255.254.nip.io.crt
docker@minikube:~$ exit
We have prepared for you a script that generates a valid values YAML file for the installation of ArgoCD. Ensure you have MINIKUBEIP variable set.
NOTE: If you prefer, you can manually create your own minikube_install_values.yaml file manually using the template minikube_install_values.template.yaml as example (change MINIKUBEIP with your value).
IMPORTANT NOTE: In ArgoCD we will include the GitLab CA. You can use the create_values.sh script or change the GITLABCA with your value, obtained using kubectl get secret -n gitlab gitlab-wildcard-tls-ca -o jsonpath='{.data.cfssl_ca}' (you can use the procedure shown for obtaining the GitLab root password).
Chapter13$ kubectl get secret -n gitlab gitlab-wildcard-tls-ca -o jsonpath='{.data.cfssl_ca}' |base64 -d -----BEGIN CERTIFICATE----- MIIFTDCCAzSgAwIBAgIUWyQorye5QhY+xkqAv34NlfAOeJIwDQYJKoZIhvcNAQEN BQAwPjEPMA0GA1UEChMGZ2l0bGFiMQ8wDQYDVQQLEwZnaXRsYWIxGjAYBgNVBAMT EUdpdExhYiBIZWxtIENoYXJ0MB4XDTIzMDgyMDE1MjQwMFoXDTI4MDgxODE1MjQw MFowPjEPMA0GA1UEChMGZ2l0bGFiMQ8wDQYDVQQLEwZnaXRsYWIxGjAYBgNVBAMT . . . . . . . . 7OvbJ4Ja6uDM5hkRvauGzNaMsf8VroLDnAt6a5B8WB56D4hDywv7zYmIgWwoXdlS 9D3gyLIDSBhgD92P5e2zGq/wZLchaHWNiWS/5iYgaWCKfbQNq2anhTGtnDFl5/VQ X7LTPHXP84yGqwGOZIejLw== -----END CERTIFICATE-----Identation in YAML files is key. If you have prepared the minikube_install_values.yaml manually, be very careful with the identation once you include your GitLab CA.
- Linux
Chapter13$ export MINIKUBEIP=$(minikube ip)
Chapter13$ echo $MINIKUBEIP
172.31.255.254
Chapter13$ cd ArgoCD
Chapter13/ArgoCD$ ls
create_values.sh argo-cd-5.43.5.tgz minikube_install_values.template.yaml minikube_install_values.yaml
Chapter13/ArgoCD$ bash create_values.sh
We can now proceed to install ArgoCD:
Chapter13/ArgoCD$ helm upgrade --install argocd argo-cd-5.43.5.tgz --namespace argocd --create-namespace --values minikube_install_values.yaml
Release "argocd" does not exist. Installing it now.
NAME: argocd
LAST DEPLOYED: Sun Aug 20 10:46:12 2023
NAMESPACE: argocd
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
In order to access the server UI you have the following options:
1. kubectl port-forward service/argocd-server -n argocd 8080:443
and then open the browser on http://localhost:8080 and accept the certificate
2. enable ingress in the values file `server.ingress.enabled` and either
- Add the annotation for ssl passthrough: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-1-ssl-passthrough
- Set the `configs.params."server.insecure"` in the values file and terminate SSL at your ingress: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-2-multiple-ingress-objects-and-hosts
After reaching the UI the first time you can login with username: admin and the random password generated during the installation. You can find the password by running:
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
(You should delete the initial secret afterwards as suggested by the Getting Started Guide: https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli)
We can quiclky take a look at the Ingress generated:
Chapter13$ kubectl get ingress -n argocd
NAME CLASS HOSTS ADDRESS PORTS AGE
argocd-server nginx argocd.172.31.255.254.nip.io 172.31.255.254 80 99s
Lets retrieve the admin password from the argocd-initial-admin-secret:
- Windows
Chapter13$ $ArdgoCDAdmin64Pass=kubectl get secret -n argocd argocd-initial-admin-secret -ojsonpath='{.data.password}'
Chapter13$ echo $ArdgoCDAdmin64Pass
Szc4NzcxSVF3bUpxcTlHdg==
Chapter13$ echo $ArdgoCDAdmin64Pass >$env:TEMP\ArdgoCDAdmin64Pass.encoded.txt
Chapter13$ gc $env:TEMP\ArdgoCDAdmin64Pass.encoded.txt
Szc4NzcxSVF3bUpxcTlHdg==
Chapter13$ certutil -f -decode $env:TEMP\ArdgoCDAdmin64Pass.encoded.txt $env:TEMP\ArdgoCDAdmin64Pass.decoded.txt
Input Length = 182
Output Length = 64
CertUtil: -decode command completed successfully.
Chapter13$ gc $env:TEMP\ArdgoCDAdmin64Pass.decoded.txt
QDKoHBovDxdO0rt1
- Linux
Chapter13$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
QDKoHBovDxdO0rt1
And now we can access the ArgoCD URL in https://argocd.172.31.255.254.nip.io/ using admin user and its associated password.
We will now move to the SimplestLab/Code directory. In this folder is located the simplestapp component code.
But first, we have to create the coder user. This is the user that will be used during the labs for pushing artifacts to the Gitlab reporitory and image registry.
Create coder user in GitLab:
1 - Using the root user, we go to the Admin settings and create the coder user.
username: coder
password: c0der000
email: coder@labs.local
2 - We now change the coder user password as we are not using real emails. We used c0der000 (8 characters minimum an cant' be usual words)
3 - Now we sign out from root user and login into GitLab with coder user.
NOTE: The first time you login into GitLab with coder user you will be asked to change your password, but you can reuse the old one added as root user.
4 - We now create SimplestLab group in GitLab. Code, Values, Images and HelmCharts subgroups will be included later. This will help us managea full project:
5 - Create SimplestLab group as Public (this will make things simple).
6 - Disable Auto DevOps in this group
7 - We will create Code, HelmCharts, Values and Images subgroups inside the SimplestLab group. We have simplified the full process for a quick demo environment and used only one project for the code, we will just include the App component of the application. All other images used are already built (Postgres Database and Nginx Loadbalancer). This subgroup will be Private.
We create Images project as Public to simplify the demo environment avoiding the need of using registry authentication to pull images from Kubernetes. This subgroup will be Public.
We create HelmCharts project as Public too to simplify the demo environment avoiding the need of using registry authentication to pull charts from ArgoCD.
Final distribution is show in the following screenshot.
- Code and Values - Private
- HelmCarts and Images - Public (just to make things easier during the demo, we will not need to authenticate for pulling container images or Helm Chart packages).
4 - Now we create the simplestapp project inside the Code subgroup and push our code.
We are now ready to upload our code.
Chapter13$ cd Simplestlab/Code/simplestapp
Chapter13$ tree
.
├── dbconfig.json
├── Dockerfile
├── package.json
├── README.md
├── reset.html
├── simplestapp.html
└── simplestapp.js
We follow the common steps for working with the previously created https://gitlab.172.31.255.254.nip.io/simplestlab/code/simplestapp.git repository:
Push an existing folder cd existing_folder git init --initial-branch=main git remote add origin https://gitlab.172.31.255.254.nip.io/simplestlab/code/teste.git git add . git commit -m "Initial commit" git push --set-upstream origin main
In our case:
Chapter13$ git init --initial-branch=main
Initialized empty Git repository in .../Chapter13/simplestlab/code/simplestapp/.git/
Chapter13$ git remote add origin https://gitlab.172.31.255.254.nip.io/simplestlab/code/simplestapp.git
Chapter13$ git add .
Chapter13$ git commit -m "Initial commit"
Author identity unknown
*** Please tell me who you are.
Run
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
to set your account's default identity.
Omit --global to set the identity only in this repository.
Chapter13$ git config user.name "coder"
Chapter13$ git config user.email "coder@labs.local"
Chapter13$ git commit -m "Initial commit"
[main (root-commit) 9425445] Initial commit
6 files changed, 413 insertions(+)
create mode 100644 Dockerfile
create mode 100644 dbconfig.json
create mode 100644 package.json
create mode 100644 reset.html
create mode 100644 simplestapp.html
create mode 100644 simplestapp.js
Chapter13$ git push --set-upstream origin main
fatal: unable to access 'https://gitlab.172.31.255.254.nip.io/simplestlab/code/simplestapp.git/': SSL certificate problem: unable to get local issuer certificate
When we try to push, we may get a certificate error. To avoid this, you can use git config http.sslverify false on each repository or use git config --global http.sslverify false to modify this setting globally during the labs.
Chapter13$ git config --global http.sslverify false
IMPORTANT NOTE: Don't forget to change back this setting after executing the labs.
Now, we can proceed normally:
Chapter13$ git push --set-upstream origin main
warning: missing OAuth configuration for gitlab.172.31.255.254.nip.io - see https://aka.ms/gcm/gitlab for more information
Enumerating objects: 8, done.
Counting objects: 100% (8/8), done.
Delta compression using up to 24 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 4.70 KiB | 4.70 MiB/s, done.
Total 8 (delta 0), reused 0 (delta 0), pack-reused 0
To https://gitlab.172.31.255.254.nip.io/simplestlab/code/simplestapp.git
* [new branch] main -> main
branch 'main' set up to track 'origin/main'.
And now we can verify our code in GitLab GUI directly:
We can test the full process changing something (create a README.md file for example using the GUI) and then pulling our code.
Chapter13$ git pull
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), 254 bytes | 84.00 KiB/s, done.
From https://gitlab.172.31.255.254.nip.io/simplestlab/code/simplestapp
9425445..4430abc main -> origin/main
Updating 9425445..4430abc
Fast-forward
README.md | 1 +
1 file changed, 1 insertion(+)
create mode 100644 README.md
We will just verify the build process prepared for you with a few tests for the SimplestApp component.
We are going to use Kaniko to simplify and improve security in our CI pipelines.
kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster. It doesn't depend on Docker Container Runtime and executes each command within a Dockerfile completely in userspace, which is great for security. This way we can build images inside any standard Kubernetes cluster.
You can review the .gitlab-ci.yml file included within our code:
stages:
- test
- security
- build
.common-build:
stage: build
image:
name: gcr.io/kaniko-project/executor:v1.9.1-debug
entrypoint: [""]
variables:
IMAGE: "$CI_REGISTRY_IMAGE:dev"
before_script:
- printf '%s\n' "${LABS_LOCAL_GITLAB_CERTIFICATE}" |sed 's/- /-\n/g; s/ -/\n-/g' | sed '/CERTIFICATE/! s/ /\n/g' >> /kaniko/ssl/certs/ca-certificates.crt
.release-build:
stage: build
image:
name: gcr.io/kaniko-project/executor:v1.9.1-debug
entrypoint: [""]
variables:
IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_TAG"
before_script:
- printf '%s\n' "${LABS_LOCAL_GITLAB_CERTIFICATE}" |sed 's/- /-\n/g; s/ -/\n-/g' | sed '/CERTIFICATE/! s/ /\n/g' >> /kaniko/ssl/certs/ca-certificates.crt
- export RELEASE_REGISTRY_IMAGE="$(echo $CI_REGISTRY_IMAGE|sed -e "s|/code/|/images/|g" )"
- export RELEASE_IMAGE="$RELEASE_REGISTRY_IMAGE:$CI_COMMIT_TAG"
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$PROJECTGROUP_USER\",\"password\":\"$PROJECTGROUP_PASSWORD\"}}}" > /kaniko/.docker/config.json
validate-dockerfile:
stage: test
image:
name: docker.io/hadolint/hadolint:v2.12.0-alpine
script:
- hadolint "${CI_PROJECT_DIR}/Dockerfile"
rules:
- if: $CI_COMMIT_BRANCH == "dev" || $CI_COMMIT_TAG =~ /^\d+\.\d+\.\d+$/
trivy-scan:
stage: security
image:
name: docker.io/aquasec/trivy:latest
entrypoint: [""]
script:
- trivy fs --no-progress --scanners vuln,secret,config ${CI_PROJECT_DIR}
rules:
- if: $CI_COMMIT_BRANCH == "dev" || $CI_COMMIT_TAG =~ /^\d+\.\d+\.\d+$/
build:
stage: build
extends: .common-build
script:
- /kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${IMAGE}"
rules:
- if: $CI_COMMIT_BRANCH == "dev"
build-release:
stage: build
extends: .release-build
script:
- /kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${RELEASE_IMAGE}"
rules:
- if: $CI_COMMIT_TAG =~ /^\d+\.\d+\.\d+$/
This file includes the following sections:
-
stages - In this key we declared the stages defined in this pipeline (test, security and build)
-
.common-build and .release-build - Declare some variables and scripts that will be included in real stages.
-
validate-dockerfile - Using a Hadolint docker image we will validate our Dockerfile.
-
trivy-scan - We will use a Trivy docker image to scan our code files for misconfigurations, secrets and vulnerable binaries.
-
build - This stage will create development images inside the code/Simplestapp repository.
-
build-release - This stage will only build release images, inside the Images/Simplestapp repositoy
IMPORTANT NOTE:
Notice that we have included following variables:
- PROJECTGROUP_USERNAME - coder
- PROJECTGROUP_PASSWORD - C0der000
- LABS_LOCAL_GITLAB_CERTIFICATE - Complete Gitlab TLS certificate chain base64 decoded value from
kubectl get secret -n gitlab -o yaml gitlab-wildcard-tls-chain -o jsonpath='{.data.gitlab\.172\.31\.255\.254\.nip\.io\.crt}'They should be created at group level, in the SimplestLab main group (Settings>CI/CD>Variables). This will ensure they are used in all the contained projects. Do not mark the 'Protected' checkbox as we will not use any of these variables on protected repositories. Just check Masked on the user password variable and leave others empty for all the variables.
Following screenshot shows the final situation, when all are created:
Defined variables will be used to authenticate using the coder user and push release images in the SimplestLab/Images/Simplestapp repository, under Deploy>Container Registry section.
In this repository different actions may be triggered:
- Whenever you change your code, commit and push using a dev branch, the DEV pipeline will be triggered:
validate-dockerfile ---> trivy-scan ---> build (DEVELOPMENT container image)
- Whenever you change your code, commit, tag using the common release X.Y.Z format and push using any branch, the RELEASE pipeline will be triggered:
validate-dockerfile ---> trivy-scan ---> build-release (RELEASE container image)
We have separated in two different registry repositories the different images created. This way we can provide different accesses (in this example only the coder user is able to push and pull inside the Code subgroup). This is just an example, you will never be allowed to publicly push images to a release repository, but we showed have you how to manage images in different repositories for different purposes.
Execute some small and traceable code changes and verify how the pipelines work.
Chapter13$ git switch -c dev
Switched to a new branch 'dev'
Chapter13$ git merge main
Already up to date.
Chapter13$ git push --set-upstream origin dev
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for dev, visit:
remote: https://gitlab.172.31.255.254.nip.io/simplestlab/code/simplestapp/-/merge_requests/new?merge_request%5Bsource_branch%5D=dev
remote:
To https://gitlab.172.31.255.254.nip.io/simplestlab/code/simplestapp.git
* [new branch] dev -> dev
branch 'dev' set up to track 'origin/dev'.
We can verify the Pipeline section:
And the Jobs:
Dev container images will be created in the Deploy>Container Registry in the same code repository. And release images will go to SimplestLab/Images/Simplestapp repository.
NOTE: Before pushing a production release, ensure you already have configured required variables at group level.
Now we can push a production ready code:
Chapter13$ git switch -c prod
Switched to a new branch 'prod'
Chapter13$ git push --set-upstream origin prod
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for prod, visit:
remote: https://gitlab.172.31.255.254.nip.io/simplestlab/code/simplestapp/-/merge_requests/new?merge_request%5Bsource_branch%5D=prod
remote:
To https://gitlab.172.31.255.254.nip.io/simplestlab/code/simplestapp.git
* [new branch] prod -> prod
branch 'prod' set up to track 'origin/prod'.
Chapter13$ git tag 1.0.0
Chapter13$ git push --tags
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://gitlab.172.31.255.254.nip.io/simplestlab/code/simplestapp.git
* [new tag] 1.0.0 -> 1.0.0
And the release images will be created inside the Images subgroup, in Images/SimplestApp:
We can now continue pushing the Helm Charts code. But first, we must create the different Helm Charts repositories under HelmCharts subgroup. We will create the following structure
Simplestlab
└─ HelmCharts
├── simplestlab
│ ├── Chart.lock
│ ├── charts
│ │ ├── simplestlab-app-1.0.0.tgz
│ │ ├── simplestlab-db-1.0.0.tgz
│ │ └── simplestlab-lb-1.0.0.tgz
│ ├── Chart.yaml
│ ├── templates
│ │ ├── _helpers.tpl
│ │ ├── ingress.yaml
│ │ └── NOTES.txt
│ └── values.yaml
├── simplestlab-app
│ ├── Chart.yaml
│ ├── templates
│ │ ├── deployment.yaml
│ │ ├── _helpers.tpl
│ │ ├── hpa.yaml
│ │ ├── NOTES.txt
│ │ ├── secret.yaml
│ │ ├── serviceaccount.yaml
│ │ ├── service.yaml
│ │ └── tests
│ │ └── test-connection.yaml
│ └── values.yaml
├── simplestlab-db
│ ├── Chart.yaml
│ ├── templates
│ │ ├── dbcredentials.secret.yaml
│ │ ├── _helpers.tpl
│ │ ├── initdb.secret.yaml
│ │ ├── NOTES.txt
│ │ ├── serviceaccount.yaml
│ │ ├── service.yaml
│ │ ├── statefulset.yaml
│ │ └── tests
│ │ └── test-connection.yaml
│ └── values.yaml
└── simplestlab-lb
├── Chart.yaml
├── templates
│ ├── configmap.yaml
│ ├── daemonset.yaml
│ ├── _helpers.tpl
│ ├── NOTES.txt
│ ├── serviceaccount.yaml
│ ├── service.yaml
│ └── tests
│ └── test-connection.yaml
└── values.yaml
Four project repositories must be created:
- simplestlab - This Chart defines the umbrella Helm Chart used to deploy all the components at once and its Ingress resource. We didn't add any Ingress on any other component.
- simplestlab-app - Describes the Application Backend component Deployment resource deployment.
- simplestlab-db - Describes the Database component StatefulSet deployment.
- simplestlab-lb - This describes the Load Balancer DaemonSet deployment.
This projects should be Public in this demo because we will not declare any credentials in ArgoCD. You will use credentials and Private repositories in your production and development platforms, but this will definetly require more configurations for this demo environment.
IMPORTANT NOTE: SimplestLab umbrella Chart depends on SimplestLab-app, SimplestLab-db and SimplestLab-lb Charts. Whatever change you made into any of these projects requires a Helm Chart dependencies update on the SimplestLab umbrella Chart. While you use the prepared CI/CD environment, you will need to run the SimplestLab umbrella Chart project pipeline again to rebuild these dependecies. If you want do manually update them, you will use helm dependencies update in the HelmCharts/SimplestLab directory. We prepared various scenarios in the Chart.yaml file in case you want to test it locally (LOCAL comments).
We will now push the Helm Charts code into their Gitlab associated repositories. Following code snippet shows an example for simplestlab umbrella:
Chapter13$ pwd
Chapter13/SimplestLab
Chapter13$ cd HelmCharts/simplestlab
Chapter13$ git init --initial-branch=main
Initialized empty Git repository in ... Chapter13/simplestlab/HelmCharts/simplestlab/.git/
Chapter13$ git remote add origin https://gitlab.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab.git
Chapter13$ git add .
Chapter13$ git commit -m "Initial commit"
[main (root-commit) ce1106d] Initial commit
10 files changed, 253 insertions(+)
create mode 100644 .helmignore
create mode 100644 Chart.lock
create mode 100644 Chart.yaml
create mode 100644 charts/simplestlab-app-1.0.0.tgz
create mode 100644 charts/simplestlab-db-1.0.0.tgz
create mode 100644 charts/simplestlab-lb-1.0.0.tgz
create mode 100644 templates/NOTES.txt
create mode 100644 templates/_helpers.tpl
create mode 100644 templates/ingress.yaml
create mode 100644 values.yaml
Chapter13$ git push --set-upstream origin main
Enumerating objects: 14, done.
Counting objects: 100% (14/14), done.
Delta compression using up to 24 threads
Compressing objects: 100% (13/13), done.
Writing objects: 100% (14/14), 11.99 KiB | 11.99 MiB/s, done.
Total 14 (delta 0), reused 0 (delta 0), pack-reused 0
To https://gitlab.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab.git
* [new branch] main -> main
branch 'main' set up to track 'origin/main'.
This will not trigger any pipeline. If you want to build a dev package or a release package you will need to push your changes to the dev branch (DEV) or tagged code release (PROD). Let's quickly review the associated .gitlab-ci.yml automation file:
stages:
- test
- dependencies
- build
.common:
before_script:
- CHART_VERSION=$(helm inspect chart . | grep -i '^version' | awk '{ sub(/^[^:]*:[ \t]*/, ""); print }')
- export CHART_VERSION
- echo "Helm Login into Gitlab registry."
- helm registry login -u $PROJECTGROUP_USER -p $PROJECTGROUP_PASSWORD $CI_REGISTRY --insecure
- printf '%s\n' "${LABS_LOCAL_GITLAB_CERTIFICATE}" |sed 's/- /-\n/g; s/ -/\n-/g' | sed '/CERTIFICATE/! s/ /\n/g' >> /apps/ca-certificates.crt
- export SSL_CERT_FILE=/apps/ca-certificates.crt
# If you prefer not to load the cetificate
#- export HELM_PUSH="helm push --insecure-skip-tls-verify"
- export HELM_PUSH="helm push"
image:
name: docker.io/alpine/helm:3.12.3
validate-helmfile:
stage: test
script:
- helm lint .
rules:
- if: $CI_COMMIT_BRANCH == "dev" || $CI_COMMIT_TAG =~ /^\d+\.\d+\.\d+$/
check-dependencies:
stage: dependencies
extends: .common
script:
- echo "Updating dependencies for validation."
- helm dependencies update
rules:
- if: $CI_COMMIT_BRANCH == "dev" || $CI_COMMIT_TAG =~ /^\d+\.\d+\.\d+$/
package-dev:
extends: .common
stage: build
script:
- helm dependencies update
- helm package . -d package --version ${CHART_VERSION}-dev
- echo "Pushing DEV to oci://${CI_REGISTRY_IMAGE}${CHARTREGISTRYIMAGE_SUFFIX}."
- ${HELM_PUSH} package/* oci://${CI_REGISTRY_IMAGE}${CHARTREGISTRYIMAGE_SUFFIX}
rules:
- if: $CI_COMMIT_BRANCH == "dev"
package-release:
extends: .common
stage: build
script:
- helm dependencies update
- helm package . -d package --version $CHART_VERSION
- echo "Pushing RELEASE to oci://${CI_REGISTRY_IMAGE}${CHARTREGISTRYIMAGE_SUFFIX}."
- ${HELM_PUSH} package/* oci://${CI_REGISTRY_IMAGE}${CHARTREGISTRYIMAGE_SUFFIX}
# Workaround for OCI certificate errors in ArgoCD. https://github.com/argoproj/argo-cd/issues/12371
- helm registry login docker.io -u $DOCKERHUB_USERNAME -p $DOCKERHUB_PASSWORD
- echo "Pushing RELEASE to oci://docker.io/${DOCKERHUB_USERNAME}."
- helm push package/* oci://docker.io/${DOCKERHUB_USERNAME}
rules:
- if: $CI_COMMIT_TAG =~ /^\d+\.\d+\.\d+$/
We defined 3 stages:
- test - A linter will be executed to validate our Helm Chart.
- dependencies - Dependencies will be checked.
- build - Package will be created (we need to update its dependencies too because we separate these steps for the demo).
In this pipeline, we will create -dev packages and X.Y.X release packages, depending on the branch you use for your Helm Chart code.
IMPORTANT NOTE:
Notice that I added two new variables here:
- DOCKERHUB_USERNAME - Your DockerHub username
- DOCKERHUB_PASSWORD - Your DockerHub password
These variables are defined in the HelmChart/SimplestLab umbrella chart only. This repository is Public and anyone will read your password, but you are using your own demo environment. You can secure this password making it Private, but you will need to prepare some username authentication (new user or even coder user here) and include it in ArgoCD OCI repository.
These credentials will be used to push release Chart packages to DockerHub (docker.io). This is used as workaround to the issue you will find trying to download OCI packages from a self-signed TLS protected repository in ArgoCD (argoproj/argo-cd#12371 issue is still open). We will push release images to docker.io and then we will configure this repository in ArgoCD. We are considering here all our OCI packages as public, hence anyone can pulled for demo purposes.
We will push all the Helm Charts code:
Chapter13$ cd HelmCharts/simplestlab-lb/
Chapter13$ git init --initial-branch=main
Chapter13$ git remote add origin https://gitlab.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab-lb.git
Chapter13$ git add .
Chapter13$ git commit -m "Initial commit"
Chapter13$ git push --set-upstream origin main
Chapter13$ cd ../simplestlab-app/
Chapter13$ git init --initial-branch=main
Chapter13$ git remote add origin https://gitlab.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab-app.git
Chapter13$ cd ../simplestlab-db/
Chapter13$ git init --initial-branch=main
Chapter13$ git remote add origin https://gitlab.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab-db.git
Chapter13$ git add .
Chapter13$ git commit -m "Initial commit"
Chapter13$ git push --set-upstream origin main
We can now create and push 1.0.0 tag to trigger the construction of a release package:
Chapter13$ cd ../simplestlab-db/
Chapter13$ git tag 1.0.0
Chapter13$ git push --tags
Ensure you build all the dependencies before triggering the construction of the umbrella package. This will fail if any of the required packages defined in its Chart.yaml file is not availabel:
apiVersion: v2
name: simplestlab
description: SimplestLab FULL STACK Helm chart for Kubernetes
type: application
version: 1.0.0
appVersion: "1.0.0"
dependencies:
- name: simplestlab-db
version: 1.0.0
#repository: oci://myregistry/projectgroup/helmcharts/simplestlab-db
#repository: "file://../simplestlab-db" # LOCAL for testing locally
repository: oci://registry.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab-db
- name: simplestlab-app
version: 1.0.0
#repository: oci://myregistry/projectgroup/helmcharts/simplestlab-app
#repository: "file://../simplestlab-app" # LOCAL for testing locally
repository: oci://registry.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab-app
- name: simplestlab-lb
version: 1.0.0
#repository: oci://myregistry/projectgroup/helmcharts/simplestlab-lb
#repository: "file://../simplestlab-lb" # LOCAL for testing locally
repository: oci://registry.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab-lb
You can use the local file key for local testing of the constructions. In such case, you will just need to ensure appropriate version numbers are defined on each Helm Chart.
Following simple and common git procedure will trigger the construction of a -dev package. All the Helm Chart packages will be included in the GitLab repository in the Deploy>Container Registry section:
Chapter13$ git switch -c dev
Switched to a new branch 'dev'
Chapter13$ git merge main
Already up to date.
Chapter13> git commit -a -m "Testing"
[dev f4f97e2] Testing
1 file changed, 53 insertions(+)
create mode 100644 .gitlab-ci.yml
Chapter13$ git merge main^C
Chapter13$ git push --set-upstream origin dev
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 24 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 875 bytes | 875.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for dev, visit:
remote: https://gitlab.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab/-/merge_requests/new?merge_request%5Bsource_branch%5D=dev
remote:
To https://gitlab.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab.git
* [new branch] dev -> dev
branch 'dev' set up to track 'origin/dev'.
If every thing worked as expected, you will have a -dev package inside Deploy>Container Registry section. And now we can build a release package creating a tag, using 1.0.0 dependencies (This is just a quick example).
Chapter13$ git push --tags
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 24 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 296 bytes | 296.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
To https://gitlab.172.18.158.255.nip.io/simplestlab/helmcharts/simplestlab.git
* [new tag] 1.0.0 -> 1.0.0
The package build will be triggered:
And the final package will pushed to DockerHub, as defined in our .gitlab-ci.yml file for the HelmCharts/SimplestLab repository only:
We will now prepare the values file that will manage the complete SimplestLab application.
We will create Simplestlab/values/simplestlab respository to manage a simple values file that will be used to deploy the SimplestLab application using the SimplestLab umbrella Helm Chart.
Here is the values file:
ingress:
enabled: true
className: "nginx"
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: simplestlab.172.31.255.254.nip.io
paths:
- path: /
pathType: ImplementationSpecific
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
service:
port: 8080
name: simplestlab-simplestlab-lb
simplestlab-db:
image:
repository: docker.io/frjaraur/simplestdb
pullPolicy: IfNotPresent
tag: "1.0"
envVariables:
# Never use clear values. This file is stored in GitLab. Use Sealed Secrets to encrypt and use here the encrypted values.
POSTGRES_PASSWORD: changeme
# Never use clear values, even on this script because it includes passwords. This file is stored in GitLab. Use Sealed Secrets to encrypt and use here the encrypted values, files can also be encrypted using this mechanism.
initdbScript: |
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
CREATE USER demo with PASSWORD 'd3m0' ;
CREATE DATABASE demo owner demo;
GRANT ALL PRIVILEGES ON DATABASE demo TO demo;
\connect demo;
CREATE TABLE IF NOT EXISTS hits
(
hitid serial,
serverip varchar(15) NOT NULL,
clientip varchar(15) NOT NULL,
date timestamp without time zone,
PRIMARY KEY (hitid)
);
ALTER TABLE hits OWNER TO demo;
EOSQL
simplestlab-app:
image:
repository: registry.172.31.255.254.nip.io/simplestlab/images/simplestapp
pullPolicy: Always
tag: "1.0.0"
replicaCount: 1
envVariables:
# Never use clear values. This file is stored in GitLab. Use Sealed Secrets to encrypt and use here the encrypted values.
dbhost: db
###################################
## First Test Update -- Change the value of this key
#dbhost: simplestlab-simplestlab-db
####################################
dbname: demo
dbpasswd: d3m0
dbuser: demo
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
simplestlab-lb:
image:
repository: docker.io/nginx
pullPolicy: IfNotPresent
tag: "alpine"
service:
port: 8080
##################################
# Second Test Update -- Uncomment whti section
# nginxConfig: |
# user nginx;
# worker_processes auto;
# error_log /tmp/nginx/error.log warn;
# pid /tmp/nginx/nginx.pid;
# events {
# worker_connections 1024;
# }
# http {
# server {
# listen 8080;
# location /healthz {
# add_header Content-Type text/plain;
# return 200 'OK';
# }
# location / {
# proxy_pass http://simplestlab-simplestlab-app:3000;
# }
# }
# }
###################################
We can observe in this file four different sections (starting from the end):
- simplestlab-lb - Defines the values to overwrite when deploying simplestlab-lb Helm Chart, added as dependency in the umbrella Chart.
- simplestlab-app - Defines the values to overwrite when deploying simplestlab-app Helm Chart, added as dependency in the umbrella Chart.
- simplestlab-db - Defines the values to overwrite when deploying simplestlab-db Helm Chart, added as dependency in the umbrella Chart.
- All the rest - The rest of the definitions included will manage the behavior of the umbrella Chart. We included parameters for deploying the Ingress resource here.
IMPORTANT NOTE:
We have defined two Labs in the values file.
First test will deploy a configuration with wrong database service name for the Application component (dbhost: db). The correct data is commented,dbhost: simplestlab-simplestlab-db. Thus, when you create the ArgoCD application for the first time, the application component, and the load balancer components will fail. Until you change the correct the mentioned value in the values YAML file, but this will not fix the problem in the load balancer component.
Second test will deploy a new configuration that will fix the load balancer component by deploying a completely new nginx.conf ConfigMap. To make this happen, uncomment the nginxConfig key in simplestlab-lb. Identation is key, uncomment all the lines (you can leave the "###################################" line).
When the Application resource is created in ArgoCD, the synchronization with the different reports starts and everytime you change either the Helm Chart package or the values file, the misconfigurations will be reflected in the ArgoCD environment.
At this very momment we will just push the current values.yaml file to the main branch in the SimplestLab/Values/SimplestLab repository:
Chapter13$ cd Values/simplestlab/
Chapter13$ git init --initial-branch=main
Chapter13$ git remote add origin https://gitlab.172.31.255.254.nip.io/simplestlab/values/simplestlab.git
Chapter13$ git add .
Chapter13$ git commit -m "Initial Commit"
[main 6531c8b] Initial Commit
1 file changed, 22 insertions(+), 22 deletions(-)
Chapter13$ git push
Username for 'https://gitlab.172.31.255.254.nip.io': coder
Password for 'https://coder@gitlab.172.31.255.254.nip.io':
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 24 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 435 bytes | 435.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
To https://gitlab.172.31.255.254.nip.io/simplestlab/values/simplestlab.git
69c8c1a..6531c8b main -> main
We can now integrate our application into ArgoCD.
We will use ArgoCD CLI to manage the integration of our Kubernetes cluster with ArgoCD. To connect Kubernetes with ArgoCD, this will create a ServiceAccount resource with cluster privileges to manage applications cluster wide.
We will use the password of the ArgoCD admin user and the current Kubernetes context (Minikube in this example):
Chapter13$ kubectl get secret -n argocd argocd-initial-admin-secret -o jsonpath='{.data.password}'|base64 -d
QDKoHBovDxdO0rt1
Chapter13$ argocd login --insecure --username admin --password QDKoHBovDxdO0rt1 --grpc-web argocd.$MINIKUBEIP.nip.io
'admin:login' logged in successfully
Context 'argocd.172.31.255.254.nip.io' updated
We have just connected the cluster, but we need to really integrate the cluster. I prefer adding a new cluster to ensure all integration requirements are automatically created by ArgoCD, hence we add minikube cluster using --in-cluster to make ArgoCD integrate the its own cluster (where it is running) using internal Kubernetes API service (kubernetes.defaul.svc):
Chapter13$ argocd cluster add minikube --name minikube --in-cluster
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `minikube` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0005] ServiceAccount "argocd-manager" created in namespace "kube-system"
INFO[0005] ClusterRole "argocd-manager-role" created
INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" created
INFO[0010] Created bearer token secret for ServiceAccount "argocd-manager"
Cluster 'https://kubernetes.default.svc' added
As you may have noticed, "argocd-manager" ServiceAccount is created with all its required privileges.
We can now login as admin user into ArgoCD and create the SimplestLab Project.
And now we can create the different repositories used for our project:
-
Code Repo type - https://gitlab.172.31.255.254.nip.io/simplestlab/values/simplestlab.git - This repository will be used to integrate our values YAML file, used to run the umbrella Helm Chart and deploy the full application. This repository requires authentication, we will use coder as username and c0der000 as password.
-
OCI type - registry.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab.git - Which includes the simplestlab-chart package.
-
OCI type - docker.io - Which includes the simplestlab-chart package uploaded at DockerHub as a WORKAROUND to the issue in ArgoCD with self-signed certificates.
NOTE: Code repositories doesn't require a name while OCI repositories do.
Now, we go back to the project settings and verify the right code and artifacts repositories are available. We will also configure here the clusters in which the project is allowed to be deployed. We will configure the following settings:
- SOURCE REPOSITORIES: *
- DESTINATIONS:
Server: https://kubernetes.default.svc
Following screenshot shows the final configuration:
In such situation, we are able to use any repository and kubernetes.default.svc cluster.
We will now prepare the Application resource for ArgoCD.
The ArgoCD GUI does not allow us to use multiple repositories, hence we will not be able to use a code repository for the values file and another one for the Helm Chart package artifact. Under this circunstances, we need to prepare the Application resource using a YAML file:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: minikube-simplestlab
namespace: argocd
spec:
destination:
name: minikube
namespace: simplestlab
project: simplestlab
sources:
- chart: simplestlab
helm:
releaseName: simplestlab
valueFiles:
- $values/values.yaml
repoURL: docker.io/frjaraur
#repoURL: docker.io/YOURUSERNAME
#repoURL: registry.172.31.255.254.nip.io/simplestlab/helmcharts/simplestlab
targetRevision: 1.0.0
- ref: values
repoURL: https://gitlab.172.31.255.254.nip.io/simplestlab/values/simplestlab.git
targetRevision: main
A copy of this YAML is included in the Chapter13/ArgoCD/Applications directory, minikube-simplestlab.yaml.
In this YAML file we declare:
- destination:
- minikube cluster
- simplestlab namespace
- sources:
- chart --> docker.io/frjaraur/simplestlab:1.0.0 package (notice the values of repoURL, chart and targetRevision keys)
- values --> values.yaml file included in the main branch of the https://gitlab.172.31.255.254.nip.io/simplestlab/values/simplestlab.git repository.
NOTE: I included other options for your repoURL you can test. As mentioned before, at the time of writting this lab there's an issue with self-signed certificates in ArgoCD. There isn't any parameter to include our GitLab certificate and pulling the Helm Charts will not work. You can use your DockerHub repository to test the complete CI/CD workflow.
We now create the Application resource (must be always included into argocd namespace):
Chapter13$ kubectl create -f ArgoCD/Applications/minikube-simplestlab.yaml
application.argoproj.io/minikube-simplestlab created
NOTE: Ensure that your simplestlab namespace exists.
As soon it is created, we will see the application in the ArgoCD GUI:
At this point, the application doesn't work:
Two components are in error state. We can easily locate them by using the filters available on the left panel:
The App component does not work because it can't connect to db database service. This makes also fail the Lb component, because it connects and serves the App component.
We will fix the application changing the values YAML file located in SimplestLab/Values/SimplestLab repository.
In this part of the Lab we will make some changes to our SimplestLab values file to fix the configuration errors. At this stage we are assume all the Helm Charts and Application Component code is working properly.
We will just modify locally our values.yaml file and change the database server (dbhost key):
1 - First verify the new name, automatically created by the Helm Chart template (These names could have been fixed but this is a usual error and we can see how to solve it in this example)
Chapter13$ kubectl get svc -n simplestlab
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
simplestlab-simplestlab-app ClusterIP 10.96.6.93 <none> 3000/TCP 2d14h
simplestlab-simplestlab-db ClusterIP 10.98.159.97 <none> 5432/TCP 2d14h
simplestlab-simplestlab-lb ClusterIP 10.103.79.186 <none> 8080/TCP 2d14h
2 - Now we know the new name for the Database server and we can change the dbhost value in the values.yaml file:
....
....
envVariables:
# Never use clear values. This file is stored in GitLab. Use Sealed Secrets to encrypt and use here the encrypted values.
dbhost: simplestlab-simplestlab-db
....
....
3 - Commit the new changes and push the file to our repository:
Chapter13$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: values.yaml
no changes added to commit (use "git add" and/or "git commit -a")
Chapter13$ git commit -a -m "Fixed database dbhost key"
[main 7995db3] Fixed database dbhost key
1 file changed, 2 insertions(+), 2 deletions(-)
Chapter13$ git push
Username for 'https://gitlab.172.31.255.254.nip.io': coder
Password for 'https://coder@gitlab.172.31.255.254.nip.io':
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 24 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 287 bytes | 287.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
To https://gitlab.172.31.255.254.nip.io/simplestlab/values/simplestlab.git
6531c8b..7995db3 main -> main
And now the changes are shown in our ArgoCD GUI. We haven't configured auto-sync, hence we will see a misconfiguration of the values (out-of-sync). Current values in the cluster are different from those expected by the configuration:
4 - We can now proceed to sync the creation of the secret with the right values. We gill push sync in the Secret resource.
5 - Once the Secret reource is replaced, we need to delete the old Pod, which uses the old Secret resource. Kubernetes ReplicaSet Controller will create anew Pod for us.
6 - App component works, but we need to fix the Lb component. We can review the new Pod's log and see it is working now:
But Lb component doesn't work.
In this case we will change the nginx.conf file required by the Nginx Lb. It is included as a ConfigMap resource and managed by the nginxConfig key in the values file. We need to change the name of the application backend service (App component). By default it uses app as you can see in the default values file included in the simplest-lb Helm Chart (SimplestLab/HelmCharts/simplestlab/values.yaml)
1 - We first verify the name of the App component service:
Chapter13$ kubectl get svc -n simplestlab
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
simplestlab-simplestlab-app ClusterIP 10.96.6.93 <none> 3000/TCP 2d14h
simplestlab-simplestlab-db ClusterIP 10.98.159.97 <none> 5432/TCP 2d14h
simplestlab-simplestlab-lb ClusterIP 10.103.79.186 <none> 8080/TCP 2d14h
Chapter13$ pwd
2 - Now we uncomment the nginxConfig key value prepared for you. After uncommenting the value, you should hae something like this:
....
....
simplestlab-lb:
image:
repository: docker.io/nginx
pullPolicy: IfNotPresent
tag: "alpine"
service:
port: 8080
##################################
# Second Test Update -- Uncomment whti section
nginxConfig: |
user nginx;
worker_processes auto;
error_log /tmp/nginx/error.log warn;
pid /tmp/nginx/nginx.pid;
events {
worker_connections 1024;
}
http {
server {
listen 8080;
location /healthz {
add_header Content-Type text/plain;
return 200 'OK';
}
location / {
proxy_pass http://simplestlab-simplestlab-app:3000;
}
}
}
###################################
3 - We commit and push the new changes.
Chapter13$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: values.yaml
no changes added to commit (use "git add" and/or "git commit -a")
Chapter13$ git add values.yaml
Chapter13$ git commit -m "Second fix"
[main fd40ee7] Second fix
1 file changed, 20 insertions(+), 20 deletions(-)
Chapter13$ git push
Username for 'https://gitlab.172.31.255.254.nip.io': coder
Password for 'https://coder@gitlab.172.31.255.254.nip.io':
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 24 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 434 bytes | 434.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
To https://gitlab.172.31.255.254.nip.io/simplestlab/values/simplestlab.git
7995db3..fd40ee7 main -> main
4 - New changes are not synced in ArgoCD:
5 - We sync the change and delete the Lb Pod, associated with the DaemonSet to fix the Nginx configuration issue. After the syncronization and the removal of the Pod, the new Pod works fine. The ArgoCD shows the application Healthy and Synced.
This is the last step on this long lab. You can make changes to either your configurations, code or Helm Charts and trigger the pipelines or GitOps integration to manage your application status and behavior.
We can't explain all the configurations in this lab as we have done to make all the workflow work, we gave you all the tips and you can deep dive by yourself exploring the already prepared configuration and script steps.
It would be nice to follow the lab by including now the NetworkPolicy resources created in Chapter 11 and the Nginx and Postgres Prometheus exporters prepared in Chapter 12.