Skip to content

Commit

Permalink
ci: added cron job to delete old tekton pipeline runs
Browse files Browse the repository at this point in the history
  • Loading branch information
nithinputhenveettil committed Apr 2, 2024
1 parent c591a51 commit 16ed942
Show file tree
Hide file tree
Showing 3 changed files with 76 additions and 9 deletions.
29 changes: 20 additions & 9 deletions .tekton/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers
# Install Tekton interceptors
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml

# Install Tekton dashboard - full installation is needed for read/write capabilities. eg: to make changes in the pipeline, such as re-running a pipeline run or deleting a pipeline run.
# Install Tekton dashboard - full installation is needed for read/write capabilities. eg: to make changes in the pipeline, such as re-running a pipeline run or deleting a pipeline run.
kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/release-full.yaml

# Make sure all pods are in the ready state before proceeding further by issuing the following command.
Expand All @@ -30,43 +30,54 @@ kubectl proxy

- If you have successfully completed the above mentioned steps, you should be able to access the Tekton Dashboard from [here](http://localhost:8001/api/v1/namespaces/tekton-pipelines/services/tekton-dashboard:http/proxy/)


## Install Tekton pipelines for Go Tracer

- You will find all the required YAML configurations in the `.tekton` folder, present in the root directory of the go-tracer repo. This includes all the required tasks, pipelines, and GitHub triggers, etc.

### Prerequisites before applying the YAML files

- You need three secrets to run the Go tracer pipelines successfully:
1. **GitHub bot token** - You need a GitHub bot token with write access to the repo. This is for sending commit statuses.
2. **GitHub Webhook Secret** - Create a very long random secret. You need to add this to the GitHub UI when creating a webhook for PR events.
3. **Cosmos URL and Secret** - This is for running azcosmos integration tests.

1. **GitHub bot token** - You need a GitHub bot token with write access to the repo. This is for sending commit statuses.
2. **GitHub Webhook Secret** - Create a very long random secret. You need to add this to the GitHub UI when creating a webhook for PR events.
3. **Cosmos URL and Secret** - This is for running azcosmos integration tests.

- Once you have access to the above secrets, replace them in the `secrets.yaml` file.
- You need an ingress controller for the GitHub Webhook to come through.
- Replace the `ingressClassName` and ingress domain or subdomain URL in the `github-webhook-ingress.yaml` file.
- Make sure you create two GitHub webhooks for both `pull_request` and `push` events in the settings tab of the repo. Please add the previously created webhook secret and `<<ingress_url/pr-hooks>>` and `<<ingress_url/push-hooks>>` as the Payload URL in the appropriate place when creating the webhook.

### Installation

- Once you are ready with the above steps, please use the below command to apply the YAML files.

```sh
sh deploy.sh
```

- Congrats! You have successfully configured Tekton CI pipeline for Go Tracer. You will see a status posted in Github for the Tekton runs, whenever a new PR is created.

## Trigger Tekton Pipeline

- Tekton pipeline can be triggered in two ways:
1. Raising a PR
- Tekton pipeline won't be immediately triggered when you raise a PR. You must apply the `tekton_ci` label to the PR to start the Tekton pipeline. Please note that if you raise a PR with a working copy, apply the label when it's ready for review. This label is for ensuring the pipelines won't trigger for every change to the PR. For any external PRs, one of the maintainers will add this label after a review.
2. Pushing something to the `main` branch
- Tekton pipeline will be triggered for every commit to the `main` branch.
1. Raising a PR
- Tekton pipeline won't be immediately triggered when you raise a PR. You must apply the `tekton_ci` label to the PR to start the Tekton pipeline. Please note that if you raise a PR with a working copy, apply the label when it's ready for review. This label is for ensuring the pipelines won't trigger for every change to the PR. For any external PRs, one of the maintainers will add this label after a review.
2. Pushing something to the `main` branch
- Tekton pipeline will be triggered for every commit to the `main` branch.

## How to debug/re-run a pipeline run

- You will find the Tekton dashboard URL for a specific pipeline run from the `details` section of the commit status.
- You can access the Tekton dashboard if you had set up the `ibmcloud` cli and authenticated the cluster in your local machine, by using the `kubectl proxy` command. For detailed information on accessing the IBM Cloud cluster via `ibmcloud` cli, you can refer to this [documentation](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster#access_public_se).
- Once you have access to the dashboard, you can see the logs for each run and will be able to re-run the `PipelineRun` .
- The status of the Tekton CI pipeline run for the PR will be updated once you initiate a re-run.

## Deleting pipeline run resources

- Deletion of old pipeline run resources will be automatically handled by a cron job by default. You can review the configuration in `cleanup-cron-job.yaml`. Feel free to edit the `NUM_TO_KEEP` variable to specify the number of old pipeline runs you wish to retain. The default value is `50`.

## Helpful resources

- [Ingress in IBM Cloud](https://cloud.ibm.com/docs/containers?topic=containers-managed-ingress-about)
- [Tekton: Getting Started](https://tekton.dev/docs/getting-started/)
- [Accessing clusters through the public cloud service endpoint on ibm cloud](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster#access_public_se)
Expand Down
55 changes: 55 additions & 0 deletions .tekton/cleanup-cron-job.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# (c) Copyright IBM Corp. 2024
# Ref : https://github.com/tektoncd/experimental/issues/479

---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cleaner
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cleaner
rules:
- apiGroups: ["tekton.dev"]
resources: ["pipelineruns"]
verbs: ["delete", "get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cleaner-to-cleaner
roleRef:
kind: Role
name: cleaner
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: cleaner
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: tekton-pipelinerun-cleaner
spec:
schedule: "0 2 * * *"
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
serviceAccount: cleaner
containers:
- name: kubectl
image: ghcr.io/ctron/kubectl:latest
env:
- name: NUM_TO_KEEP
value: "50"
command:
- /bin/bash
- -c
- |
TO_DELETE="$(kubectl get pipelinerun -o jsonpath='{range .items[?(@.status.completionTime)]}{.status.completionTime}{" "}{.metadata.name}{"\n"}{end}' | sort | head -n -${NUM_TO_KEEP} | awk '{ print $2}')"
test -n "$TO_DELETE" && kubectl delete pipelinerun ${TO_DELETE} || true
1 change: 1 addition & 0 deletions .tekton/deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ kubectl apply --filename pipeline.yaml
kubectl apply --filename tekton-triggers-eventlistener-serviceaccount.yaml
kubectl apply --filename github-eventlistener.yaml
kubectl apply --filename github-webhook-ingress.yaml
kubectl apply --filename cleanup-cron-job.yaml

0 comments on commit 16ed942

Please sign in to comment.