Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Documentation Improvements #23

Merged
merged 3 commits into from
Sep 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 9 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,18 @@ Welcome to Piper! Piper is open source project that aimed at providing multibran

## Table of Contents

- [Getting Started](#getting-started)
- [Reporting Issues](#reporting-issues)
- [How to Contribute](docs/CONTRIBUTING.md#how-to-contribute)
- [License](#license)
- [Piper](#piper)
- [Table of Contents](#table-of-contents)
- [Getting Started](#getting-started)
- [Reporting Issues](#reporting-issues)
- [How to Contribute](#how-to-contribute)
- [License](#license)

## Getting Started

Piper configures a webhook in git provider and listens to the webhooks sends. It will create a Workflow CRD out of branches that contains `.workflows` folder.
This folder should contain declarations of the templates and main DAG that will be running.
Finally, it will submit the Workflow as a K8s resource in the cluster.
Piper configures a webhook in git provider and listens to the webhooks sends. It will create a Workflow CRD out of branches that contains `.workflows` folder.
This folder should contain declarations of the templates and main DAG that will be running.
Finally, it will submit the Workflow as a K8s resource in the cluster.
To access more detailed explanations, please navigate to the [Documentation site](https://piper.quickube.com).

https://github.com/quickube/piper/assets/106976988/09b3a5d8-3428-4bdc-9146-3034d81164bf
Expand All @@ -29,7 +31,6 @@ If you encounter any issues or bugs while using Piper, please help us improve by
4. Add appropriate labels to categorize the issue (e.g., bug, enhancement, question).
5. Submit the issue, and our team will review and address it as soon as possible.


## How to Contribute

If you're interested in contributing to this project, please feel free to submit a pull request. We welcome all contributions and feedback.
Expand All @@ -38,6 +39,3 @@ Please check out our [Contribution guidelines for this project](docs/CONTRIBUTIN
## License

This project is licensed under the Apache License. Please see the [LICENSE](LICENSE) file for details.



37 changes: 19 additions & 18 deletions docs/configuration/environment_variables.md
Original file line number Diff line number Diff line change
@@ -1,49 +1,49 @@
## Environment Variables

The environment variables used by Piper to configure its functionality.
The helm chart populate them using [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml) file
Piper uses the following environment variables to configure its functionality.
The helm chart populates them using [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml) file

### Git

* GIT_PROVIDER
The git provider that Piper will use, possible variables: GitHub (will support bitbucket and gitlab)
The git provider that Piper will use, possible variables: GitHub . We plan to support Bitbucket and GitLab, as well.

* GIT_TOKEN
The git token that will be used.
The git token that will be used to connect to the git provider.

* GIT_ORG_NAME
The organization name.

* GIT_ORG_LEVEL_WEBHOOK
Boolean variable, whether to config webhook in organization level. default `false`
Boolean variable, whether to config webhook at the organization level. Defaults to `false`.

* GIT_WEBHOOK_REPO_LIST
List of repositories to configure webhooks to.

* GIT_WEBHOOK_URL
URL of piper ingress, to configure webhooks.
URL of Piper ingress, to configure webhooks.

* GIT_WEBHOOK_AUTO_CLEANUP
Will cleanup all webhook that were created with piper.
Notice that there will be a race conditions between pod that being terminated and the new one.
Boolean variable that, if true, will cause Piper to automatically cleanup all webhooks that it creates when they are no longer necessary.
Notice that there is a race condition between a pod that is being terminated and the new one being scheduled.

* GIT_ENFORCE_ORG_BELONGING
Boolean variable, whether to enforce organizational belonging of git event creator. default `false`
Boolean variable that, if true, will cause Piper to enforce organizational belonging of git event creator. Defaults to `false`.

* GIT_FULL_HEALTH_CHECK
Enables full health check of webhook. Full health check contains expecting and validating ping event from a webhook.
Doesn't work for bitbucket, because the API call doesn't exists.

Boolean variable that, if true, enables full health checks on webhooks. A full health check contains expecting and validating ping event from a webhook.
Doesn't work for Bitbucket, because the API call doesn't exist on that platform.

### Argo Workflows Server

* ARGO_WORKFLOWS_TOKEN
The token of Argo Workflows server.
This token is used to authenticate with the Argo Workflows server.

* ARGO_WORKFLOWS_ADDRESS
The address of Argo Workflows Server.

* ARGO_WORKFLOWS_CREATE_CRD
Whether to directly send Workflows instruction or create a CRD in the Cluster.
Boolean variable that deterines whether to directly send Workflows instructions or create a CRD in the Cluster.

* ARGO_WORKFLOWS_NAMESPACE
The namespace of Workflows creation for Argo Workflows.
Expand All @@ -52,9 +52,10 @@ The helm chart populate them using [values.yaml](https://github.com/quickube/pip
Used to configure Argo Workflows client with local kube configurations.

### Rookout

* ROOKOUT_TOKEN
The token used to configure Rookout agent. If not provided, will not start the agent.
* ROOKOUT_LABELS
The labels to label instances at Rookout, default to "service:piper"
* ROOKOUT_LABELS
The labels to label instances at Rookout, defaults to "service:piper"
* ROOKOUT_REMOTE_ORIGIN
The repo URL for source code fetching, default:"https://github.com/quickube/piper.git".
The repo URL for source code fetching, default:"https://github.com/quickube/piper.git".
10 changes: 5 additions & 5 deletions docs/configuration/health_check.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
## Health Check

Health check executed every 1 minute as configured in the helm chart under `livenessProbe`, and triggered by `/healthz` endpoint:
The following examples shows a health check being executed every 1 minute as configured in the helm chart under `livenessProbe`, and triggered by `/healthz` endpoint:

```yaml
livenessProbe:
httpGet:
Expand All @@ -18,14 +19,13 @@ The mechanism for checking the health of Piper is:

1. Piper set health status of all webhooks to not-healthy.

2. Piper requests ping from all the webhooks configured.
2. Piper requests ping from all the webhooks configured.

3. Git Provider send ping to `/webhook` endpoint, this will set the health status to `healthy` with timeout of 5 seconds.

4. Piper check the status of all webhooks configured.

Therefore, the criteria for health checking are:
1. The registered webhook exists.
2. The webhook send a ping in 5 seconds.


1. The registered webhook exists.
2. The webhook send a ping in 5 seconds.
47 changes: 25 additions & 22 deletions docs/getting_started/installation.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
## Instalation
## Installation

Piper should be deployed in the cluster with Argo Workflows.
Piper will create a CRD that Argo Workflows will pick, so install or configure Piper to create those CRDs in the right namespace.
You must deploy Piper to a cluster with a pre-existing Argo Workflows deployment.
Piper will create a CRD that Argo Workflows will pick, so install or configure Piper to create those CRDs in the right namespace.

Please check out [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml) file of the helm chart configurations.

To add piper helm repo run:

```bash
helm repo add piper https://piper.quickube.com
```

After configuring Piper [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml), run the following command for installation:

```bash
helm upgrade --install piper piper/piper \
-f YOUR_VALUES_FILE.yaml
Expand All @@ -22,33 +24,34 @@ helm upgrade --install piper piper/piper \

### Ingress

Piper should listen to webhooks from your git provider.
Expose it using ingress or service, then provide the address to `piper.webhook.url` as followed:
`https://PIPER_EXPOESED_URL/webhook`
Piper works best when it is able to listen to webhooks from your git provider.
Expose Piper using an ingress or service, then provide the address to `piper.webhook.url` as follows:
`https://PIPER_EXPOSED_URL/webhook`

Checkout [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml)
Refer to [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml) for more information.

### Git

Piper will use git for fetching `.workflows` folder and receiving events using webhooks.
Piper will use git to fetch the `.workflows` folder and receive events using webhooks.

To pick which git provider you are using provide `gitProvider.name` configuration in helm chart (Now only supports GitHub and Bitbucket).
To pick which git provider you are using provide `gitProvider.name` configuration in helm chart (Currently we only support GitHub and Bitbucket).

Also configure you organization (Github) or workspace (Bitbucket) name using `gitProvider.organization.name` in helm chart.
You must also configure your organization (GitHub) or workspace (Bitbucket) name using `gitProvider.organization.name` in the helm chart.

#### Git Token Permissions

The token should have access for creating webhooks and read repositories content.
For GitHub configure `admin:org` and `write:org` permissions in Classic Token.
For Bitbucket configure `Repositories:read`, `Webhooks:read and write` and `Pull requests:read` permissions (for multiple repos use workspace token).
For GitHub, configure `admin:org` and `write:org` permissions in Classic Token.
For Bitbucket, configure `Repositories:read`, `Webhooks:read and write` and `Pull requests:read` permissions (for multiple repos use workspace token).

#### Token

The git token should be passed as secret in the helm chart at `gitProvider.token`.
Can be passed as parameter in helm install command using `--set piper.gitProvider.token=YOUR_GIT_TOKEN`
The git token should be passed as secret in the helm chart at `gitProvider.token`.
The token can be passed as parameter via helm install command using `--set piper.gitProvider.token=YOUR_GIT_TOKEN`

Alternatively, you can consume an already existing secret by configuring `piper.gipProvider.existingSecret`.
The key should have the name `token`. You can be create a Secret using this command:

Alternatively, you can consume already existing secret and fill up `piper.gipProvider.existingSecret`.
The key should be name `token`. Can be created using
```bash
kubectl create secret generic piper-git-token --from-literal=token=YOUR_GIT_OKEN
```
Expand All @@ -57,17 +60,17 @@ kubectl create secret generic piper-git-token --from-literal=token=YOUR_GIT_OKEN

Piper will create a webhook configuration for you, for the whole organization or for each repo you configure.

Configure `piper.webhook.url` the address of piper that exposed with ingress with `/webhook` postfix.
Configure `piper.webhook.url` with the address of Piper that you exposed using an Ingress or Service with `/webhook` postfix.

For organization level configure: `gitProvider.webhook.orgLevel` to `true`.
For organization level configuration: `gitProvider.webhook.orgLevel` to `true`.

For granular repo webhook provide list of repos at: `gitProvider.webhook.repoList`.
For granular repo webhook provide list of repos at: `gitProvider.webhook.repoList`.

Piper implements graceful shutdown, it will delete all the webhooks when terminated.
Piper implements graceful shutdown, it will delete all the webhooks when terminated.

#### Status check

Piper will handle status checks for you.
Piper will handle status checks for you.
It will notify the GitProvider for the status of Workflow for specific commit that triggered Piper.
For linking provide valid URL of your Argo Workflows server address at: `argoWorkflows.server.address`

Expand All @@ -81,4 +84,4 @@ To lint the workflow before submitting it, please configure the internal address

#### Skip CRD Creation (On development)

Piper can communicate directly to Argo Workflow using ARGO_WORKFLOWS_CREATE_CRD environment variable, if you want to skip the creation of CRD change `argoWorkflows.crdCreation` to `false`.
Piper can communicate directly to Argo Workflow using ARGO_WORKFLOWS_CREATE_CRD environment variable, if you want to skip the creation of CRD change `argoWorkflows.crdCreation` to `false`.
14 changes: 6 additions & 8 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,17 @@
<p align="center">
<img src="https://www.quickube.com/wp-content/uploads/2022/10/ArgoPipeline_1.0_Hero.png.webp?raw=true" />
</p>

Welcome to Piper!

Piper is an open source project that aimed at providing multibranch pipeline functionality to Argo Workflows, allows users to create distinct Workflows based on Git branches. Supports GitHub and Bitbucket.
Welcome to Piper!

## General explanation
Piper is an open source project that aimed at providing multibranch pipeline functionality to Argo Workflows. This allows users to create distinct Workflows based on Git branches. We supports GitHub and Bitbucket.

## General Explanation

<p align="center">
<img src="https://raw.githubusercontent.com/quickube/piper/main/docs/img/flow.svg" />
</p>

To achieve multibranch pipeline functionality Piper will do the hard works for us.
At initialization, it will load all configuration and create a webhook in repository or organization scope.
Then each branch that have `.workflows` folder will create a Workflow CRD out of the files in this folder.
Piper handles the hard work of configuring multibranch pipelines for us! At initialization, it will load all configuration and create a webhook in repository or organization scope. Then, for each branch that has a `.workflows` folder, Piper will create a Workflow CRD out of the files in this folder. Finally, when Piper detects changes in the repository via the webhook, it triggers the workflows that match the branch and event.

![type:video](./img/piper-demo-1080.mp4)
![type:video](./img/piper-demo-1080.mp4)
2 changes: 1 addition & 1 deletion docs/usage/global_variables.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## Global variables
## Global Variables

Piper will automatically add Workflow scope parameters that can be referenced from any template.
The parameters taken from webhook metadata, and will be populated respectively to GitProvider and event that triggered the workflow.
Expand Down
11 changes: 7 additions & 4 deletions docs/usage/workflows_config.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,18 @@ Piper can inject configuration for Workflows that Piper creates.
`default` config used as a convention for all Workflows that piper will create, even if not explicitly mentioned in triggers.yaml file.

### ConfigMap

Piper will mount a configMap when helm used.
`piper.workflowsConfig` variable in helm chart, will create a configMap that hold set of configuration for Piper.
Here is an [examples](https://github.com/quickube/piper/tree/main/examples/config.yaml) of such configuration.

### Spec
This will be injected to Workflow spec field. can hold all configuration of the Workflow.

This will be injected to Workflow spec field. can hold all configuration of the Workflow.
> :warning: Please notice that the fields `entrypoint` and `onExit` should not exist in spec. both of them are managed fields.

### onExit
This is the exit handler for each of the Workflows create by piper.
It configures a DAG that will be executed when the workflow ends.
You can provide the templates to it us in the following [Examples](https://github.com/quickube/piper/tree/main/examples/config.yaml).

This is the exit handler for each of the Workflows create by piper.
It configures a DAG that will be executed when the workflow ends.
You can provide the templates to it us in the following [Examples](https://github.com/quickube/piper/tree/main/examples/config.yaml).
32 changes: 20 additions & 12 deletions docs/usage/workflows_folder.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ We will explain each of the files that should be included in the `.workflows` fo

### triggers.yaml (convention name)

This file holds a list of triggers that will be executed `onStart` by `events` from specific `branches`.
This file holds a list of triggers that will be executed `onStart` by `events` from specific `branches`.
Piper will execute each of matching triggers, so configure it wisely.

```yaml
- events:
- push
Expand All @@ -17,50 +18,57 @@ Piper will execute each of matching triggers, so configure it wisely.
templates: ["templates.yaml"]
config: "default"
```
Can be found [here](https://github.com/quickube/piper/tree/main/examples/.workflows/triggers.yaml).

This example can be found [here](https://github.com/quickube/piper/tree/main/examples/.workflows/triggers.yaml).

In this example `main.yaml` will be executed as DAG when `push` or `pull_request.synchronize` events will be applied in `main` branch.
`onExit` will be executed `exit.yaml` when finished the workflow as exit handler.


`onExit` can overwrite the default `onExit` configuration from by reference existing DAG tasks as in the [example](https://github.com/quickube/piper/tree/main/examples/.workflows/exit.yaml).

`config` field used for workflow configuration selection. the default value is `default` configuration.

#### events
Events field used to terminate when the trigger will be executed. name of the event depends on the git provider.

Events field used to terminate when the trigger will be executed. name of the event depends on the git provider.

For instance, GitHub pull_request event have few action, one of them is synchronize.

#### branches

For which branch that trigger will be executed.

#### onStart
#### onStart

This [file](https://github.com/quickube/piper/tree/main/examples/.workflows/main.yaml) can be named as you wish and will be referenced in `triggers.yaml` file. It will define an entrypoint DAG that the Workflow will execute.

As a best practice, this file should contain the dependencies logic and parametrization of each of referenced templates. It should not implement new templates, for this, use template.yaml file.

#### onExit
#### onExit

This field used to pass verbose exitHandler to the triggered workflow.
It will override the default onExit from the provided `config` or the default `config`.

In the provided `exit.yaml` describes a DAG that will overwrite the default `onExit` configuration.
[Example](https://github.com/quickube/piper/tree/main/examples/.workflows/exit.yaml)

#### templates
This field will have additional templates that will be injected to the workflows.
#### templates

This field will have additional templates that will be injected to the workflows.
The purpose of this field is to create repository scope templates that can be referenced from the DAGs templates at `onStart` or `onExit`.
[Example](https://github.com/quickube/piper/tree/main/examples/.workflows/templates.yaml)

As a best practice, use this field for template implementation and reference them from executed.
[Example](https://github.com/quickube/piper/tree/main/examples/.workflows/main.yaml).

### config
configured by `piper-workflows-config` [configMap](workflows_config.md).

configured by `piper-workflows-config` [configMap](workflows_config.md).
Can be passed explicitly, or will use `deafault` configuration.

### parameters.yaml (convention name)
Will hold a list of global parameters of the Workflow.
can be referenced from any template with `{{ workflow.parameters.___ }}.`

[Example](https://github.com/quickube/piper/tree/main/examples/.workflows/parameters.yaml)
Will hold a list of global parameters of the Workflow.
Can be referenced from any template with `{{ workflow.parameters.___ }}.`

[Example](https://github.com/quickube/piper/tree/main/examples/.workflows/parameters.yaml)
4 changes: 2 additions & 2 deletions examples/.workflows/templates.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
image: alpine
command: [ sh ]
source: |
echo "wellcome to {{ workflow.parameters.global }}
echo "{{ inputs.parameters.message }}"
echo "welcome to {{ workflow.parameters.global }}
echo "{{ inputs.parameters.message }}"
Loading
Loading