|
| 1 | +# Zuul users guide |
| 2 | + |
| 3 | +## Prerequisites |
| 4 | + |
| 5 | +1. Repository is known by [SCS Zuul](https://zuul.scs.community) |
| 6 | +2. Basic ansible knowledge |
| 7 | +3. Basic yaml knowledge |
| 8 | +4. zuul-client installed (Only if you want to create secrets. [See also](#what-about-secrets)) |
| 9 | + |
| 10 | +Check [SCS Zuul projects](https://zuul.scs.community/t/SCS/projects) for your repository to |
| 11 | +be available. If it is missing you need an administrator to get your repository |
| 12 | +configured to Zuul. |
| 13 | + |
| 14 | +## Who is it for? |
| 15 | + |
| 16 | +You may have heard about Zuul and may ask yourself if it is capable to support you. |
| 17 | +Basically everything you use ansible for can be done using Zuul. That is not always |
| 18 | +a good thing since you may get careless and your workload will exceed the CI/CD concept. |
| 19 | + |
| 20 | +If you find yourself doing things under the following list you are at the right place. |
| 21 | + |
| 22 | +1. Code testing |
| 23 | +2. Deployment tests using IaC |
| 24 | + |
| 25 | +If you want to, let's say, monitor something using Zuul, that is possible but not the |
| 26 | +intended use case. |
| 27 | + |
| 28 | +## Where do I start? |
| 29 | + |
| 30 | +Right in your project's repository! The only prerequisite is that |
| 31 | +your repository you want Zuul to work on is known by Zuul. This is done by the Zuul's |
| 32 | +tenant configuration. To update this configuration you need access to the Zuul instance |
| 33 | +or ask an administrator for help. |
| 34 | + |
| 35 | +We assume that Zuul knows about your repository so we can get started. There are three |
| 36 | +topics that you should know about. To get jobs running you need the "job" itself. Jobs run |
| 37 | +within a "pipeline". The third important thing is to provide a "project" definition. |
| 38 | + |
| 39 | +## Where to save the Zuul relevant data? |
| 40 | + |
| 41 | +Zuul will parse all branches of the untrusted repositories that Zuul knows about. |
| 42 | +Your repository is most likely an untrusted one since only the configuration repositories should |
| 43 | +have the "trusted" state. |
| 44 | +So it doesn't matter whether you have just one branch containing Zuul files or all branches. Zuul |
| 45 | +is looking for the following pathes on your repositories root. |
| 46 | + |
| 47 | +```bash |
| 48 | +./zuul.yaml # everything is in here |
| 49 | + |
| 50 | +./.zuul.yaml # ... or here |
| 51 | + |
| 52 | +./zuul.d/ # use directory style to get a bit of a structure |
| 53 | +├── jobs.yaml |
| 54 | +└── project.yaml |
| 55 | + |
| 56 | +./.zuul.d/ # the same as before just hidden |
| 57 | +├── jobs.yaml |
| 58 | +└── project.yaml |
| 59 | +``` |
| 60 | + |
| 61 | +Just use exactly one of the four possibilities. |
| 62 | + |
| 63 | +If using the directory style configuration all `yaml` files within this directory will be |
| 64 | +processed. If your projects configuration is small enough you may put all information in |
| 65 | +a single file called `zuul.yaml`. It is also possible to create the file or the directory |
| 66 | +with a leading dot to hide them for non zuul related work within the repository. |
| 67 | + |
| 68 | +### Projects |
| 69 | + |
| 70 | +If Zuul is configured to observe your repository it will have a look at your projects |
| 71 | +definition. Minimal example: |
| 72 | + |
| 73 | +```yaml |
| 74 | +- project: |
| 75 | + name: my-org/my-repo |
| 76 | + default-branch: main |
| 77 | + merge-mode: "squash-merge" |
| 78 | + my_pipeline1: |
| 79 | + jobs: |
| 80 | + - my_job1 |
| 81 | + - my_job2 |
| 82 | + ...... |
| 83 | + my_pipeline2: |
| 84 | + jobs: |
| 85 | + - my_jobs |
| 86 | + ... |
| 87 | + |
| 88 | +``` |
| 89 | + |
| 90 | +By default Zuul will observe all branches for such files. We have to set the repository name |
| 91 | +that have to match the exact value that was set for Zuul. Set a default-branch where actions |
| 92 | +that don't match an explicit branch are executed on. Set the merge-mode that Zuul has to use. |
| 93 | +But beware that not all issue tracker support all methods. For github squash-merge will work. |
| 94 | + |
| 95 | +After these three properties add the pipelines you want to use to the project definition. |
| 96 | +With the `jobs` list you define which jobs to run in which pipeline. |
| 97 | + |
| 98 | +[See official documentation](https://zuul-ci.org/docs/zuul/latest/config/project.html) |
| 99 | + |
| 100 | +### Pipelines |
| 101 | + |
| 102 | +Every Zuul instance will have at least one repository that is used for configuration. There |
| 103 | +you will find the available pipelines. Pipelines are used to run your jobs on a periodic or |
| 104 | +event driven base. Pipelines can be used to run other pipelines and to keep your jobs in a |
| 105 | +defined order if you need this. |
| 106 | + |
| 107 | +Have a look at the configuration repository to utilize the pipelines for your repository. |
| 108 | +See available [pipelines](https://github.com/SovereignCloudStack/zuul-config/blob/main/zuul.d/gh_pipelines.yaml) for SCS. |
| 109 | +You are not able to define new pipelines outside of a so called "configuration" repository. Since, |
| 110 | +by default your repo is considered "untrusted". So in the first place you don't need to |
| 111 | +think about, how to create a pipeline. Just use one that fits your needs as close as possible. Next you will |
| 112 | +find an enumeration and a small description about the available pipelines in SCS Zuul. |
| 113 | + |
| 114 | +Pipelines available in SCS Zuul: |
| 115 | + |
| 116 | +#### 1. check |
| 117 | + |
| 118 | +* event driven pipeline |
| 119 | +* runs if a pull request is created, changed or reopened |
| 120 | +* re-runs if a comment contains `recheck` |
| 121 | + |
| 122 | +#### 2. gate |
| 123 | + |
| 124 | +* event driven pipeline |
| 125 | +* trigger events: pull_request_review, pull_request, check_run |
| 126 | + |
| 127 | +#### 3. post |
| 128 | + |
| 129 | +* event driven pipeline |
| 130 | +* trigger event: post |
| 131 | + |
| 132 | +#### 4. tag |
| 133 | + |
| 134 | +* event driven pipeline |
| 135 | +* trigger event: push |
| 136 | + |
| 137 | +#### 5. e2e-test |
| 138 | + |
| 139 | +* event driven pipeline |
| 140 | +* trigger event: pull_request |
| 141 | + |
| 142 | +#### 6. e2e-quick-test |
| 143 | + |
| 144 | +* event driven pipeline |
| 145 | +* trigger event: pull_request |
| 146 | + |
| 147 | +#### 7. unlabel-on-update-e2e-test |
| 148 | + |
| 149 | +* event driven pipeline |
| 150 | +* trigger event: pull_request |
| 151 | + |
| 152 | +#### 8. unlabel-on-update-e2e-quick-test |
| 153 | + |
| 154 | +* event driven pipeline |
| 155 | +* trigger event: pull_request |
| 156 | + |
| 157 | +#### 9. periodic-hourly |
| 158 | + |
| 159 | +* time based pipeline that runs every hour |
| 160 | + |
| 161 | +#### 10. periodic-daily |
| 162 | + |
| 163 | +* time based pipeline that runs every day at 3 o'clock am. |
| 164 | + |
| 165 | +#### 11. compliance_check |
| 166 | + |
| 167 | +* time based pipeline that runs every 15 minutes |
| 168 | + |
| 169 | +If you want to know more about pipelines: [See official documentation](https://zuul-ci.org/docs/zuul/latest/config/pipeline.html) |
| 170 | + |
| 171 | +### Jobs |
| 172 | + |
| 173 | +All jobs that your Zuul instances knows of can be used for your own purposes. |
| 174 | +Call them directly or implement a job that uses an existing job as parent. |
| 175 | +Didn't find the right job? Than we have to create a new one. Existing jobs |
| 176 | +can be found in the web ui of your Zuul instance: [Example](https://zuul.scs.community/t/SCS/jobs) |
| 177 | + |
| 178 | +First have a look on a basic job example: |
| 179 | + |
| 180 | +```yaml |
| 181 | +- job: |
| 182 | + name: base |
| 183 | + parent: null |
| 184 | + description: | |
| 185 | + The recommended base job. |
| 186 | +
|
| 187 | + All jobs ultimately inherit from this. It runs a pre-playbook |
| 188 | + which copies all of the job's prepared git repos on to all of |
| 189 | + the nodes in the nodeset. |
| 190 | +
|
| 191 | + It also sets a default timeout value (which may be overidden). |
| 192 | + pre-run: playbooks/base/pre.yaml |
| 193 | + post-run: |
| 194 | + - playbooks/base/post.yaml |
| 195 | + - playbooks/base/post-logs.yaml |
| 196 | + roles: |
| 197 | + - zuul: zuul/zuul-jobs |
| 198 | + timeout: 1800 |
| 199 | + nodeset: |
| 200 | + nodes: |
| 201 | + - name: ubuntu-jammy |
| 202 | + label: ubuntu-jammy |
| 203 | + |
| 204 | +``` |
| 205 | + |
| 206 | +Each job needs a name that has to be unique within the whole tenant. |
| 207 | +A useful convention to achieve this is to prepend the name of the repository. |
| 208 | +Each job need to define whether there is parent job or not. |
| 209 | +Jobs without a parent are called "base" jobs. Usually you don't want to implement base jobs since |
| 210 | +there are already some base jobs that implement often used stuff. A description may not be mandatory |
| 211 | +but is obviously useful. |
| 212 | + |
| 213 | +Necessary for Zuul to do anything you just need to add a `run` or `roles` property. Within a job that is |
| 214 | +like a `noop` job or just printing something to stdout that is everything you need to run your first job. |
| 215 | +Since anything we want to do requires a little bit more you have to define a nodeset. The nodes |
| 216 | +are used to run your playbooks on. In 99,9% you will need this too. |
| 217 | + |
| 218 | +The properties `pre-run` and `post-run` are useful for bootstrap and cleanup. If your actual job wants to create |
| 219 | +bootstrap some infrastructure you can to this in the `pre-run`. Using an cloud provider you want to release |
| 220 | +no longer used resources. That can be done in the `post-run`. If you are using a parent job it is likely |
| 221 | +that the parent job may has pre- and post-run playbooks. In this case your pre- and post-run playbooks are |
| 222 | +"nested". Example: |
| 223 | + |
| 224 | +1. pre-run parent |
| 225 | +2. pre-run my job |
| 226 | +3. post-run my job |
| 227 | +4. post-run parent |
| 228 | + |
| 229 | +If your job exceeds the defined timeout, the job is considered as failed. |
| 230 | + |
| 231 | +[See official documentation](https://zuul-ci.org/docs/zuul/latest/config/job.html) |
| 232 | + |
| 233 | +#### What about secrets? |
| 234 | + |
| 235 | +Right now you should be able to run basic tasks. But what if you try to test something |
| 236 | +that needs credentials to connect to an outside service? Or you have to address additional |
| 237 | +ressources in an openstack environment and you have to use something like app credentials? |
| 238 | + |
| 239 | +That is where job secrets are used. Example: |
| 240 | + |
| 241 | +```yaml |
| 242 | +- job: |
| 243 | + name: SOME_JOB |
| 244 | + parent: base |
| 245 | + description: | |
| 246 | + A job basic job used as example |
| 247 | + secrets: |
| 248 | + - name: clouds_conf |
| 249 | + secret: app_credential_cloud_conf |
| 250 | + run: playbooks/my-playbook.yaml |
| 251 | +``` |
| 252 | +
|
| 253 | +Secrets for a job are simply defined by the keyword `secrets`. |
| 254 | +Each secret needs a name that can be used in your playbooks. |
| 255 | +The property `secret` references the secret that is defined within your project. |
| 256 | + |
| 257 | +**ATTENTION!** If your job is using a secret `job.post-review` is automatically |
| 258 | +set to `true`. For untrusted projects, that means that your job is only called |
| 259 | +in piplines that have the `pipeline.post-review` flag set to `true`. In SCS context |
| 260 | +that means you may run these jobs only with the pipelines `tag` and `post`. |
| 261 | + |
| 262 | +If you want to run jobs on pipelines that have `post-review` set to `false`, which |
| 263 | +is default, and your job needs a secret, the secret may be defined in the zuul-config repository. |
| 264 | + |
| 265 | +Example: |
| 266 | + |
| 267 | +```yaml |
| 268 | +- secret: |
| 269 | + name: app_credential_cloud_conf |
| 270 | + data: |
| 271 | + credentials: my-secret-value |
| 272 | +``` |
| 273 | + |
| 274 | +Within `my-playbook.yaml` you can reference the secret value using `"{{ clouds_conf.credentials }}"`. |
| 275 | +In this example `my-secret-value` is clear readable text. That is not something we want to keep |
| 276 | +secrets. But how do you encrypt secrets in a way that they are secure and also can be decrypted by |
| 277 | +Zuul? |
| 278 | + |
| 279 | +For this purpose Zuul creates its own public/private key pair for each project. Everyone may use the |
| 280 | +public key to create secrets. But only Zuul will be able to decrypt these values. To avoid the user |
| 281 | +to be responsible for the correct encryption there is an zuul-client tool that will do this for you. |
| 282 | + |
| 283 | +Example: |
| 284 | + |
| 285 | +```bash |
| 286 | +zuul-client --zuul-url ZUUL_URL encrypt --tenant TENANT --project ORGANIZATION/PROJECT --infile creds.yaml --outfile clouds.yaml.enc |
| 287 | +``` |
| 288 | + |
| 289 | +The content may look like this: |
| 290 | + |
| 291 | +```yaml |
| 292 | +- secret: |
| 293 | + name: <name> |
| 294 | + data: |
| 295 | + <fieldname>: !encrypted/pkcs1-oaep |
| 296 | + - IGZ2Wu47R9mEY4fjetbxSAUGNaz4HR1mjk9lCLq3HsUMjHGj9YPlb2MvnPQw1LCJSvpaK |
| 297 | + ogth7hi2zYwrs5tNAik/qlVSB7AM+LQRP7lmlM4JmD6WOyR7DisHu7oMD1Gqem2ZuMggA |
| 298 | + DIBn5+DeBIvnwihDOcS+BKPTVMEtXOJNkuObZHE8DweB/RQIGUvjyeq5yoAmz/y+qGVqe |
| 299 | + 0Vk4pTYFIBgk5DMzwVnDzDkqs/QokoOupMUoBcpapmM11do4ymjbDpeINjayoro6VXTtX |
| 300 | + Mkk9fDv9wuJIQTuyHAOfMD+UYS/HqVSF/Hm9ScUvfhw02gTdzKCxliWhFHJOj7RbdUUMK |
| 301 | + OYYcUkNp5cXZUYFnflMhxVEnzREbdAIklNPfoHOizsxLPaUZ9yk6XcFRflFfMvqBtUS00 |
| 302 | + LCx0Uh906NwdaEUrv2ZdrN123rwfwfw4333232rDFDFfsdfddsfdDFSFSdqrrtwms5Mi0 |
| 303 | + szUBaM4j+Mayep+41vl0cpsLU91GzXEATWMaPIN8OnEHF6qQIv0wB6VaKd5aeAyERisb3 |
| 304 | + wFdjEo4faLO70RWzR33k+4xqAYNIIFyTMpWJz21CUSfoYG8ygL6t7RJGgyjA+0KsVEyj+ |
| 305 | + ewEtiaUOLYyD7pXtqdw1HgzjqiXnfxk+wSv/y5y/TGGYpQj8zU76jS7Zj0ft/0= |
| 306 | +``` |
| 307 | + |
| 308 | +You may use this content or the file to provide it as a secret. You just have to update the `<name>` and the |
| 309 | +`<fieldname>` part. |
| 310 | + |
| 311 | +Official documentation: |
| 312 | + |
| 313 | +1. [Secrets documentation](https://zuul-ci.org/docs/zuul/latest/config/secret.html#secret) |
| 314 | +2. [Encryption documentation](https://zuul-ci.org/docs/zuul/latest/project-config.html#encryption) |
| 315 | + |
| 316 | +#### Let's put it all together |
| 317 | + |
| 318 | +For a basic but working example the following content may be written into a `zuul.yaml` file. |
| 319 | + |
| 320 | +```yaml |
| 321 | +# zuul.yaml content |
| 322 | +
|
| 323 | +--- |
| 324 | +- secret: |
| 325 | + name: mySecret |
| 326 | + data: |
| 327 | + secretValue: !encrypted/pkcs1-oaep |
| 328 | + - <ENCYPTED_DATA> |
| 329 | +
|
| 330 | +- job: |
| 331 | + name: myFirstTestJob |
| 332 | + parent: base |
| 333 | + secrets: |
| 334 | + - name: secretName # The name of the secret that is used within "playbooks/testPlaybook.yaml" |
| 335 | + secret: mySecret |
| 336 | + run: playbooks/testPlaybook.yaml |
| 337 | +
|
| 338 | +- project: |
| 339 | + check: |
| 340 | + jobs: |
| 341 | + - myFirstTestJob |
| 342 | +``` |
| 343 | + |
| 344 | +This will run you job `myFirstTestJob` when ever the `check` pipeline is triggered. |
| 345 | +Within SCS this pipeline is always triggered if you open, change or reopen a pull request. |
| 346 | +The `check` pipeline can also be triggered manually if you write a comment on an already |
| 347 | +existing pull request and place the string `recheck` in it. |
| 348 | + |
| 349 | +The path to you playbook is always the full path within the repository. The playbook |
| 350 | +contains the tasks you actually want to run on all or a specific subset of nodes. |
| 351 | +Example playbook: |
| 352 | + |
| 353 | +```yaml |
| 354 | +# playbooks/testPlaybook.yaml content |
| 355 | +
|
| 356 | +--- |
| 357 | +- hosts: all |
| 358 | + tasks: |
| 359 | + - debug: |
| 360 | + msg: "Debug print my secrets! {{ secretName.secretValue }}" # do not do this as it will expose your secrets |
| 361 | +``` |
0 commit comments