gocd-yaml-config-plugin

GoCD plugin to keep pipelines and environments configuration in source-control in YAML. See this document to find out what are GoCD configuration repositories.

This is the second GoCD configuration plugin, which enhances some of shortcomings of JSON configuration plugin

Format is concise. Unlike JSON, there are no unnecessary quotations ", brackets {} and commas ,
Tries to enforce correctness where possible. By using maps instead of lists and shorter object graphs.
Allows to have multiple pipelines and environments in single file. But doesn't force it.
Comments in configuration files - YAML supports comments, so you can explain why pipeline/environment it is configured like this.

Setup

Step 1: GoCD versions newer than 17.8.0 already have the plugin bundled. You don't need to install anything.

If you're using GoCD version older than 17.8.0, you need to install the plugin in the GoCD server. Download it from the releases page and place it on the GoCD server in plugins/external directory.

Step 2: Follow the GoCD documentation to add a new configuration repository.

You can use the example repository at: https://github.com/tomzo/gocd-yaml-config-example.git.

In your config repo (tomzo/gocd-yaml-config-example.git in this case), ensure that your GoCD yaml config is suffixed with .gocd.yaml. Any file ending in .gocd.yaml is picked up by the plugin. Give it a minute or so for the polling to happen. Once that happens, you should see your pipeline(s) on your dashboard.

Example

More examples are in test resources.

#ci.gocd.yaml
format_version: 9
environments:
  testing:
    environment_variables:
      DEPLOYMENT: testing
    secure_variables:
      ENV_PASSWORD: "s&Du#@$xsSa"
    pipelines:
      - example-deploy-testing
      - build-testing
pipelines:
  mypipe1: # definition of mypipe1 pipeline
    group: mygroup # note that the group name can contain only of alphanumeric & underscore characters
    display_order: 10
    label_template: "${mygit[:8]}"
    lock_behavior: none
    parameters: # list of parameters that can be configured for a pipeline
      param1: value1
    materials:
      mygit: # this is the name of material, the name can contain only of alphanumeric & underscore characters
        # keyword git says about type of material and url at once
        git: https://my.example.org/mygit.git
        branch: ci
      myupstream: # this name does not matter, but there should be no 2 materials with the same name
        # type is optional here, material type is implied based on presence of pipeline and stage fields
        # type: dependency
        pipeline: pipe2
        stage: test
    stages: # list of stages in order
      - build: # name of stage
          clean_workspace: true
          jobs:
            csharp: # name of the job
              resources:
               - net45
              artifacts:
               - build:
                   source: bin/
                   destination: build
               - test:
                   source: tests/
                   destination: test-reports/
               - test:
                   source: coverage.xml
              tabs:
                report: test-reports/index.html
              tasks: # ordered list of tasks to execute in job csharp
               - fetch:
                   pipeline: pipe2
                   stage: build
                   job: test
                   source: test-bin/
                   destination: bin/
               - exec: # indicates type of task
                   command: make
                   arguments:
                    - "VERBOSE=true"
               # shorthand for script-executor plugin
               - script: ./build.sh ci

File pattern

The default pattern is **/*.gocd.yaml,**/*.gocd.yml, which will recursively search the entire repository for all files ending with .gocd.yaml or .gocd.yml.

You can set a custom file pattern per configuration repository using the GoCD configuration UI:

Or in the config XML using <configuration>:

<config-repos>
  <config-repo pluginId="yaml.config.plugin" id="repo1">
    <git url="https://github.com/tomzo/gocd-yaml-config-example.git" />
    <configuration>
      <property>
        <key>file_pattern</key>
        <!-- comma-separate multiple patterns -->
        <value>pipelines/build/*.yml,pipelines/lint/*.yml</value>
      </property>
    </configuration>
  </config-repo>
</config-repos>

Validation

You can validate if proposed GoCD YAML changes will be accepted by the server. Currently, 2 options are available:

Use a GoCD mergable github action
Validate from your local machine

Validation using CLI

You may find this introductory blog post useful.

There is an ongoing effort to allow in-depth validation of configuration before pushing configuration to the source control. This is provided by GoCD's preflight API and gocd-cli.

You have several options to configure validation tools on your workstation:

If you have a local docker daemon, use the gocd-cli-dojo image. Follow the setup instructions in the image readme.
If you don't want to use docker, you'll need to setup gocd-cli on your host.

Either way you'll have gocd binary in your PATH or inside the docker container.

Syntax validation

This will check general validity of the yaml file, without talking to the GoCD server:

gocd configrepo syntax --yaml pipeline.gocd.yaml

Preflight validation

This command will parse and submit your yaml file to the configured GoCD server.

gocd configrepo preflight --yaml -r dotnet-dojo pipeline.gocd.yaml

Where -r is the configuration repository id, which you have earlier configured on GoCD server. You can check it on config repos page of your GoCD server, at /go/admin/config_repos. It is in the upper left corner of each config repo.

For new repositories, when you haven't added config repo to the server yet, you can also check if it is correct, just skip the -r argument.

gocd configrepo preflight --yaml pipeline.gocd.yaml

Format reference

See official GoCD XML configuration reference for details about each element. Below is a reference of format supported by this plugin. Feel free to improve it!

Format version
Environment
Environment variables
Parameters
Pipeline
Stage
- Approval
Job
Tasks
- rake
- ant
- nant
- exec
- fetch
- pluggabletask
- script
Materials
- dependency
- package
- git
- svn
- perforce
- tfs
- hg
- pluggable scm
- config repo
Secure variables
Retrieving secrets from Secrets Management plugins in values
YAML Aliases

Format version

Please note that it is now recommended to declare format_version in each gocd.yaml file, consistent across all your files.

GoCD server version from 20.8.0 and beyond

Supports format_version value of 10. This version removes usage of blacklist/whitelist keywords in favour of includes/ignore. This is done in a backwards compatible way - you can upgrade the server and plugin first, and update the format_version to 10 after. Just replace whitelist by includes, and blacklist by ignore.

Additionally, plugin is also "forwards compatible" since version 0.13.0. If you can't upgrade GoCD to 20.8.0, but want to migrate from blacklist/whitelist to includes/ignore, you can upgrade the plugin and keep the format_version that you are using. Plugin will migrate keywords under the hood, while your YAML can use includes/ignore.