Pipelines 101

This guide showcases how to create a simple pipeline plan - learn by doing. The goal is to make a pipeline that will compile a Maven project, and upload the .jar to persistent storage.

First Steps

First, create the pipeline through the CLI:

pipeline add tutorial

Now, set it to public (permissions are explained in Pipelines 201:

pipeline pub tutorial true

Now, go to the dashboard and start editing its configuration.

Stages

Cloning

As a first step in the pipeline, let's clone the repository:

stages:
  - name: Checkout
    image: alpine/git

This uses the a Docker image that has access to git as the stage. The name is just a human friendly description of what it does. Let's give it the clone functionality:

stages:
  - name: Checkout
    image: alpine/git
    script:
      - git clone https://github.com/Arraying/Guardian.git /home/work

The script array will allow you to define an arbitrary amount of commands to execute inside the container. In our case, we will clone a Java repository into the /home/work folder. /home/work is the working directory of the pipeline. This folder is guaranteed to be copied from one stage to the next. It must be specified here, since the alpine/git image tries to clone to /git by default.

Compiling

Let's add another step to compile and package the code. It's as simple as the first one!

stages:
  - name: Checkout
    image: alpine/git
    script:
      - git clone https://github.com/Arraying/Guardian.git /home/work
  - name: Compile
    image: maven
    script:
      - mvn package

Note that we do not have to specify /home/work here. Candor automatically sets the WORKDIR to /home/work, which can get picked up by Maven. Whether or not you need to explicitly mention /home/work depends on the image used.

Archiving

The pipeline compiles the code, but we want to retain the .jar. The file path can be specified to achieve this.

archive:
  - "target/guardian-1.0.0-SNAPSHOT.jar"
stages:
  - name: Checkout
    image: alpine/git
    script:
      - git clone https://github.com/Arraying/Guardian.git /home/work
  - name: Compile
    image: maven
    script:
      - mvn package

archive takes an arbitrary amount of paths (relative to /home/work of files (not directories) to archive.

Archived files will be uploaded to runId/filename in S3, where runId is the pipeline run's ID and filename is the base file name of the file to be archived. Note that when archiving, everything will be flattened: path structures in the working directory are disregarded. For example, foo/bar.txt and baz/bar.txt both resolve to bar.txt and will overwrite eachother. As a workaround, archived files should be renamed before achiving. Furthermore, if folders are specified, these will be skipped and not uploaded to S3.