ceph-build

A repository for Ceph (and Ceph-related projects) so that they can be automatically configured in Jenkins.

The current state of the repo is of transition from single scripts to a properly structured one with directories that represent each project.

The structure is strict and provides a convention to set the order of execution of build scripts.

Job configuration is done via the CLI app Jenkins Job Builder on the actual directory for its own job (the job has its definition and its build process automated).

The JJB configuration defines the rules needed to generate and update/create all other Jenkins Jobs in this repo as long as they define the config/definitions along with a valid YAML file.

This script should have all the rules and requirements for generating the Jenkins configurations needed from the YAML files to create/update the jenkins job.

Deprecation

Any script in the top level of this repo has been removed, and temporarily placed in the deprecated branch. If a job requires any of those it should be ported to follow the structure of the Jenkins Job Builder project, like all the current jobs in this repository.

The deprecated branch will be removed by the end of 2018.

Any jobs removed from this repo will be automatically deleted by JJB.

Enforcement

The rules and structure for the builds are strictly enforced. If the convention is not followed, the builds will not work.

Changing Jenkins jobs in Jenkins is strongly discouraged. Changing something in the Jenkins UI does not guarantee it will persist and will probably be overwritten.

By default, this is how a directory tree would look like for a build for a project called foo that uses every choice available:

foo
├── config
|   ├── config
|   └── definitions
|       └── foo.yml
├── setup
|   ├── setup
|   ├── post
|   └── pre
└── build
    ├── build
    ├── post
    └── pre

This structure consists of two directories with scripts and one for configuration. The scripts should be included in the foo.yml file in whatever order the job requires.

For example, this is how it could look in the builders section for its configuration:

builders:
  # Setup scripts
  - shell: !include-raw ../../setup/pre
  - shell: !include-raw ../../setup/setup
  - shell: !include-raw ../../setup/post
  # Build scripts
  - shell: !include-raw ../../build/pre
  - shell: !include-raw ../../build/build
  - shell: !include-raw ../../build/post

These scripts will be added to the Jenkins server so that they can be executed as part of a job.

Job Naming Conventions

Each Jenkins job has two names:

1. The main name for a job. This is the name: parameter in YAML.
2. The human-friendly "display name" for a job. This is the display-name: parameter in YAML.

For regular jobs, we name the Jenkins job after the git repository name. For example, the "ceph-deploy" package is at https://github.com/ceph/ceph-deploy, so the job name is "ceph-deploy".

For Pull Request jobs, we use a similar convention for both the internal job name and the human readable "display name". For example, if the git repository is "ceph-deploy", then we name the Jenkins job ceph-deploy-pull-requests. The display-name is set to ceph-deploy: Pull Requests. In other words, to determine a display-name for a job that handles pull requests, simply append : Pull Requests to the name value.

In other words, for building pull requests to ceph-deploy, the Jenkins job YAML will have the following settings:

• Git repo: https://github.com/ceph/ceph-deploy
• Jenkins job name: ceph-deploy-pull-requests
• Jenkins job display-name: ceph-deploy: Pull Requests

Scripts

Scripts that may hang should be using the timeout command:

timeout 600 ./bad-script.sh

The above command will make the job expire after ten minutes (the argument is in seconds).

Pull Request Jobs

When configuring a new job that will build pull requests, you must also configure GitHub's repository to notify Jenkins of new pull requests.

1. In GitHub's web interface, click the "Settings" button for your repository.
2. Click the "Webhooks & Services" link in the "Options" menu on the left.
3. Under the "Webhooks" section, set the "Payload URL" to http://jenkins.ceph.com/ghprbhook/.
4. Click the "Content type" dropdown and select application/x-www-form-urlencoded.
5. For the question "Which events would you like to trigger this webhook?", select the Let me select individual events. radio, and check the Pull Request and Issue comment boxes.
6. Click the green "Update Webhook" button to save your changes.

On the Jenkins side, you should set up the job's GitHub project URL like so:

- job:
    name: jenkins-slave-chef-pull-requests

    ...

    properties:
      - github:
          url: https://github.com/ceph/jenkins-slave-chef

This will tell the Jenkins GitHub Pull Requests plugin that it should associate the incoming webhooks with this particular job.

You should also use the triggers setting for the job, like so:

- job:
    name: jenkins-slave-chef-pull-requests

    ...

    triggers:
      - github-pull-request:
          cron: '* * * * *'
          admin-list:
            - alfredodeza
            - ktdreyer
          org-list:
            - ceph
          trigger-phrase: 'retest this please'
          only-trigger-phrase: false
          github-hooks: true
          permit-all: false
          auto-close-on-fail: false

"Document" Jobs

Some jobs don't actually run code; they simply build a project's documentation and upload the docs to ceph.com. One example is the "teuthology-docs-build" job.

For these jobs, note that the destination directory must be created on the ceph.com web server before the rsync command will succeed.

Polling and GitHub

Jenkins can periodically poll Git repos on github.com for changes, but this is slow and inefficient. Instead of polling GitHub, it's best to use GitHub's web hooks instead.

See the "jenkins-job-builder" job as an example.

1. Set up the triggers section:

   triggers:
     - github
   

2. Visit the GitHub repository's "settings" page, eg https://github.com/ceph/ceph-build/settings/hooks, and add a new web hook.

   • The Payload URL should be https://jenkins.ceph.com/github-webhook/ (note the trailing slash)
   • The Content type should be application/x-www-form-urlencoded
   • Secret should be blank
   • Select Just send the push event.

Testing JJB changes by hand, before merging to main

Sometimes it's useful to test a JJB change by hand prior to merging a pull request.

1. Install jenkins-job-builder on your local computer.

2. Create $HOME/.jenkins_jobs.ini on your local computer:

   [jenkins]
   user=ktdreyer
   password=a8b767bb9cf0938dc7f40603f33987e5
   url=https://jenkins.ceph.com/
   

Where user is your Jenkins (ie GitHub) account username, and password is your Jenkins API token. (Note, your Jenkins API token can be found @ https://jenkins.ceph.com/ , for example https://jenkins.ceph.com/user/ktdreyer/configure)

3. Switch to the Git branch with the JJB changes that you wish to test:

   git checkout <branch with your changes>
   

Let's say this git branch makes a change in the my-cool-job job.

4. Run JJB to test the syntax of your changes:

   jenkins-jobs --conf ~/.jenkins_jobs.ini test my-cool-job/config/definitions/my-cool-job.yml
   

   If everything goes ok, this will cause JJB to output the XML of your job(s). If there is a problem, JJB will print an error/backtrace instead.

5. Run JJB to push your changes live to job on the main:

   jenkins-jobs --conf ~/.jenkins_jobs.ini update my-cool-job/config/definitions/my-cool-job.yml
   

6. Run a throwaway build with your change, and verify that your change didn't break anything and does what you want it to do.

(Note: if anyone merges anything to main during this time, Jenkins will reset all jobs to the state of what is in main, and your customizations will be wiped out. This "by-hand" testing procedure is only intended for short-lived tests.)

Assigning a job to a different Jenkins Master

We found one Jenkins controller wasn't enough to handle all the jobs we were demanding of it. The CI now supports multiple Jenkins controllers. If you wish to run your job on a different Jenkins controller:

1. Create a config/JENKINS_URL file in your job directory containing only the FQDN of the target Jenkins controller:

   # Example
   $ cat my-cool-job/config/JENKINS_URL
   2.jenkins.ceph.com
   

A note on inclusive language

Like many software projects, the Ceph project has undertaken the task of migrating to more inclusive language. In the Ceph CI,

master branches are now main

slave is now builder

When referring to the main Jenkins server, master is now controller

Remaining references (like the Jenkins ssh-slaves plugin) are hardcoded and could not be changed.