Foreman's Jenkins Jobs

This repository contains all the job definitions and supporting code used in Jenkins jobs used by the Foreman project on it's own ci system ci.theforeman.org and ci.centos.org.

CentOS CI is used to provision VMs for end to end testing of installations.

Jenkins Job Builder

Jenkins Job Builder (JJB) is an OpenStack tool for generating Jenkins job definitions (an XML file) from a set of YAML job descriptions, which we store in version control.

A bootstrap job named jenkins-jobs-update runs the JJB tool to update the jobs in the live instance whenever a change is merged to this repository.

Useful resources:

• Job definitions, templates etc.
• Modules, e.g. SCM, publishers, builders

Jenkins Job Naming conventions

Note Because centos.org is a shared environment all jobs are prefixed by foreman- to denote they're ours.

Name	Convention	Example 1	Example 2
Nightly Source Builder	{git-repo}-{git-branch}-source-release	foreman-develop-source-release	hammer-cli-katello-master-source-release
Nightly Package Builder	{git-repo}-{git-branch}-package-release	foreman-develop-package-release	hammer-cli-katello-master-package-release
CI pipeline	{repository}-{environment}-{optional-concern}-pipeline	foreman-nightly-rpm-pipeline	foreman-nightly-deb-pipeline
Pull Request testing	{git-repo}-{optional-concern}-pr-test	katello-pr-test	foreman-packaging-rpm-pr-test
Branch testing	{git-repo}-{git-branch}-test	foreman-3.5-stable-test	smart-proxy-3.5-stable-test

Job configurations

Testing develop

All repos with an associated job that tests their master/develop branch should have a hook added to the repo to trigger immediate builds.

To set up the hook, an org/repo admin must:

• View the repository settings
• Click Webhooks
• Click Add webhook
  • Payload URL: https://ci.theforeman.org/github-webhook/
  • Content Type: application/json (default)
  • Secret: add from secret store
  • Just the push event

Pull request testing

Core Foreman projects have GitHub pull requests tested on our Jenkins instance so it's identical to the way we test the primary development branches themselves. Simpler and quieter projects (such as installer modules) can use Github Actions instead. There is no obligation for Foreman projects to use Jenkins.

Every project that needs PR testing has at least two Jenkins jobs. Taking core Foreman as an example:

• Test job for the main development branch (develop or master): test_develop
• Test job for each PR: test_develop_pr_core

Github Pull Request Builder

The GHPRB plugin uses webhooks installed on the repo to trigger a build, then it runs any job configured with the GHPRB trigger and a matching GitHub project set.

The plugin tests the latest commit on the PR branch only, it does not merge the PR with the base branch. The webhook may also trigger multiple jobs, and jobs may use different GitHub commit status names to easily test and report status on different types of tests.

PR jobs should be set up identically to primary branch tests, except for the SCM (which checks out ${sha1}) and to add the GHPRB trigger (see the github_pr macro in JJB).

To set up the hook, an org/repo admin goes to the repository settings, then Webhooks & Services and adds a webhook with these settings:

• Payload URL: https://ci.theforeman.org/ghprbhook/
• Content type: application/json
• Secret: redacted
• Events: Let me select individual events, Pull request, Issue comment

An org admin must then change the org teams:

• Add the repository to the Bots team with write access

Quick reference for maintainers

Current PR test jobs (used on Foreman itself) support these commands:

• ok to test - run tests for an unknown user, if the code within the patch is not malicious
• [test STATUS-NAME], e.g. [test foreman] to re-run a particular set of tests

Quick reference for plugin maintainers

Smart proxy plugin testing

Proxy plugins are tested like ordinary gems with tests run entirely from the plugin directory, installing the smart proxy as a dependency (via bundler's git support).

To add testing for a plugin smart_proxy_example do the following:

• ensure plugin tests run when doing bundle install and rake test, see testing a plugin for help
• if different branches rely on different versions of smart proxy, specify :branch in Gemfile on those branches
• add the project in theforeman.org/yaml/jobs/smart-proxy-plugin.yaml to create a smart-proxy-example-test job in Jenkins to test the master branch.
• add the project in theforeman.org/yaml/jobs/smart-proxy-plugin-pr-test.yaml to create a smart_proxy_example-pr-test job in Jenkins to test PRs.
• ensure the job is green by fixing bugs, installing dependencies etc.
• add hook to GitHub repo, see GitHub repo hook

An org admin must then:

• add the repo to the Bots team with write access

Foreman's Other Tests

Jenkins is not the only place tests are defined and executed.

GitHub Actions

Several repositories use GitHub Actions either instead of or together with Jenkins. They commonly use our reusable actions.

The definitions of these jobs are in .github/workflows/ of their respective repositories.

Failed jobs cannot be re-triggered with a comment, only from the GitHub UI which requires maintainer permissions for the repository.

Packit

Several repositories use Packit to produce RPMs based on pull requests.

The definitions of these jobs are in .packit.yaml of their respective repositories.

Failed jobs can be re-triggered with a /packit build comment in the PR.