apm-pipeline-library

Jenkins pipeline shared library for the project APM

(root)
+- src                     # Groovy source files
|   +- co
|       +- elastic
|           +- Bar.groovy  # for org.foo.Bar class
+- vars
|   +- foo.groovy          # for global 'foo' variable
|   +- foo.txt             # help for 'foo' variable
+- resources               # resource files (external libraries only)
|   +- co
|       +- elastic
|           +- bar.json    # static helper data for org.foo.Bar
+- local                   # to enable jenkins linting locally
|   +- configs
|       +- jenkins.yaml
|   +- docker-compose.yml
|   +- Dockerfile
+- test-infra                  # to validate our CI workers
|   +- test_<os|generic>.yml   # the specific tests/asserts to run in each worker.

• Pipeline
• Pipeline shared library
• Steps Documentation

Requirements

In order to test the library you need Maven 3 installed, also, it is possible to use the Maven wrapper available in .mvn folder.

mvn test

mvn test -Dtest=CLASS#TEST_NAME

./mvnw test

./mvnw test -Dtest=CLASS#TEST_NAME

In order to build the release notes it is need tho install gren

Create a new step

We have several steps created that can be used on our Jenkins pipelines, this allow us to reuse common processes along Jenkins pipelines. These are the common steps we should follow to create a new step:

• Create a new groovy file in vars/dummy.groovy
• Create a new help file in vars/dummy.txt
• Create a new test for the step in src/test/groovy/DummyStepTests.groovy
• Update the steps README.md by executing ./resources/scripts/generateReadme.sh vars/

Those steps should satisfy the following characteristics:

• It does only one thing
• It does it well
• It is short
• It is reusable

In some cases, we need to make complex task, for those cases we can use classes and should be created in the folder src/co/elastic.

Test the new step

To test a step, use the supplied Maven wrapper at the root of the project. For example, to run tests contained in src/test/groovy/DummyStepTests.groovy, execute the following from the directory at the root of this project:

./mvnw test -Dtest=DummyStepTests

To execute all tests, use:

./mvnw test

To run tests and print additional debug output to the console, use the -Pdebug flag:

./mvnw test -Dtest=DummyStepTests -Pdebug

Release a version

Every time there are enough changes, we would release a new version. A version has a name like v[:number:].[:number:].[:number:] see Semantic Versioning.

Automated release process 🚀 (preferred)

Navigate to the APM Pipeline Library job and choose Build with Parameters. Select the make_release checkbox and click Build. The build will take ~1 hour to complete.

Manual release process 👨 (replaced by the automated process above)

To create a new release please use Maven Release Plugin, which uses the pom.xml file to store the semantic version for this project.

mvn release:prepare release:perform

This command will bump the current SNAPSHOT, commit changes, and push the tag to upstream repository, as declared in the release.properties file.

Apart from the creation of that tag, we must update the current tag, pointing to the same version we just created. The current tag is used to use the last stable library version on pipelines.

git checkout master
git pull origin master
git fetch --all
git tag -f current
git push -f --tags

Finally update the Release notes and Changelog

./resources/scripts/jenkins/release-notes.sh

Upgrade repository maven wrapper

mvn -N io.takari:maven:0.7.6:wrapper -Dmaven=3.3.3

Local Development

If you'd like to speed up your local development process then you can configure your local environment.

Setup intellij idea

Open the project in IntellijIdea as a groovy project if possible, then start coding.

Setup atom

If you use atom then install https://atom.io/packages/linter-jenkins. If you click on atom://settings-view/show-package?package=linter-jenkins then you can either install it or configure it.

Then configure the CURL method which should point out to http://0.0.0.0:18080

Linting

Run a jenkins local instance as explained below:

cd local
docker-compose up --build -d

Validate whether it works as expected:

curl --silent -X POST -F "jenkinsfile=<.ci/Jenkinsfile" http://0.0.0.0:18080/pipeline-model-converter/validate

Precommit

This particular process will help to evaluate some linting before committing any changes. Therefore you need the pre-commit.

Installation

Follow https://pre-commit.com/#install and pre-commit install

Some hooks might require some extra tools such as:

• shellcheck
• yamllint

Enabled hooks

• Check case conflict
• Check executables have shebangs
• Check merge conflicts
• Check json
• Check yaml
• Check xml
• Check bash syntax
• End-of-file-fixer
• Ensure neither abstract classes nor traits are used in the shared library.
• Ensure JsonSlurperClassic is used instead of non-serializable JsonSlurper.
• Jenkinsfile linter.
• yamllint
• shellcheck
• Detect unicode non-breaking space character U+00A0 aka M-BM-
• Remove unicode non-breaking space character U+00A0 aka M-BM-
• Detect the EXTREMELY confusing unicode character U+2013
• Remove the EXTREMELY confusing unicode character U+2013

Validate JJBB files

If the local jenkins instance has been enabled then it's possible to validate whether the JJBB files are healthy enough.

Prepare test environment by first changing to the local/ directory and running:

  make start

Logs for the running Jenkins instance can then be viewed if you wish by running:

  make logs

To run the JJBB locally you must ensure that you have an /etc/hosts entry which maps jenkins to localhost.

To prepare to test most pipelines, you must first set up the APM jobs folder:

  sh local/test-jjbb.sh -j .ci/jobs/apm-shared.yml

  sh local/test-jjbb.sh -j .ci/jobs/apm-docker-images-pipeline.yml

Then open http://localhost:18080

Debugging can be made easier by passing -ldebug to test-jbb.sh.

pre-commit-hooks

Observability robots hooks for http://pre-commit.com/

Using these hooks

Add this to your .pre-commit-config.yaml

    - repo: https://github.com/elastic/apm-pipeline-library
      rev: current
      hooks:
      -   id: check-bash-syntax
      -   id: check-abstract-classes-and-trait
      -   id: check-jsonslurper-class
      -   id: check-jenkins-pipelines
      -   id: check-unicode-non-breaking-spaces
      -   id: remove-unicode-non-breaking-spaces
      -   id: check-en-dashes
      -   id: remove-en-dashes
      -   id: check-gherkin-lint

Available hooks

• check-bash-syntax - Check Shell scripts syntax corectness, requires bash
• check-abstract-classes-and-trait - Ensure neither abstract classes nor traits are used
• check-jsonslurper-class - Ensure JsonSlurperClassic is used instead of non-serializable JsonSlurper
• check-jenkins-pipelines - Check the syntax of the Jenkinsfiles, requires docker and jenkins up and running.
• check-unicode-non-breaking-spaces - Detect unicode non-breaking space character U+00A0 aka M-BM-
• remove-unicode-non-breaking-spaces - Remove unicode non-breaking space character U+00A0 aka M-BM-
• check-en-dashes - Detect the EXTREMELY confusing unicode character U+2013
• remove-en-dashes - Remove the EXTREMELY confusing unicode character U+2013
• check-gherkin-lint - Check Gherkin feature syntax corectness, requires docker.

Test Infra

This is how we test the actual state of our CI workers that are configured with Ansible. Therefore, we can validate whether the CI worker templates have been configured with the expected requirements

This particular implementation uses testinfra.

Resources

• Pipeline User Handbook
• Pipeline Development Tools
• Jenkins Pipeline Unit testing framework
• Groovy Testing guide
• Using Docker with Pipeline
• Jenkins World 2017: How to Use Jenkins Less
• Jenkins World 2017: JenkinsPipelineUnit: Test your Continuous Delivery Pipeline
• Pipeline Examples
• Jenkins Pipelines and their dirty secrets
• Introduction to Declarative Pipelines
• CD with CloudBees Core Workshop
• Introducing Blue Ocean: a new user experience for Jenkins
• Blueocean (BO) documentation
  • Dashboard
  • Activity
  • Pipeline Details
• IntelliJ Setup for Jenkins Development
  • Declarative Pipeline GDSL WiP
• Command line tool to run Jenkinsfile locally
• Autocomplete and embedded library documentation for Visual Studio Code