Skip to content

Latest commit

 

History

History
193 lines (172 loc) · 15.7 KB

UsingOurScripts.md

File metadata and controls

193 lines (172 loc) · 15.7 KB
Raw

Custom environment setting up guide

Adopt have setup their build scripts so that you can plug in configuration files and scripts you have changed while not having to duplicate and maintain Adopt's entire codebase separately. This may seem complicated at first but it's pretty simple once you get the hang of the process.

defaults.json

This file contains the default constants and paths used in the build scripts for whichever repository it is located in. As an example, Adopt's defaults.json file is located here. If you're unsure of any of the fields, see Adopt's example map below:

{
    // Git repository details
    "repository"             : {
        // Git Url of the temurin-build repository.
        "build_url"          : "https://github.com/adoptium/temurin-build.git",
        // Git branch you wish to use when running the shell scripts inside the build_url
        "build_branch"       : "master",
        // Smoke tests directory under the build repo
        "test_dirs"          : "/test/functional",
        // Git Url of the current repository.
        "pipeline_url"       : "https://github.com/adoptium/ci-jenkins-pipelines.git",
        // Git branch you wish to use when running the groovy scripts inside the pipeline_url
        "pipeline_branch"    : "master",
        // Git branch name from the https://github.com/adoptium/jenkins-helper.git repo. This can only be as branch or tag, not SHA1
        "helper_ref"      : "master"
    },
    // Jenkins server details
    "jenkinsDetails"         : {
        // The base URL of the server, usually this is where you would end up if you opened your server from a webpage
        "rootUrl"            : "https://ci.adoptium.net",
        // Jenkins directory where jobs will be generated and run
        "rootDirectory"      : "build-scripts"
    },
    // Jenkins job dsl template paths (relative to this repository root)
    "templateDirectories"    : {
        // Downstream job template (e.g. jdk8u-linux-x64-hotspot)
        "downstream"         : "pipelines/build/common/create_job_from_template.groovy",
        // Upstream job template (e.g. openjdk8-pipeline)
        "upstream"           : "pipelines/jobs/pipeline_job_template.groovy",
        // Weekly job template (e.g. weekly-openjdk8-pipeline & weekly-evaluation-openjdk17-pipeline)
        "weekly"             : "pipelines/jobs/weekly_release_pipeline_job_template.groovy",
        // Release job template (e.g release-opendjk11-pipeline)
        "release"            : "pipelines/jobs/release_pipeline_job_template.groovy"
    },
    // Job configuration file paths (relative to this repository root)
    "configDirectories"      : {
        // Build configs directory containing node details, os, arch, testlists, etc
        "build"              : "pipelines/jobs/configurations",
        // Nightly configs directory containing execution frequency, weekly tags, platforms to build.
        "nightly"            : "pipelines/jobs/configurations",
        // Release configs directory containing platforms to build.
        "release"            : "pipelines/jobs/configurations",
        // Evaluation configs directory containing execution frequency, weekly tags, platforms to build.
        "evaluation"          : "pipelines/jobs/configurations",
        // Bash platform script directory inside build_url containing jdk downloading and toolchain setups.
        "platform"           : "build-farm/platform-specific-configurations"
    },
    // Job script paths (relative to this repository root)
    "scriptDirectories"      : {
        // Upstream scripts directory containing the 1st files that are executed by the openjdkx-pipeline jobs.
        "upstream"           : "pipelines/build",
        // Upstream script file containing the 1st script that is executed by the weekly-openjdk8-pipeline jobs.
        "weekly"             : "pipelines/build/common/weekly_release_pipeline.groovy",
        // Downstream script file containing the 1st script that is executed by the jdkx-platform-arch-variant jobs.
        "downstream"         : "pipelines/build/common/kick_off_build.groovy",
        // Base script file containing the 2nd script that is executed by the pipeline_jobs_generator_jdkxx jobs
        "regeneration"       : "pipelines/build/common/config_regeneration.groovy",
        // Base PR tester file script file containing the 2nd script that is executed by the pipeline_jobs_generator_jdkxx jobs
        "tester"             : "pipelines/build/prTester/pr_test_pipeline.groovy",
        // Shell script that builds machine farm, pulling in platform configs and setting Boot JDK roots. Executed by the openjdk_build_pipeline.groovy
        "buildfarm"          : "build-farm/make-adopt-build-farm.sh"
    },
    // Job base file (the main file which is called after the 1st setup script file) paths (relative to this repository root)
    "baseFileDirectories"    : {
        // Upstream pipeline file script containing the 2nd script that is executed by the openjdkx-pipeline jobs
        "upstream"           : "pipelines/build/common/build_base_file.groovy",
        // Upstream pipeline file script containing the 2nd script that is executed by the jdkx-platform-arch-variant jobs
        "downstream"         : "pipelines/build/common/openjdk_build_pipeline.groovy"
    },
    // Test suite and inclusion details
    "testDetails"            : {
        // Boolean denoting whether the reproducible compare is needed
        "enableReproducibleCompare" : false,
        // Boolean denoting whether pipeline tests will be enabled or disabled by default
        "enableTests"            : true,
        /*
        Test targets triggered in 'nightly' build pipelines running 6 days per week
        'weekly' to be run on the weekend and 'release' from a release pipeline
        */
        "nightlyDefault"     : [
            "sanity.openjdk",
            "sanity.system",
            "extended.system",
            "sanity.perf",
            "sanity.functional",
            "extended.functional"
        ],
        /*
        Test targets triggered in 'weekly' build pipelines running once per week
        weekly + jck to be run during a 'weekly' pipeline
        weekly to be run during 'evaluation weekly' pipeline
        */
        "weeklyDefault"     : [
            "sanity.openjdk",
            "sanity.system",
            "extended.system",
            "sanity.perf",
            "sanity.functional",
            "extended.functional"
            "extended.openjdk",
            "extended.perf",
            "special.functional",
            "dev.openjdk",
            "dev.functional"
        ],
        /*
        Test targets triggered in 'release' build pipelines
        release + jck to be run during a 'release' pipeline
        */
        "releaseDefault"     : [
            "sanity.openjdk",
            "sanity.system",
            "extended.system",
            "sanity.perf",
            "sanity.functional",
            "extended.functional"
            "extended.openjdk",
            "extended.perf",
            "special.functional"
        ]
    },
    // Raw content URL of this (defaults.json) file. This is so the openjdk_build_pipeline.groovy script can set user default configs when checking out to the shell script repo
    "defaultsUrl"            : "https://raw.githubusercontent.com/adoptium/ci-jenkins-pipelines/master/pipelines/defaults.json"
}

How do I know which parameter the jenkins job will use?

The scripts have been designed with a set hierarchy in mind when choosing which parameter to use:

1. JENKINS PARAMETERS (highest priority, args entered here will be what the build scripts use over everything else)
2. USER JSON (medium priority, args entered here will be used when a jenkins parameter isn't entered)
3. ADOPT JSON (final priority, when jenkins parameters AND a user json arg can't be validated, the script will checkout to this repository and use Adopt's defaults json (linked above))

The ADOPT JSON level is only used for files and directories. Other parameters (JOB_ROOT, JENKINS_BUILD_ROOT, etc) only use the first two levels.

As an example, take a look at the build-pipeline-generator SCRIPT_FOLDER_PATH parameter:

Image of the SCRIPT_FOLDER_PATH parameter in jenkins The script will use whatever has been entered into the parameter field unless it has been left empty, in which case it will use whatever is in the user's defaults.json['scriptDirectories']['upstream'] attribute.

It will then evaluate the existence of that directory in the user's repository and, if it fails to find one, will checkout to adoptium/ci-jenkins-pipelines and use Adopt's defaults.json (the console log will warn the user of this occurring):

[WARNING] pipelines/build/common/weekly_release_pipeline.groovy does not exist in your chosen repository. Updating it to use Adopt's instead

NOTE: For the defaults that are paths to directories, the scripts will search for files of the same name as Adopt's. Custom named files are not currently supported (so for defaults.json['configDirectories']['platform'], all of the filenames in the specified folder need to be the same as Adopt's or the script will fail to pick up the user's config's and will use Adopt's instead).

This is great, but how do I add new defaults?

Create a temurin-build PR that adds the new defaults in for what they would be for Adopt. Don't forget to update Adopt's RepoHandlerTest.groovy and fakeDefaults.json, as well as any jenkins jobs if needs be (if you don't have configuration access, ask in Slack#build for assistance).

You may find that the RepoHandlerTest.groovy:adoptDefaultsGetterReturns() will fail when you add new values. This is expected as the test is pulling in Adopt's master branch defaults.json which does not yet contain the new values. Please inform any reviewers of this in the pull request.

Finally, update any scripts that will need to handle the new default, you will likely need to do a bit of searching through the objects mentioned in Adopt's defaults.json to find where Adopt's scripts will need changing.

Once it has been approved and merged, update your scripts and/or jenkins jobs to handle the new default and you're done!

Starting from scratch

  1. Create a (preferably) public repository with whatever scripts/configs you have altered. You don't need to place them in the same place as where Adopt's ones are, but they should have the same name. Currently, the list of supported files (replacing x with the JDK version number you want to alter, (u) is optional) you can modify are:
  2. Create a User JSON file containing your default constants that the build scripts will use (see #the defaults.json)
  3. Copy the build-pipeline-generator and pipeline_jobs_generator_jdk8u jobs to your Jenkins instance (replace jdk8u with whichever version you intend to build, there should be one job for each jdk version).
  4. Execute the copied build-pipeline-generator. Make sure you have filled in the parameters that are not covered by your defaults.json (e.g. DEFAULTS_URL, CHECKOUT_CREDENTIALS). You should now see that the nightly and weekly pipeline jobs have been successfully created in whatever folder was entered into JOB_ROOT
  5. Execute the copied pipeline_jobs_generator_jdkxx jobs. Again, make sure you have filled in the parameters that are not covered by your defaults.json. You should now see that the jobs/jdkxx-platform-arch-variant jobs have been successfully created in whatever folder was entered into JOB_ROOT

Congratulations! You should now be able to run Adopt's scripts inside your own Jenkins instance.