Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor workflow for common java and maven build steps #29137

Merged

Conversation

spbolton
Copy link
Contributor

@spbolton spbolton commented Jul 5, 2024

This change combines and removes a lot of common functionality and steps in our workflow process making the whole thing easier to understand and work with. Our main workflow relies on an initial maven build step that generates artifacts that can be then pulled in and used in subsequent test, deploy and release steps. We do this to allow these test stages to be run in parallel and we can then aggregate the results. To achieve this we need to use consistent naming conventions for the artefacts and to provide artefacts and caches that enable the steps to run quickly without redoing work. There are therefore common steps that we needed to add to each job and the amount of duplication of these steps causes many issues to understand or to make consistent changes system-wide. To use common DRY principles we can pull out these common functionalities into separate functions (GitHub composite actions) that allow for consistent behavior.

Part of the push for this are:

  1. In moving to Java 21 we want a consistent way to set the Java version to use by default across the build, whether a major build or a minor java version without having a mix of versions.
  2. Some caches have not been consistently used throughout the build requiring the config for cache load and save to be the same, and to know when we should save vs load caches.

Proposed Changes

  • Create setup-java composite action handling all java installation using sdkman.
    • allow override of java version
    • specify if graalvm is required (for native builds)
    • uses default grallvm 21.0.2-graalce, but enables override from a parameter
  • merge separate cleanup-runner actions into a single action that covers multiple OS
  • create prepare-runner composite action handling common tasks for all runners
    • option to setup master branch reference
    • option to cleanup runner
    • setup java by default but option to not
  • create maven-job composite action providing common functionality to all workflow steps running maven tasks
    • Handle setup native builds
    • Handle License if required for step
    • Control of caches, maven, nodejs ( node cache restore can be disabled if no node build steps are required )
    • Generate or pull in artifacts to use between initial build and subsequent steps e.g. .m2/repository
    • generation and restore of docker image artifacts
    • Handle build report artifact generation automatically using naming convention
    • Flag to say that test results are generated and creation of report artifact using naming convention
  • Use .sdkmanrc as default for obtaining jvm to use in runner
  • Update workflows and reusable workflows to use updated composite actions
  • Fix mac os runners (do not use -latest versions that could change ) x86 build not using x86 runner. Sdkman uses version appropriate for the architecture it is running on.
  • Combine 2 stage "validate" profile into single build and validate, fix validate steps and ordering.
  • pull release runner jar from artifacts created in Initial Build and stored in .m2/repository artifact
  • Changes to support use of common build steps with a non snapshot version that can be commited to .mvn/maven.config
    • supports locally created docker image to use release tag not just snapshot
    • Safety check to ensure that PR cannot be commited with .mvn/maven.config as this should only be created in the release process.
  • Protection added to ensure the initial build does not update source files that have not been committed with the PR. This prevents issues where other developers merge from master and end up with uncommitted changes without any code changes of their own.
  • Adds generation of .nvmrc file in project root to set the defaults at the project level like .sdkmanrc.
    • Note: .nvmrc file is generated from values in the nodejs-parent/pom.xml file whereas .sdkmanrc is currently the manually updated source file used within the build. We may want to choose which direction is best for both. The protection in initial build for changed files during build will prevent the .nvmrc file from being merged that does not match the version defined in Maven in case someone manually edits the file or does not run a build after modifying the version in the pom file

Example usage of maven-job


- uses: ./.github/actions/maven-job
        with:
          stage-name: "Build Native Image ${{ matrix.label }}"
          maven-args: "package -Pnative -Pdist -DskipTests=$SKIP_TESTS -pl :dotcms-cli"
          native: true
          generates-test-results: false
          artifacts-from: ${{ env.ARTIFACT_RUN_ID }}
          version: ${{ inputs.version }}
          

This resolves multiple issues

#28736
#27998

and combines the changes in #28916

Checklist

  • Tests
  • Translations
  • Security Implications Contemplated (add notes if applicable)

Additional Info

Tests of post PR workflows have been done on https://github.com/dotCMS/core-workflow-test including Master Checks, dotCLI Release

Screenshots

Original Updated
** original screenshot ** ** updated screenshot **

This PR fixes: #27998

@spbolton spbolton requested a review from a team as a code owner July 5, 2024 11:20
@spbolton spbolton force-pushed the issue-27998-task-update-workflow-to-support-any-java-version branch from 1a2a5c1 to dc308f9 Compare July 5, 2024 11:28
@spbolton spbolton marked this pull request as draft July 5, 2024 11:28
@spbolton spbolton force-pushed the issue-27998-task-update-workflow-to-support-any-java-version branch 12 times, most recently from 77dfbfd to 251118a Compare July 10, 2024 09:35
@spbolton spbolton force-pushed the issue-27998-task-update-workflow-to-support-any-java-version branch from d0d9edb to d85a0a6 Compare July 11, 2024 16:30
@spbolton spbolton marked this pull request as ready for review July 11, 2024 18:26
Copy link
Contributor

@dcolina dcolina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Copy link
Contributor

@wezell wezell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't have much to add here. Let's see how this runs.

Copy link

@spbolton spbolton enabled auto-merge July 15, 2024 11:01
@spbolton spbolton added this pull request to the merge queue Jul 15, 2024
Merged via the queue into master with commit c6c0834 Jul 15, 2024
30 checks passed
@spbolton spbolton deleted the issue-27998-task-update-workflow-to-support-any-java-version branch July 15, 2024 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants