Github Deployment Workflows

[spbolton]

Task: Enhance CI/CD Pipeline for Evergreen Deployments

Objective: Ensure consistent testing and artifact generation processes, improving security and efficiency in our CI/CD pipeline. This involves using GitHub's deployments and environments functionality, generating multiple build artifacts, and handling multi-architecture deployments effectively.

Background

To achieve evergreen deployments, we need to streamline our CI/CD pipeline, ensuring that pull requests go through consistent testing before merging into the master (trunk) branch. The core CI pipeline generates the main Docker image artifact representing the code at each commit. However, the current post-build workflows are fragmented and inefficient, often rebuilding artifacts from scratch and posing security risks due to broad access to deployment secrets.

Steps

1. Adopt GitHub's Deployments and Environments

   • Use GitHub’s deployment and environments functionality to manage artifact publishing from commits on the trunk branch.
   • Ensure that deployment secrets are restricted and securely managed.

2. Refine the Core CI/CD Build

   • Trigger the core Maven CI/CD build on various events: pull requests, merge queue, master branch commits, and manual triggers.
   • Balance performance and error detection by customizing tests based on the trigger type.
   • Generate and archive build artifacts, including status reports and test results.

3. Separate Build from Deployment

   • Maintain separation of concerns by ensuring the core CI/CD build does not handle final artifact deployment.
   • Run the main CI/CD workflow in a less secure environment to protect deployment secrets, especially for pull requests.

4. Manage Build Artifacts

   • Identify and categorize the following build artifacts:
     • Docker Image -> Dockerhub (multi-architectures)
     • CLI Executables (multiple architectures) -> NPM
     • Maven Artifacts -> Artifactory
     • Documentation (Swagger Doc, Java Doc)
     • DotCMS SDK
   • Use dedicated deployment scripts (e.g., deploy-artifact-dockerhub.yml) for different artifact types and deployment targets.

5. Integrate with DotCMS Infrastructure Deployments

   • Coordinate with existing K8s infrastructure workflows, triggering updates on docker tag changes.
   • Ensure deployment scripts wait for successful completion status before proceeding.

6. Handle Versioned Artifacts

   • Integrate Maven CI-friendly versions to streamline versioned builds.
   • Add conditional logic to the main CI/CD workflow for setting or using provided version numbers.
   • Enable manual, tag-based, or scheduled triggers for versioned builds, promoting artifacts without rebuilding.

7. Support Multi-Architecture Deployments

   • Build and test default architecture in the core CI/CD workflow, with conditional logic for multi-architecture builds.
   • Ensure multi-architecture Docker builds are part of the deployment steps, avoiding source rebuilds.

Checklist

• Update core CI pipeline to use GitHub’s deployments and environments.
• Refactor the core Maven CI/CD build to trigger on various events and balance performance with error detection.
• Ensure build artifacts are properly generated and archived.
• Maintain separation between build and deployment processes.
• Implement dedicated deployment scripts for different artifact types.
• Integrate CI/CD with DotCMS infrastructure deployments.
• Utilize Maven CI-friendly versions for versioned builds.
• Implement multi-architecture support in deployment steps.
• Document the CI/CD pipeline and provide clear instructions for managing deployment secrets and handling conflicts.

Tasks

Beta Give feedback

1. Replace docker-build-master-branch.yml with deployment-trunk.yml #27857
   QA : Not Needed Team : Scout Type : CI/CD +3
2. Replace docker-build-dev-image.yml with build-nightly.yml and deployment-nightly.yml #27858
   Merged QA : Not Needed Team : Scout Type : CI/CD +4
3. Spike Changes and add detail to remaining tasks to integrate full product release into new modular workflow structure #28240
   QA : Not Needed Team : Scout +2
4. Update reusable-ci-build.yml to be triggered with version and create versioned artifacts #27859
   Team : Scout Type : CI/CD +2
5. Create build-release-candidate.yml and deploy-release-candidate.yml #27860
   Team : Scout Type : CI/CD +2
6. Create deploy-release.yml #27861
   Team : Scout Type : CI/CD +2
7. Add check to PR workflow (initialize step) to fail if .mvn/maven.config exists #28559
   Team : Scout +1
8. Update documentation and flowchart for new workflows #29454
   +0
9. Create reusable-promote.yml to promote and tag changes from source branch to destination #28548
   Team : Scout Type : CI/CD +2
10. Convert nightly build to a versioned build #29455
    +0
11. Create build-test-release.yml triggered on push to release branch to build deploy and run release actions #28549
    Team : Scout Type : CI/CD stale +3
12. Create deploy-artifact-javadoc composite action based on Generate/Push Javadoc step in maven-release-process.yml #28550
    Team : Scout Type : CI/CD +2
13. Update build-report.yml to handle release notification #28551
    Team : Scout Type : CI/CD +2
14. Create finalize-release composite action based on Create Release step in maven-release-process.yml #28552
    Team : Scout Type : CI/CD +2
15. Pull in cli-release-process into integrated release in build-test-release.yml #28553
    Team : Scout Type : CI/CD +2
16. Refactor github workflows to group key top level workflows and reusable components into separate folders #28554
    Gardening Day QA : Not Needed Team : Scout Type : CI/CD +4
17. Create deploy-update-plugins composite action based on Update Plugins step in maven-release-process.yml #28555
    Team : Scout Type : CI/CD +2
18. Move deployment secrets to github "environments" #28557
    Team : Scout Type : CI/CD +2

Additional Notes

• Ensure that deployment secrets are managed securely and access is restricted.
• Provide clear instructions for handling potential conflicts during the upgrade process.
• Maintain a rollback plan to address any critical issues arising from the deployment.