[Enhancement] Generate Input Manifest to support multiple major versions

[peterzhuamazon]

This is a follow up issue to

• Add placeholder plugins when auto-generating next version's manifests #3707

We have a discussion on this and will go with approach 2 + Removing the input manifest checks section as the distribution build already taken care of the issue creation:

Approach 1 : Populate all the entries of the new input manifest, from corresponding templates

• Steps:

  • Using the existing manifest templates located here: https://github.com/opensearch-project/opensearch-build/tree/main/manifests/templates
  • Update all templates to include every component from the latest release of their respective versions.
    • 1.x uses 1.3.14 as base
    • 2.x uses 2.11.1 as base
    • 3.x uses 3.0.0 as base
  • Update src/manifests_workflow to generate a new manifest based on the corresponding template, then replace the version number of the release and git reference of each component

• Pros:

  • Easy and simple approach, with PR ready: Allow manifest templates to include all the contents while changing version and ref based on core branch properties #4271
  • Consistent output, where component dependencies and build order is always maintained, and reviewed
  • Minimal changes on both the python code and the github action workflow
  • Allow each major version to have its own template, to avoid current situation where 1.x/2.x is using 3.x as base

• Cons:

  • A manual PR is required to add a new component
  • Adding all components at once, when not all of them have bumped their versions, can lead to build failures if the --continue-on-error parameter is not set

Approach 2: Populate all the entries of the new input manifest, from latest input manifest of major version
(Will approach)

• Steps:
  • Update Python code, to find the latest manifest path of corresponding major version. e.g. If 1.x, use 1.3.14
  • Copy that manifest, then replace the version number of the release and git reference of each component
• Pros:
  • Even easier approach then 1, as you do not need to maintain templates
  • Even smaller code changes to both the python code and the github action workflow.
  • Just edit latest version search logic to look for the latest version within the same major version category
  • Do not need manual PR to add new components for new manifest
• Cons:
  • Relies on the latest version of the manifest to be correct, and maintain dependencies
  • Not reliable enough
  • Adding all components at once, when not all of them have bumped their versions, can lead to build failures if the --continue-on-error parameter is not set

Approach 3: Use Approach 1 + Add Placeholder parameter

• Steps:
  • Same as Approach 1
  • Check every component, to see if version bump is available
  • Add placeholder parameter as true to plugins that needs version bump
  • Update build_workflow to take placeholder parameter into account

components:
  - name: OpenSearch
    repository: https://github.com/opensearch-project/OpenSearch.git
    ref: 2.x
  - name: common-utils
    repository: https://github.com/opensearch-project/common-utils.git
    ref: 2.x
    placeholder: true # this is not a real component!

• Pros:
  • Easy and simple approach, with PR ready: Allow manifest templates to include all the contents while changing version and ref based on core branch properties #4271
  • Consistent output, where component dependencies and build order is always maintained, and reviewed
  • Minimal changes on both the python code and the github action workflow
  • Allow each major version to have its own template, to avoid current situation where 1.x/2.x is using 3.x as base
  • Build and Assemble workflow will skip placeholder: true plugins
• Cons:
  • A manual PR is required to add a new component
  • Placeholder: true is nothing different from --continue-on-error
  • Complicated implementation compares to both Approach 1 & 2

Approach 4: Use Approach 1 or 2 + Only adding components that has bumped the version already

• Result will be the same as now, nothing changes

[peterzhuamazon]

PRs:

• Remove opensearch 2.12.0/1.3.15 checks as distribution-build already taken care of issue reporting #4381
• Generate new input manifest with previous latest version #4523
• Revert "[AUTO] Update input manifests (#4522)" to rerun the generation from #4523 #4526

[peterzhuamazon]

Runs well:

• https://build.ci.opensearch.org/job/manifest-update/338/console

[peterzhuamazon]

Re-open to rerun the generation with 1.3.16 using new logics in #4523.

[peterzhuamazon]

https://build.ci.opensearch.org/job/manifest-update/340/console

[peterzhuamazon]

• [AUTO] Update input manifests #4527

[peterzhuamazon]

Runs well, will close this issues for now.