[WIP] bundle: fix bundle create, bump operator-registry v1.6.0

[estroz]

Description of the change: some cleanup of bundle code, bump operator-registry to v1.6.0, add --output-dir (from #2662), various other changes described in the motivation section.

• doc,website: update generated CLI docs
• CHANGELOG.md: --output-dir addition and breaking change for bundle.Dockerfile location
• doc/migration: add breaking change to migration guide

Motivation for the change: From #2662:

Description of the change:

*Update operator-registry dependency to 1.6.0
*Adds fix for generated dockerfiles
*Adds optional output-directory parameter

Motivation for the change:

Include fixes for operator-sdk bundle create

This PR fixes an issue where manifests/ is expected by the operator-registry image builder but not generated by the SDK. In case deploy/olm-catalog/<operator-name>/manifests doesn't exist, create one and delete it after the image is done building. If --generate-only is set, do not remove manifests/. --version lets a user specify which operator version they want to build an image for, and --latest uses the highest semver'd bundle.

This change partially allows the SDK to support the new manifests format for bundles while being backwards-compatible with versioned bundle directories. Still some work to do on the CSV generation side of things, but at least bundle create now works with current projects.

TODO:

• discuss breaking changes in behavior
• CHANGELOG.md updates
• test flow from bundle create [--generate-only] -> bundle validate
• update CLI and usage docs

/area dependency

[estroz]

@kevinrizza the v1.6 update breaks --generate-only behavior, since the argument to --directory, which is for a named bundle dir ex deploy/olm-catalog/my-operator/0.0.1, is placed in a top-level bundle.Dockerfile. Running operator-sdk bundle create --directory deploy/olm-catalog/my-operator/0.0.2 after generating a metadata/Dockerfile for 0.0.1 results in a bundle for 0.0.1.

How does opm work with a directory structure like this?

[kevinrizza]

we had a weird implementation where the metadata/ folder was getting inserted inside the manifest folder and then the dockerfile had to extract it out. we modified it so that the scaffold matched what actually appeared on the directory structure in the container image.

that was one of the big purposes of this update. You shouldn't be creating versioned folders, instead you should have a single manifests/ folder and a matching metadata/ folder. the registry api update pushes the metadata/ folder to the same directory level as the manifest/ folder

[estroz]

So users are expected to use a VCS to manage bundle versions going forward, and everything in manifests/ is always latest?

@joelanford @jmccormick2001 @dmesser we're going to have to make a serious breaking change to the CSV generator/dir structure unless we can come up with a middle ground.

[kevinrizza]

Yes, this is what i was trying to say for a while. The CI pipelines are going to be built around having just one bundle in your repository and when you want to push a change you just update that bundle -- this is for a few reasons, but the big one is because the operator bundles are just single versions, and if we didn't do it this way we would have to come up with a bunch of complex CI that would need to know that a new folder was generated in a specific subdirectory and know how to aggregate it. Also, the annotations.yaml file is aggregated at the specific bundle version and not at the package layer.

[camilamacedo86]

Hi @kevinrizza,

move your bundle Dockerfile at <project-root>/deploy/olm-catalog/<operator-name>/Dockerfile to <project-root>/bundle.Dockerfile

Note that what you are asking for the user does will break the commands for SDK cli.

if we didn't do it this way we would have to come up with a bunch of complex CI that would need to know that a new folder was generated in a specific subdirectory and know how to aggregate it.

Would not be possible to add flags there to pass the current location and the CI job looking for the files in the path informed and/or copy them for anyplace according to this needs?

[estroz]

@kevinrizza #2746 helps users migrate to a manifests/ format while supporting versioned bundle dirs. Next is for generate csv to do the same.

[estroz]

/hold

Waiting to resolve #2715 (comment)

[estroz]

/kind bug

[estroz]

Now blocked by operator-framework/operator-registry#237 since we'll have to bump versions once a new one is released with this fix.

[estroz]

Closing in favor of #2746