Implement the plain+v0 bundle provisioner

[timflannagan]

This is largely a tracking issue for implementing the first rukpak provisioner.

A skeleton of this provisioner was introduced in #7 which creating several no-op controllers for the Bundle and ProvisionerClass resources. These controllers are largely responsible for watching for ProvisionerClass custom resources that match this k8s bundle provisioner unique ID, and any Bundle that specifies the metadata.Name of an existing k8s bundle ProvisionerClass.

Likely next steps for a POC:

• Documentation around the k8s bundle format
• Documentation around the usage of the k8s bundle provisioner
• Create additional GH actions (e.g. running e2e tests)
• Create a mechanism for pulling and storing remote Bundle content (Add support for pulling and storing bundle content #12)
  • The pulling process is done through CRI
    • Needs to support manifestlist (i.e. multi-arch) images
  • Determine mechanism for storing content:
    • An aggregate API server
    • An in-cluster database
    • Everything is housed in a local PV
    • Bundle content is sharded across ConfigMap(s)
      • Storing bundle content is a single ConfigMap doesn't scale due to etcd size limits for a single resource
    • HTTP/gRPC API for GET/POST-ing bundle content
      • Provisioner ships a singleton filesystem server with a local PV mounted that houses resources
      • Bundles can be stored as tar files after being pulled and unpacked during creation time
    • Custom CSI driver for mounting Bundle images
      • Investigate usage of https://github.com/warm-metal/csi-driver-image
• Create a mechanism for retrieving and creating stored Bundle content (Add mechanism for exposing bundle content to end users #13)
  • Ordering of resources can be communicated by prefixing filenames with 0<n>_<file_name> (e.g. 01_namespace.yaml) similar to how kubectl currently operates
  • Provide retry mechanism if resources fail to be created
  • Determine how updates to the Bundle container image are handled
    • Require container images to be pinned using sha instead of tags?
    • Track the sha value in the Bundle status?
  • Example implementation
    • Read from the storage bundle directory, decode the opened file descriptor into an unstructured.Unstructured resource use controller-runtime's dynamic client to create this resource
  • Determine how to handle namespaced-scoped resources missing a metadata.Namespace specification
• Introduce YAML manifests for installing the rukpak CRDs and requisite provisioner resources
  • The provisioner ships with a default ProvisionerClass custom resource that references the unique k8s bundle provisioner ID
  • RBAC around provisioner's ServiceAccount resource can have wide range of permissions

Follow-up work:

• Creation of an o-f enhancement
• Implement and document the Instance API
  • Additional work can introduce defining nested Bundle specs within a single Instance resource
  • Creation of Bundle content is gated on authorization of content
  • Support android style permission handling
• Implement a kubectl plugin for manually approving/denying Bundle resources
• Support referencing local Bundle content
• Support referencing multiple Bundle sources
• More restrictive RBAC given to the provisioner

Potential future work:

• Helm-based provisioner
• OLM-based provisioner

Open Questions:

• What would be the default authorization mechanism for the default k8s bundle provisioner ProvisionerClass?
• How do we communicate dependencies on other Bundle's being present on-cluster?
• What's the ideal mechanism for providing a way to update stored Bundle content without rebuilding the referenced container image?
• Is there a way to run the k8s bundle provisioner off-cluster?
• Where does authorization of an individual Bundle's content live? The Instance resource's status conditions or a top-level field in the status sub-resource?

[timflannagan]

I'm going to keep this open as an overall tracking ticket for essentially productizing the plain+v0 provisioner, as it doesn't look like we have any Projects that are detailing this kind of functionality. We can close this out once the issues that are tracked in the 0.1 milestone have all been closed out.

[exdx]

Closing as this has largely been implemented up to this point.