Creation of buckets, accounts, and policies as custom resources

[vbr-cts]

Is your feature request related to a problem? Please describe.

I use minio as a backend for a highly multi-tenant SaaS app. We rely on fine-grained policies to make sure each service only accesses whatever it is needs to access. Currently, we provision, manage and revoke these accesses by calling the created tenant's API. This has several drawbacks:

• we maintain provisioning scripts outside our classic helm workflow
• disaster recovery is complicated as we need to re-create all ACLs and accounts
• developer experience and CI is complicated as we can't spin up an entire clean env from a helm chart
• modifying existing policies is complex and error-prone

Describe the solution you'd like

I'd like to have something where I can have custom resources for:

• buckets
• policies
• service accounts

These would be linked to a minio tenant by name.

Ideally, the service account CRDs would point to a secret by namespace and name, and the key would be stored there (or read from there if it already exists).

This way, when deploying a new tenant, I can have helm charts generate a MinioBucket, a MinioServiceAccount and the required MinioPolicies onto a pre-existing minio tenant. The services would wait for the secret described in the MinioServiceAccount to be populated by the operator before starting.

Similarly:

• creating/deleting the MinioBucket does just that
• modifying a MinioPolicy updates the policies of linked buckets/service accounts

Describe alternatives you've considered

• bunch of scripts: currently doing this, not great
• s3-operator (https://github.com/agill17/s3-operator) might be an option, but it is unmaintained, and having it play nicely with minio-operator's multi-tenancy is proving really hard, it seems like we need to deploy a separate operator for each minio tenant.
• integration within our own (in-house) operator: can work, but we'd rather have

[MarkusBauerBE]

Seconding this - iirc the reason this wasn't implemented was because it was to be outsourced.
I've personally solved this by running a K8s job that scans the content of a configmap. This is then deployed via Helm and a Helm hook triggers the job to run, whenever the CM gets updated (only adding the buckets that don't already exist).
It works, but this should have frankly been implemented as a crd a long long time ago.

[mbrancato]

I mentioned this in the Slack channel prior to seeing this, but I think this would definitely further align this operator with Kubernetes principles of declarative configuration. As it currently stands, I think the Operator console will sync its tenant changes back to the cluster as Tenant resources. This would be extremely useful to have a similar setup when a bucket, user, policy, etc is created.

[mbrancato]

I wanted to add some use-case context here based on what I've read is required for the operator today. For me, I almost never do a kubectl apply, let alone deploying a kubectl plugin for Minio and then running kubectl minio init or something to install. There is also a helm chart, which I can work with easily on a declarative basis using the HelmRelease resource. Even in a testing environment, which I'm trying to deploy here, I don't do anything manually. We deploy all kubernetes configurations and resources using GitOps, and nobody is directly applying resources.

To get a little deeper in this, when I consider using a tool, I start to think about reproducibility. Can I tear this environment down and stand it back up easily? Outside a few small things related to bootstrapping an environment, there are no manual commands run. Infrastructure is built using automated tools and then an in-cluster CD operator like flux picks up the kubernetes resource configs, and the principle of eventual consistency takes hold. Take for example, I want to deploy a new environment, configure it, and that environment has some app that uses a Minio bucket. Where this starts to break down is we would have failures in rebuilding the environment without basically a second bootstrapping step to run some scripts to create the users, policies, and buckets.

[addisonautomates]

Please support using configmaps, secrets, or some custom crds for scrubbing to provision buckets and policies after initial deployment. This is the only tool we run on kubernetes that I can't just use Flux to instantly get to a fully configured state. Having to fire kubernetes jobs to run scripts is an anti-pattern that kubernetes and progressive deployment tools are trying to avoid. Take a look at how grafana handles the creation of new folders and dashboards to get a good idea how to organize things. Would love to be able to add to a master configmap or crd resource for bucket creation or be able to add new configmaps with bucket references and have minio pick them up as long as a certain kubernetes label is used. The minio client is being treated as too foundational for configuring the environment.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[MyIgel]

That is still an issue :)

[D1StrX]

Any updates on this from Minio? Also running in the issue that adding buckets/users doesn't provision when upgrading with the helm chart minio/tenant. Lack of Policies definition support through the values.yml is still a +1 requested feature.

[WoodyWoodsta]

I'm here for this as well. I'm not entirely sure what the use is of the Tenant CRD as a k8s IaC resource object if there isn't the ability to manage the tenant via this object after provisioning.

[sathieu]

You can manage buckets and policies with: https://github.com/InseeFrLab/s3-operator

[kamikaze]

+1