Garage Kubernetes Operator

S3-Compatible Object Storage on Kubernetes

A Kubernetes operator for Garage - distributed, self-hosted object storage with multi-cluster federation.

• Declarative cluster lifecycle — StatefulSet, config, and layout managed via CRDs
• Bucket & key management — create buckets, quotas, and S3 credentials with kubectl
• Multi-cluster federation — span storage across Kubernetes clusters with automatic node discovery
• Gateway clusters — stateless S3 proxies that scale independently from storage
• COSI driver — optional Kubernetes-native object storage provisioning

Custom Resources

CRD	Description
GarageCluster	Deploys and manages a Garage cluster (storage or gateway)
GarageBucket	Creates buckets with quotas and website hosting
GarageKey	Provisions S3 access keys with per-bucket permissions
GarageNode	Fine-grained node layout control (zone, capacity, tags)
GarageAdminToken	Manages admin API tokens

Install

helm install garage-operator oci://ghcr.io/rajsinghtech/charts/garage-operator \
  --namespace garage-operator-system \
  --create-namespace

Quick Start

First, create an admin token secret for the operator to manage Garage resources:

kubectl create secret generic garage-admin-token \
  --from-literal=admin-token=$(openssl rand -hex 32)

Create a 3-node Garage cluster (full example):

apiVersion: garage.rajsingh.info/v1alpha1
kind: GarageCluster
metadata:
  name: garage
spec:
  replicas: 3
  zone: us-east-1
  replication:
    factor: 3
  storage:
    data:
      size: 100Gi
  network:
    rpcBindPort: 3901
    service:
      type: ClusterIP
  admin:
    enabled: true
    bindPort: 3903
    adminTokenSecretRef:
      name: garage-admin-token
      key: admin-token

Wait for the cluster to be ready:

kubectl wait --for=condition=Ready garagecluster/garage --timeout=300s

Create a bucket:

apiVersion: garage.rajsingh.info/v1alpha1
kind: GarageBucket
metadata:
  name: my-bucket
spec:
  clusterRef:
    name: garage
  quotas:
    maxSize: 10Gi

Create access credentials:

apiVersion: garage.rajsingh.info/v1alpha1
kind: GarageKey
metadata:
  name: my-key
spec:
  clusterRef:
    name: garage
  bucketPermissions:
    - bucketRef: my-bucket
      read: true
      write: true

Or grant access to all buckets in the cluster — useful for admin tools, monitoring, or mountpoint-s3 workloads that span multiple buckets:

apiVersion: garage.rajsingh.info/v1alpha1
kind: GarageKey
metadata:
  name: admin-key
spec:
  clusterRef:
    name: garage
  allBuckets:
    read: true
    write: true
    owner: true

Per-bucket overrides layer on top of allBuckets, so you can combine cluster-wide read with owner on a specific bucket:

  allBuckets:
    read: true
  bucketPermissions:
    - bucketRef: metrics-bucket
      owner: true

Get S3 credentials:

kubectl get secret my-key -o jsonpath='{.data.access-key-id}' | base64 -d && echo
kubectl get secret my-key -o jsonpath='{.data.secret-access-key}' | base64 -d && echo
kubectl get secret my-key -o jsonpath='{.data.endpoint}' | base64 -d && echo

Gateway Clusters

Gateway clusters handle S3 API requests without storing data. They connect to a storage cluster and scale independently, ideal for edge deployments or handling high request volumes. See gateway examples for more configurations.

apiVersion: garage.rajsingh.info/v1alpha1
kind: GarageCluster
metadata:
  name: garage-gateway
spec:
  replicas: 5
  gateway: true
  connectTo:
    clusterRef:
      name: garage  # Reference to storage cluster
  replication:
    factor: 3       # Must match storage cluster
  admin:
    enabled: true
    adminTokenSecretRef:
      name: garage-admin-token
      key: admin-token

Key differences from storage clusters:

• Uses a StatefulSet with metadata PVC for node identity persistence (no data PVC)
• Registers pods as gateway nodes in the layout (capacity=null)
• Requires connectTo to reference a storage cluster
• Lightweight and horizontally scalable

For cross-namespace or external storage clusters, use rpcSecretRef and adminApiEndpoint:

connectTo:
  rpcSecretRef:
    name: garage-rpc-secret
    key: rpc-secret
  adminApiEndpoint: "http://garage.storage-namespace.svc.cluster.local:3903"
  adminTokenSecretRef:
    name: storage-admin-token
    key: admin-token

External Storage (NAS, Bare Metal)

To connect a gateway to a Garage instance running outside Kubernetes (e.g., on a NAS or bare-metal server), use bootstrapPeers instead of clusterRef. Get the node ID from your external Garage with garage node id.

apiVersion: garage.rajsingh.info/v1alpha1
kind: GarageCluster
metadata:
  name: garage-gateway
spec:
  replicas: 2
  gateway: true
  replication:
    factor: 3  # Must match the external cluster
  connectTo:
    rpcSecretRef:
      name: garage-rpc-secret
      key: rpc-secret
    bootstrapPeers:
      - "563e1ac825ee3323aa441e72c26d1030d6d4414aeb3dd25287c531e7fc2bc95d@nas.local:3901"
  admin:
    enabled: true
    adminTokenSecretRef:
      name: garage-admin-token
      key: admin-token

The gateway pods will connect to the external nodes via the RPC port and register as gateway nodes in the existing cluster layout.

Multi-Cluster Federation

Garage supports federating clusters across Kubernetes clusters for geo-distributed storage. All clusters share the same RPC secret and Garage distributes replicas across zones automatically.

1. Create the same RPC secret in every Kubernetes cluster:

   SECRET=$(openssl rand -hex 32)
   kubectl create secret generic garage-rpc-secret --from-literal=rpc-secret=$SECRET

2. Configure remoteClusters and publicEndpoint on each GarageCluster:

   apiVersion: garage.rajsingh.info/v1alpha1
   kind: GarageCluster
   metadata:
     name: garage
   spec:
     replicas: 3
     zone: us-east-1
     replication:
       factor: 3
     network:
       rpcSecretRef:
         name: garage-rpc-secret
         key: rpc-secret
     publicEndpoint:
       type: LoadBalancer
       loadBalancer:
         perNode: true
     remoteClusters:
       - name: eu-west
         zone: eu-west-1
         connection:
           adminApiEndpoint: "http://garage-eu.example.com:3903"
           adminTokenSecretRef:
             name: garage-admin-token
             key: admin-token
     admin:
       enabled: true
       adminTokenSecretRef:
         name: garage-admin-token
         key: admin-token

The operator handles node discovery, layout coordination, and health monitoring across clusters. Each cluster needs a publicEndpoint so remote nodes can reach it on the RPC port. See the Garage documentation for networking requirements.

COSI Support (Optional)

The operator includes an optional COSI (Container Object Storage Interface) driver that provides Kubernetes-native object storage provisioning.

Enabling COSI

1. Install the COSI CRDs:

   for crd in bucketclaims bucketaccesses bucketclasses bucketaccessclasses buckets; do
     kubectl apply -f "https://raw.githubusercontent.com/kubernetes-sigs/container-object-storage-interface/main/client/config/crd/objectstorage.k8s.io_${crd}.yaml"
   done

2. Deploy the COSI controller:

   kubectl apply -k "github.com/kubernetes-sigs/container-object-storage-interface/controller?ref=main"

3. Install the operator with COSI enabled:

   helm install garage-operator oci://ghcr.io/rajsinghtech/charts/garage-operator \
     --namespace garage-operator-system \
     --create-namespace \
     --set cosi.enabled=true

Using COSI

1. Create a BucketClass:

   apiVersion: objectstorage.k8s.io/v1alpha2
   kind: BucketClass
   metadata:
     name: garage-standard
   spec:
     driverName: garage.rajsingh.info
     deletionPolicy: Delete
     parameters:
       clusterRef: garage
       clusterNamespace: garage-operator-system

2. Create a BucketAccessClass:

   apiVersion: objectstorage.k8s.io/v1alpha2
   kind: BucketAccessClass
   metadata:
     name: garage-readwrite
   spec:
     driverName: garage.rajsingh.info
     authenticationType: Key
     parameters:
       clusterRef: garage
       clusterNamespace: garage-operator-system

3. Request a bucket:

   apiVersion: objectstorage.k8s.io/v1alpha2
   kind: BucketClaim
   metadata:
     name: my-bucket
   spec:
     bucketClassName: garage-standard
     protocols:
     - S3

4. Request access credentials:

   apiVersion: objectstorage.k8s.io/v1alpha2
   kind: BucketAccess
   metadata:
     name: my-bucket-access
   spec:
     bucketAccessClassName: garage-readwrite
     protocol: S3
     bucketClaims:
       - bucketClaimName: my-bucket
         accessMode: ReadWrite
         accessSecretName: my-bucket-creds

5. Use the credentials in your application:

   env:
   - name: S3_ENDPOINT
     valueFrom:
       secretKeyRef:
         name: my-bucket-creds
         key: COSI_S3_ENDPOINT
   - name: AWS_ACCESS_KEY_ID
     valueFrom:
       secretKeyRef:
         name: my-bucket-creds
         key: COSI_S3_ACCESS_KEY_ID
   - name: AWS_SECRET_ACCESS_KEY
     valueFrom:
       secretKeyRef:
         name: my-bucket-creds
         key: COSI_S3_ACCESS_SECRET_KEY

COSI Limitations

• Only S3 protocol is supported
• Only Key authentication is supported (no IAM)
• Bucket deletion requires the bucket to be empty first
• Upstream COSI controller does not yet implement deletion — DriverDeleteBucket and DriverRevokeBucketAccess are implemented but won't be called until upstream adds support

Documentation

• Helm Chart - Installation and configuration
• Garage Docs - Garage project documentation

Development

make dev-up       # Start kind cluster with operator
make dev-test     # Apply test resources
make dev-status   # View cluster status
make dev-logs     # Stream operator logs
make dev-down     # Tear down