Skip to content

Support multiple Hetzner Cloud projects in one cluster (configurable driver name) #1436

Description

@cbcoutinho

Use case

I run a single Kubernetes cluster whose nodes are spread across two separate Hetzner Cloud projects. This is a first-class pattern for managed offerings built on hcloud — Cloudfleet (CFKE), for example, models each provider account as a distinct "fleet" and autoscales nodes into each, so one cluster routinely spans two or more hcloud projects with independent API tokens.

Nodes from both projects join the same cluster and are individually selectable (each carries a nodepool label, and the provider encodes the owning account in the providerID). Compute works fine. Storage does not: I would like a workload pinned to project A's nodes to provision volumes in project A, and one pinned to project B's nodes to provision in project B.

To be explicit about scope: I am not asking for volume migration between projects. Volumes are project-scoped and that's fine — moving data across accounts can be a manual, out-of-band operation. I only want each project's nodes to be able to provision and attach volumes within their own project.

What doesn't work today

The natural approach — install the chart twice, once per token, with a distinct StorageClass each and mutually exclusive node selectors on the node DaemonSets — cannot work, because the driver name is a compile-time constant:

// internal/driver/driver.go
PluginName = "csi.hetzner.cloud"

It's returned by GetPluginInfo and there's no flag or env override. That single constant blocks the approach at several layers at once:

  1. StorageClass routing. A StorageClass selects a controller solely via provisioner: <driverName>. Two installs both claim csi.hetzner.cloud, so two StorageClasses backed by different tokens are indistinguishable to the storage layer — there's no parameter that says "use this controller."
  2. Leader election. Both external-provisioner sidecars derive their lease from the provisioner name, so two releases in the same namespace contend for one lease. A single leader wins and serves every PVC with its token.
  3. CSIDriver object. Cluster-scoped and named csi.hetzner.cloud, so the two releases fight over one object.
  4. Kubelet plugin path. plugins/csi.hetzner.cloud/socket is shared, so two node DaemonSets on the same node collide on registration.

The failure mode is unpleasant because it passes scheduling. Nodes in both projects advertise the same csi.hetzner.cloud/location=<region> topology, so CSI reports the node as compatible and the pod schedules happily onto the "wrong" project's node. Only the attach fails, and it fails as:

AttachVolume.Attach failed ... code = NotFound
desc = failed to publish volume: server not found

...because the controller resolved a server ID belonging to the other project against its own token. Nothing in the storage path consults the providerID, which is where the real account boundary lives. There is also a quieter version of this: a node DaemonSet with broad tolerations and no node selector will run on the other project's nodes and register those servers' IDs into the shared driver name, so CSINode entries silently reference servers the controller cannot see.

Proposed change

Make the driver name configurable, defaulting to csi.hetzner.cloud so existing installs are unaffected:

  • Thread a flag/env (e.g. --driver-name / DRIVER_NAME) through to PluginName, defaulting to the current value.
  • Template the corresponding chart references: the CSIDriver object name, the StorageClass provisioner, and the --kubelet-registration-path / plugin socket directory.

Note that TopologySegmentLocation is currently derived as PluginName + "/location". Deriving it from the configured name keeps each install's topology keys distinct, which is what stops the cross-project "compatible node" mis-scheduling described above — worth deciding deliberately rather than leaving it pinned to the default.

That would let a second install run as, say, driver project-b.csi.hetzner.cloud with StorageClass hcloud-volumes-project-b, node DaemonSets constrained to their own project's nodes by label. Workloads then pick a project by pairing a node selector with the matching StorageClass, and the two controllers never see each other's PVCs.

Alternatives considered

  • Consolidating both projects into one — works, but defeats the point when the accounts are separate for billing or ownership reasons.
  • Keeping one project stateless-only — what I do today; it means half the cluster's capacity can't host stateful workloads.
  • A generic storage layer (Longhorn/OpenEBS) on the second project — introduces a separate failure domain and gives up hcloud volumes entirely.

Happy to work on a PR if the maintainers are open to the direction — the change looks additive and behaviour-preserving by default, but I'd rather confirm the approach (particularly the TopologySegmentLocation question) before writing code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions