Create control plane without publicly addressable IPs

[anurag]

As an operator I'd like to create clusters with nodes that are completely isolated from the public internet. Instead, they should only be accessible through authorized IPs or bastion nodes.

Detailed Description

In CAPP's current implementation, all control plane and worker nodes have public IP addresses, kubelets are configured to speak the public IP address for the control plane, and anyone on the internet can attempt connections to control plane nodes. While TLS+authentication add a layer of security, it would be nice to have the ability to create cluster nodes without public IP addresses, analogous to GKE private clusters. All egress traffic would be directed to NAT gateways and inbound traffic will only be allowed from authorized IPs..

/kind feature

[displague]

From an Equinix Metal product offering perspective, this is possible in one of two ways.

Every Equinix Metal device has a management address assigned to the first bond, by default. This per device /30 (by default) assignment is part of a 10.x.x.x/25 address range isolated to device peers within the same EM project and facility (a /56 is also created as an IPv6 management range). This https://github.com/equinix/terraform-metal-anthos-on-baremetal project used this solution, keeping the node communication on the 10.x.x.x network.

Alternatively, a device may be provisioned with VLANs. Until recently, all EM Devices used to have to toggle Layer2 support on a port or bond to enroll in this behavior. This is represented by the layer2-bonded, layer2-individual, hybrid network modes available in the Terraform provider with similar names in the EM console UI.

It is now possible to add VLANs to a device without pre-enrolling the device in Layer2 modes, but this new functionality is limited to devices in a subset of facilities (those with the IBX feature flag). We'll have to actively enable layer2 features or detect when this step can be skipped.

It is worth noting that the network modes reported in the UI and represented in the Terraform provider are not settings that can be toggled in the EM API. Network mode is a product of port bondedness, layer-2 or layer-3 state, the presence of management addresses, and the presence of VLANs.

After a few different iterations on how to represent these features in Terraform, we are coming around to the idea of giving the user control over the ports directly without offering network mode hand-waving.

Expressed statefully, and adhering closely to the EM API spec, this could look like this:

- plan: "n2.x-large"
  ports: # network_ports in the API naming
    bond0: # always comprised of ports eth0+eth1 on 2 port devices, or eth0+eth2 on 4 port devices
      bonded: true # default
      layer2: false # this is default on new devices. changes result in /ports/id/convert/layer-[2|3] API calls
      vlans: [1234, 5678, 1234] # shortened this from the API "virtual_networks"
      native_vlan: 5678 # optional, must be one of the above if set
      ip_addresses:
      - reservation_id: uuid # reserved addresses by uuid, may include global ips
      - cidr_notation: "1.2.3.4/24" # reserved addresses by address, may include global ips
      - type: private_ipv4 # dynamically assigned addresses, available via metadata.platformequinix.net
        cidr: 30 
    bond1: # comprised of port eth1+eth3 on 4 port devices
      bonded: false
    eth1: # unbonded eth ports can use most of the same attributes bond ports can use
      layer2: true
      vlans: [7654]
    eth3:
      layer2: true
      vlans: [9876]

Users are free to customize the ports, bonds, addresses, and VLANs with all the flexibility afforded by the API. Invalid configurations will bubble up through API errors and input does not have to be validated by the CAPP provider. It is difficult to validate these values because the plan, facility, state of the port, state of the bonds, and other factors affect the success of toggling a field. Through Kubernetes reconciliation loops, eventually, all bonds will be connected, disconnected, or the device will reach the desired layer-2/layer-3 mode. The desired addresses will be provisioned and assigned.

A port and bond configuration not supported today, maybe supported later or may be supported in a different plan or facility setting.

This was discussed for consideration in the Terraform provider within a packngo PR: equinixmetal-archive/packngo#239 (comment)

I don't know how well CAPP can capture this approach. CAPP would need to know which address ranges to use for what purposes, dynamically allocated ranges like the project management range could be referred to generically without knowing the precise range. In other cases, the precise range may be known in advance (IP reservations). In VLAN cases, the nodes would need special userdata to configure the network and CAPP would need to be told what ranges to manage.

Userdata scripts would need to be hearty, allowing time for the ports to be bonded, VLANs to be attached, and addresses to become available.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[detiber]

/lifecycle frozen

[davidspek]

I think this shouldn't be limited to just the control plane, but rather all nodes should only have private IPs and any service that needs to be exposed should be done through a load balancer. This is similar to how EKS, GKE and AKS do things.

[displague]

CPEM added support for clusters with node IP addresses drawn from a VLAN accessible range in v3.5.0.

CAPP should take advantage of this for clusters that take advantage of Equinix Metal Gateway and VRF features.

[Lirt]

Hi,

I implemented change in CAPP for the servers to not use public IPs (I can open PR for that). But the blocker on this feature is that servers without public IP address don't have Internet connectivity and there is no NAT Gateway service or similar service that would enable servers to reach Internet.

This can be feasible for use-cases where Internet connectivity is not needed and all dependencies are served from internal network, but overall for most of the standard use-cases the Internet needs to be reachable (at least for kubelet to download container images).

Correct me if I'm wrong but I don't see how Metal Gateway enables servers to not have public IPs - as we can see in the Metal Gateway example the servers need to be configured with IP address on OS level (which is even less automatable).

One possible way to overcome this is to have a server that serves as NAT Gateway using standard Linux with enabled ip_forward and iptables, but that's configuration overhead with single point of failure and increases costs by 1 server.

Is there a plan to have NAT service or something similar to make this easier? Or can you provide future plans around this feature?

If I wrote anything wrong, please correct me and provide me more information.