diff --git a/geps/gep-1651.md b/geps/gep-1651.md new file mode 100644 index 0000000000..87c85bb5f9 --- /dev/null +++ b/geps/gep-1651.md @@ -0,0 +1,411 @@ +# GEP-1651: Gateway Routability + +* Issue: [#1651](https://github.com/kubernetes-sigs/gateway-api/issues/1651) +* Status: Provisional + +(See status definitions [here](overview.md#status).) + +## TLDR + +Allow users to configure a Gateway so that it is only routable within +a specific scope (ie. public/private/cluster) + +## Goals + +- Define a mechanic to set the routability on a Gateway +- Provide a default set of routability options +- Provide a way for vendors to support custom options + +## Non-Goals + +- Per-request/route scope +- Not a lightweight service mesh + +## Introduction + +One of the early feature requests for Knative was the ability to deploy an +application using Knative's HTTP routing support, but make it only available +within the cluster. I want to be able to specify both the "cluster" +(service.namespace.svc) and "external" (service.namespace.example.com). +Gateways using the same GatewayClass on the cluster, but ensure that the +"cluster" service is only routable within the cluster. This would greatly +simplify deployment for users over the instructions we have today. + +Likewise another use case is to provide load balancing capabilities within a virtual +private network. Different IaaS providers offer private load balancers to support +these use cases. + +## API + +We propose adding a new `routability` field under the `spec.infrastructure` stanza of a Gateway. + +### Predefined Routability Values + +Implementations MAY implement the following values for 'routability' and MUST abide by +their defined semantics. + +Value | Scope +-|- +`Public`|The address is routable on the public internet +`Private`|The address is routable inside a private network larger than a single cluster (ie. VPC) and MAY include RFC1918 address space +`Cluster`|The address is routable inside the [cluster's network](https://kubernetes.io/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-network-model) + +Values can be compared semantically - `Public` has a larger scope than `Private`, while `Private` has a larger scope than `Cluster`. + +### Vendor prefixed values + +Implementations can define custom 'routability' values by specifying a vendor prefix followed +by a slash `/` and a custom name ie. `com.example.com/my-routability`. + +Comparing vendor prefixed scopes with the pre-defined ones in implementation specific. + +### Default Routability + +The default value of `routability` is implementation specific. It is RECOMMENDED that +the default `routability` remains consistent for Gateways with the same +`gatewayClassName`. + +Implementations MAY leave the `spec.infrastructure.routability` property unset and instead +signal the default routability using the Gateway's `status.addresses`. See 'Status Addresses` +for more details. + +### Mutability + +Implementations MAY prevent end-users from updating the `routability` value of a Gateway. If +updates are allowed the semantics and behaviour will depend on the underlying implementation. + +If a Gateway is mutated but does not support the desired routability it MUST set the conditions +`Accepted`, `Programmed` to `False` with `Reason` set to `UnsupportedRoutability`. Implementations +MAY choose to leave the old Gateway running with the previous generation's configuration. + +### Go + +```go + +// GatewayRoutablility represents the routability of a Gateway +// +// The pre-defined values listed in this package can be compared semantically. +// [GatewayRoutabilityPublic] has a larger scope than [GatewayRoutabilityPrivate], +// while [GatewayRoutabilityPrivate] has a larger scope than +// [GatewayRoutabilityCluster]. +// +// Implementations can define custom routability values by specifying a vendor +// prefix followed by a slash '/' and a custom name ie. `dev.example.com/my-routability`. +type GatewayRoutability string + +const ( + // GatewayRoutabilityPublic means the Gateway's address MUST + // be routable on the public internet + // + // Implementations MAY support this routability + GatewayRoutabilityPublic GatewayRoutability = "Public" + + // GatewayRoutabilityPrivate means the Gateway's address MUST + // be routable inside a private network larger than a single + // cluster (ie. VPC) and MAY include the RFC1918 address space + // + // Implementations MAY support this routability + GatewayRoutabilityPrivate GatewayRoutability = "Private" + + // GatewayRoutabilityCluster means the Gateway's address MUST + // be only be routable inside the [cluster's network] + // + // Implementations MAY support this routability + // + // [cluster's network](https://kubernetes.io/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-network-model) + GatewayRoutabilityCluster GatewayRoutability = "Cluster" +) + +type GatewaySpec struct { + // Infrastructure defines infrastructure level attributes about this Gateway instance. + Infrastructure GatewayInfrastructure `json:"infrastructure"` + // ... +} +type GatewayInfrastructure struct { + // Routability allows the Gateway to specify the accessibility of it's addresses. Setting + // this property will override the default value defined by the [GatewayClass] + // + // If the desired Gateway routability is incompatible with the [GatewayClass] implementations + // MUST set the condition `Accepted` to `False` with `Reason` set to `UnsupportedRoutability`. + + // The default value of routability is implementation specific. + // It is RECOMMENDED that the default routability remains consistent for Gateways with the same + // gatewayClassName + // + // Implementations MAY leave this property unset and signal the default + // routability in the [GatewayStatus] + // + // Implementations MAY prevent end-users from updating the routability value of a Gateway. + // If updates are allowed the semantics and behaviour will depend on the underlying implementation. + // If a Gateway is mutated but does not support the desired routability it MUST set the conditions + // `Accepted`, `Programmed` to `False` with `Reason` set to `UnsupportedRoutability`. + // + // It is RECOMMENDED that in-cluster gateways SHOULD NOT support 'Private' routability. + // Kubernetes doesn't have a concept of 'Private' routability for Services. In the future this may + // change upstream. + // + // +optional + Routability *GatewayRoutability `json:"routability,omitempty"` +} + +type GatewayStatus struct { + // Addresses lists the IP addresses that have actually been + // bound to the Gateway. These addresses may differ from the + // addresses in the Spec, e.g. if the Gateway automatically + // assigns an address from a reserved pool. + // + // Implementations that support Gateway routability MUST include an address + // that has the same routable semantics as defined in the Gateway spec. + // + // Implementations MAY add additional addresses in status, but they MUST be + // semantically less than the scope of the requested scope. For example if a + // user requests a `Cluster` routable Gateway then the list of addresses + // MUST NOT have a routability of `Public` or `Private`. + // + // +optional + // +kubebuilder:validation:MaxItems=16 + Addresses []GatewayStatusAddress `json:"addresses,omitempty"` + // ... +} + +type GatewayStatusAddress struct { + // Routability specifies the routable bounds of this address + // Predefined values are: 'Private', 'Public', Cluster + // Other values MUST have a vendor prefix. + // + // Implementations that support Routability MUST populate this + // field + // + // +optional + Routability *GatewayRoutability `json:"routability,omitempty"` + + // ... +} + +type GatewayClassStatus struct { + // Routabilities specifies a list of supported routabilities offered by + // the GatewayClass. The first entry in this list will be the default + // routability used when Gateways of this class are created. + // + // Implemenations MAY provide a pre-defined set of GatewayClasses that + // limit the routability choices of a Gateway. + // + // Implementations that support routability MUST populate this list with + // a subset of the pre-defined [GatewayRoutability] values or vendored + // prefix values. + // + // +optional + Routabilities []GatewayRoutability `json:"routabilities"` +} +``` + +### YAML +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: prod-web +spec: + gatewayClassName: acme-lb + infrastructure: + routability: Public + listeners: + - protocol: HTTP + port: 80 +``` + +### Semantics + +#### Interaction with GatewayClass + +An infrastructure provider MAY provide a pre-defined set of GatewayClasses that limit the +routability choices of a Gateway. If the desired Gateway routability is incompatible with the +GatewayClass it MUST set the condition `Accepted` to `False` with `Reason` set to `UnsupportedRoutability`. + +If an implementation supports 'routability' then the GatewayClass MUST list the supported +routabilities in the status stanza. The `status.routabilities` MUST contain either +a subset of the pre-defined values mentioned above or contain vendored prefixed values. + +The first value in the list will be used as the default value when Gateways of this class +are created. This can be overriden by setting the Gateway's `spec.infrastructure.routability`. + +#### Unsupported routability & address values + +If a Gateway is unable to provide an address for the desired routability it MUST set the condition `Accepted` +to `False` with `Reason` set to `UnsupportedRoutability` + +#### Status.Addresses + +If a Gateway supports the desired 'routability' implementations MUST populate the `status.addresses` with +an address that has the same routable semantics. The GatewayAddress field `routability` MUST be populated. + +Implementations MAY add additional addresses in status, but they MUST be semantically less than the scope +of the requested scope. For example if a user requests a `Cluster` routable Gateway then the list of addresses +MUST NOT have a routability of `Public` or `Private`. + +We plan on introducing a new type `GatewayStatusAddress` and change Gateway's `status.addresses` to be +`[]GatewayStatusAddress`. This will allow the status address type to evolve separately from the spec address. + +#### In-cluster Gateways and 'Private' Routability + +It is RECOMMENDED that in-cluster gateways SHOULD NOT support 'Private' routability. Kubernetes doesn't have +a concept of 'Private' routability for Services. In the future this may change upstream. + +## Examples + +#### 1. Request a GatewayAddress that is routable within the same cluster + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: prod-web +spec: + gatewayClassName: acme-lb + infrastructure: + routability: Cluster + listeners: + - protocol: HTTP + port: 80 +``` + +#### 2. Request a GatewayAddress with a specific routability and address +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: prod-web +spec: + gatewayClassName: acme-lb + infrastructure: + routability: Cluster + listeners: + - protocol: HTTP + port: 80 + addresses: + - value: 10.0.0.8 +``` + +#### 3. Request a GatewayAddress that is routable on the public internet +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: prod-web +spec: + gatewayClassName: acme-lb + infrastructure: + routability: Public + listeners: + - protocol: HTTP + port: 80 +``` + +#### 4. Request a GatewayAddress that is a cloud provider's VPC +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: prod-web +spec: + gatewayClassName: acme-lb + infrastructure: + routability: Private + listeners: + - protocol: HTTP + port: 80 +``` + +## Alternatives + +### Introducing new GatewayAddress Types + +We could introduce additional `AddressTypes` (ie. `ClusterLocalIPAddress`) but +this would lead to a combinatorial explosion as new dimensions (ie. IPv6) are +introduced. + +From: [https://github.com/kubernetes-sigs/gateway-api/pull/1653#issuecomment-1451246877](https://github.com/kubernetes-sigs/gateway-api/pull/1653#issuecomment-1451246877) + +> Although this makes sense in isolation, I'm worried about the long term impacts this could have. In my opinion, ClusterLocal is a modifier, not exactly an address type. For example, it's possible in the future that we'll have a way to provision cluster-local DNS names, we may want to use the same kind of mechanism to request a ClusterLocal DNS name for the Gateway. +> +> It's also possible that users will want to explicitly request an IP Families (v4, v6, or both). I'd really hate to get into a situation where we have the following options: +> +> - IPAddress +> - IPv4Address +> - IPv6Address +> - ClusterLocalIPAddress +> - ClusterLocalIPv4Address +> - ClusterLocalIPv6Address +> +> For each dimension we avoid adding a separate field for and instead try to embed into a single name, we risk this kind of name explosion. Of course none of the above even begins to cover my idea of NetworkLocal which could further complicate this. + +### Scope/reachability/routability field on GatewayAddress + +This would allow Gateways to have multiple scopes. + +From: [https://github.com/kubernetes-sigs/gateway-api/pull/1653#issuecomment-1486271913](https://github.com/kubernetes-sigs/gateway-api/pull/1653#issuecomment-1486271913) +> The obvious application for multiple scopes seems to be saving on boilerplate, which is a win, but are there are any other advantages to allowing one Gateway to have multiple scopes? +> +> Multiple scopes Pros: +> +> Allows a single Gateway to express multiple networks, saving on needing to attach HTTPRoutes to multiple Gateways for each network scope. +> +> Multiple scopes Cons: +> +> Complicates the Gateway's purpose. Instead of one Gateway being one set of Listeners, now a Gateway is two sets of listeners that have a totally different scope (and presumably, security context). Personally, I'm also concerned how this will interact with other features like merging and preprovisioning that GEP-1867: Per-Gateway Infrastructure #1868 will allow. + +### Adding `routability` attribute to GatewayClass + +See [Prior Art - Multiple Gateways Classes](#multiple-gateway-classes) + +## Survey of Prior Art + +These alternatives are a survey of existing approaches to support cluster +local Gateways. Most are implementation specific and are not portable. + +### Special annotation/label + +Istio let's you specify an annotation `networking.istio.io/service-type` to +change the underlying Kubernetes Service type to make it a ClusterIP type. + +### Re-use of AddressType Hostname + +Istio let's you re-use existing Gateway deployments by setting the address +type to `Hostname` and the value to the Istio ingress Kubernetes Service. If an +operator configures the Istio deployment to support cluster local traffic a +Gateway implementation can select it using the `HostName` attribute. + +### Multiple Gateway Classes + +Some implementations support multiple deployments on a single cluster where each maps to a +GatewayClass. One of these deployments can be configured to serve cluster local traffic. This is +sub-optimal because this is implementation specific and the end-user is effectively managing the +deployments themselves rather than infrastructure being automatically provisioned. + +Likewise, infrastructure providers may provide a fixed set of GatewayClasses with unique and fixed +routability. Thus GatewayClass name is a viable option to control routability. There may be a +non-zero cost when requiring additional GatewayClasses - but this depends on the implementation. + +Additionally, if more attributes are added to GatewayClass to constrain Gateways in some +form this leads to a combinatorial number of GatewayClassNames. For example, `foo-public` and +`foo-cluster` are two GatewayClasses surfacing the values of a single attribute `routability`. +Let's say we want to enforce address types to just IP then our `gatewayClassName` would be: + +- `foo-public-ipv4` +- `foo-public-ipv6` +- `foo-cluster-ipv4` +- `foo-cluster-ipv6` + +This may not be as flexible for end-users compared to configuring `routability` when creating +a Gateway. + +As howardjohn mentioned [here](https://github.com/kubernetes-sigs/gateway-api/pull/1653#issuecomment-1429992160): +> having the ability to configure things at a higher level seems nice for Gateway, but being able to configure them on a per-Gateway basis remains important. + +## References + +- [Knative - Private Services](https://knative.dev/docs/serving/services/private-services/#configuring-private-services) +- [Initial Gateway GitHub Discussion](https://github.com/kubernetes-sigs/gateway-api/discussions/1247) +- [Istio Support for Private Gateways](https://istio.io/latest/docs/tasks/traffic-management/ingress/gateway-api/#automated-deployment) +- [Envoy Gateway Support for Private Gateways](https://gateway.envoyproxy.io/latest/api/config_types.html#kubernetesservicespec) + diff --git a/mkdocs.yml b/mkdocs.yml index dadbf44682..6721d06cff 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -88,6 +88,7 @@ nav: - geps/gep-1426.md - geps/gep-1324.md - geps/gep-1619.md + - geps/gep-1651.md - geps/gep-1897.md - geps/gep-1867.md - Prototyping: