Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

design: Contour utilizing Gateway API Policy Attachment and Route Filters #4749

Closed
wants to merge 4 commits into from
Closed

design: Contour utilizing Gateway API Policy Attachment and Route Filters #4749

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ba14539
WIP
sunjayBhatia Sep 23, 2022
81cd90f
updates around filter spike and notes around backendref
sunjayBhatia Oct 5, 2022
cef6fcc
Add notes about policy spike
sunjayBhatia Oct 6, 2022
68f916a
some more clarification on policies
sunjayBhatia Oct 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 126 additions & 0 deletions design/gw-api-policy-and-filter-design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Implementing advanced features in Gateway API with Policy Attachment and Route Filters

Status: Draft

## Abstract
This design describes how Contour may implement features in Gateway API outside the "core" or "extended" features using the well defined extension points in the API: Policy attachment, route level filters, and GatewayClass parameters.
We will also attempt to categorize existing and new Contour features to specify which extension mechanism should be used to express them in Contour's implementation of Gateway API.

## Background
Contour is up to date with Gateway API conformance and will continue to be, however there are many features Contour supports that Gateway API does not.
Even as the Gateway API spec moves to GA, not everything Contour implements will be available, since the Gateway API is a more general API, with core and extended functionality intended to be supported by many implementations.
For example, things needed for access control, authorization, "day two operations", observability, and more fine grained control over HTTP middleware are still needed and not yet expressed in Gateway API.
We've outlined the feature parity gap between Contour's HTTPProxy and the existing core and extended Gateway API [here](https://docs.google.com/document/d/1F2o7R10H5ZDG9rW3m1K2BVkl-SwhTp-EShylMW1wlBQ/).

Gateway API does offer a few extension points intended to support many of the features Contour and other ingress controllers provide:
- [Policy Attachment](https://gateway-api.sigs.k8s.io/references/policy-attachment/)
- [HTTPRoute filters](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPRouteFilter)
- [GatewayClass parameters](https://gateway-api.sigs.k8s.io/api-types/gatewayclass/#gatewayclass-parameters)

These configuration points are intended for different classes of configuration.
We will mainly focus on the first two above, though the third is worth mentioning as an option.
The Contour Gateway Provisioner uses GatewayClass parameters for passing general configuration for an instance of Contour.

Policy attachment is described in detail [here](https://gateway-api.sigs.k8s.io/references/policy-attachment/).
Policy resources must follow a standard structure, referencing the resource they apply to via a [`targetRef`](https://gateway-api.sigs.k8s.io/references/policy-attachment/#target-reference-api) field.
They could be "attached" to a Gateway, Route, Service (or other backend reference resource), or even a Namespace (howeer the resources a particular policy is allowed to attach to is implementation-dependent).
A key feature of Policy attachment is the idea that Policies can be applied with a hierarchy.
[Overrides and Defaults can be cascaded up/down the hierarchy of Gateway API resources](https://gateway-api.sigs.k8s.io/references/policy-attachment/#hierarchy).

(Note that Policies could be attached to GatewayClasses, however it is [more complicated and probably not recommended](https://gateway-api.sigs.k8s.io/references/policy-attachment/#attaching-policy-to-gatewayclass))

HTTPRoute also supports a per-route or per-backend [`HTTPRouteFilter`](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPRouteFilter) type.
This provides built-in HTTP request modification/redirection capabilities (varying between core and extended conformance levels).
It also supports an `ExtensionRef` filter type, which can be to reference a custom resource for implementation-specific configuration.

Policies and Filters represent extension points that could overlap in scope.
Upstream Gateway API documentation provides some guidance [here](https://gateway-api.sigs.k8s.io/references/policy-attachment/#interaction-with-custom-route-filters).
In particular the community recommends [differentiating what configurations are supported by each method](https://gateway-api.sigs.k8s.io/references/policy-attachment/#2-custom-filters-and-policies-should-not-overlap)

## Goals
- Differentiating which Contour features should be included in each configuration/extension type
- Give maintainers/contributors/users an agreed upon framework/pattern in which to think about extension points in Gateway API so we can make the implementation and API consistent
- Use abstractions/conventions of Gateway API as much as possible to avoid possible future of piecemeal attaching new features/configurations

## Non Goals
- Cover configuration of upstream TLS or other "backend protocol details"
- These will hopefully be covered by [this GEP](https://gateway-api.sigs.k8s.io/geps/gep-1282/) and subsequent features
- Contour flags we've added for smaller surface area coverage or convention etc.
- e.g. `contourconfiguration.spec.envoy.listener.disableAllowChunkedLength`

## High-Level Design
Summary TBD

## Detailed Design

### Categorizing features between Policies or Filters
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some useful quotes from the spec:

Filters define processing steps that must be completed during the request or response lifecycle. Filters act as an extension point to express additional processing that may be performed in Gateway implementations. Some examples include request or response modification, implementing authentication strategies, rate-limiting, and traffic shaping.

(ref. https://gateway-api.sigs.k8s.io/api-types/httproute/#filters-optional)

Custom Route filters provide a way to configure request/response modifiers or middleware embedded inside Route rules or backend references.

Policy attachment is more broad in scope. In contrast with filters, policies can be attached to a wide variety of Gateway API resources, and include a concept of hierarchical defaulting and overrides. Although Policy attachment can be used to target an entire Route or Backend, it cannot currently be used to target specific Route rules or backend references. If there are sufficient use cases for this, policy attachment may be expanded in the future to support this fine grained targeting.

(ref. https://gateway-api.sigs.k8s.io/references/policy-attachment/#interaction-with-custom-route-filters)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even just looking at the built-in filters -- e.g. the request header modifier -- in Contour/HTTPProxy, we support both global default header modifications, and per-route modifications. So to implement that in Gateway API, you'd need to use policy attachment, except there's also the built-in filter, and having both contradicts the guidance in https://gateway-api.sigs.k8s.io/references/policy-attachment/#2-custom-filters-and-policies-should-not-overlap. 🤔

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, that and what you mention with rate limiting + auth are what im struggling with a little, though there is a guideline about separating concerns, it doesnt seem like it is really that cut and dry with these examples

maybe something to bring up upstream

Here is a list of features we want to configure likely outside scope of the Gateway API standardized features that are already in Contour:
- rate limiting
- auth (external auth and/or built-in Envoy filters)
Comment on lines +64 to +65
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rate limiting and auth are interesting -- although they're specifically called out in the description for filters, and I can easily see them being implemented as filters, they feel like things where we'd want to be able to apply defaults at the Gateway/Listener level, and also be able to override at the route level. I guess that pushes them towards Policy Attachment.

- load balancing between endpoints of a service
- CORS configuration
- cookie rewriting
- direct HTTP response configuration (HTTPProxy directResponsePolicy, could also be contributed to GW API upstream?)

Things not in Contour yet:
- tracing
- IP allow/block lists
- etc.

We can categorize these a few ways, but to differentiate what should be configured in a Gateway API Policy vs. Filter, I propose that we consider two categories:

#### Features that modify content in the data path
sunjayBhatia marked this conversation as resolved.
Show resolved Hide resolved
These features should be considered ones that we use HTTPRoute Filters to implement.
Similar to the existing Filter types that modify the data path, we can use custom Filters to implement things like:
- cookie reqriting
- direct HTTP responses
- etc.

Note: I think we should not consider these "Filters" equivalent to Envoy filters in the general sense, as Envoy filters are often associated with a single virtualhost, and these are route-level Filters.

#### Features that aid in operability, traffic distribution, authorization, access control
The above title is not an exhaustive list, but rather a bit more of a set of examples that help illustrate the category.
THey include configuration that needs to be applied to accomodate a specific workload, e.g. a timeout that needs to be applied to a particular deployment/service.
These types of features will be implemented via Policy resources that target specific Gateway API or Service resources.

Some examples and discussion of how they may work are below:
- TBD

### Status
While it is not explicitly statued in the Gateway API documentation, Contour should follow the conventions in setting Status Conditions being established by Gateway API.
We should utilize an `Attached` or `Accepted` Condition that signifies whether a Policy or Filter resource has been applied.
This encompasses validity, whether it is applied to an allowed resource type, and whether it is an allowed cross-namespace reference.
More detailed error conditions may be added as well to point out specific errors.

## Alternatives Considered
TBD

## Security Considerations

### Cross-namespace references
It is not specified explicitly in the Gateway API documentation, but if Policy or Filter custom resources live in a separate namespace from the resource it references/is referenced by, we will likely need to ensure the appropriate ReferenceGrant is present that allows the resources to be "attached."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExtensionRef is a LocalObjectRef so the ref'd resource should be in the same ns.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yep, good catch, should be corrected to match the Rate Limiting: Filter section which mentions this


## Compatibility
TBD

## Implementation

### Reconciling Policy resources
While the Policy resource `targetRef` allows referencing any resource, we should be judicious about what resources a Policy can attach to, especially initially.
This is to ensure clarity for implementers and more importantly users as to which Policy is in effect.
This should allow us to incrementally grow our implementation and API surface area.
For example, rather than initally allow a Policy to reference *any* resource, we may restrict it to reference only a Gateway or only an HTTPProxy.

Contour will add informers for each Policy resource we end up creating, collect and caching them as we do with other resources.
Typically we cache resources by namespace and name of the resource.
As an optimization we may consider caching Policy resources by the namespace, name, and kind of the resource they reference instead.
This may aid in lookup of applicable Policies when processing Gateways, HTTPRoutes, and Services, and make it easier to construct the final values we must apply for some configuration setting as the number of Policies in a cluster grows.

### Reconciling Filter resources
Reconciling Filters should be simpler than Policies, as they can only be referenced in route rules or as part of a backend configuration.
Contour will add informers for each Filter resource we end up creating, collect and caching them as we do with other resources.
When processing HTTPRoutes, any referenced Filter resources will be looked up in the cache.
Status should be set accordingly on the referencing HTTPRoute and Filter if invalid.

## Open Issues
N/A for now