Extension Mechanisms: Revisit guidelines about Policy Attachment overlapping with Filters

[sunjayBhatia]

The Policy Attachment reference section on interaction with custom Route Filters includes:

2. Custom filters and policies should not overlap

Although it's possible that arbitrary fields could be supported by custom policy, custom route filters, and core/extended fields
concurrently, it is strongly recommended that implementations not use multiple mechanisms for representing the same fields. A
given field should only be supported through a single extension method. An example of potential conflict is policy precedence and
structured hierarchy, which only applies to custom policies. Allowing a field to exist in custom policies and also other areas of the
API, which are not part of the structured hierarchy, breaks the precedence model. Note that this guidance may change in the future
as we gain a better understanding of how extension mechanisms of the Gateway API can interoperate.

However, it might be quite useful to be able to configure a Policy to set hierarchical configuration that override or are overridden by a per-Route or per-Backend Filter. For example, using existing built-in Filters, it seems quite useful to be able to set a HTTP header policy on a Gateway so all Routes attached to it configure a standard set of headers, but can also override headers per-Route or per-Backend. Contour implements this with a set of HTTP header policies configured at startup (outside of Gateway API mechanisms) that can be overridden by per-Route Filters. We can likely come up with other examples for other custom configurations not covered by the built-in Filters (for rate limiting, auth, etc.).

Of course, if Policies can override Filters, this does again create another complication for observability. The process for a user figuring out what configuration is actually in-effect on a resource/Route/etc. becomes more convoluted. Some version of Status reporting what configuration is in effect or other mechanism should be involved if these resources can interact.

If PolicyTargetReference is expanded to include a section name to target subsections of a resource, this may mean this discussion is moot and that custom Filters are not super useful.

Would be great to get people's thoughts on this!

Related to #1489 (and maybe can be folded into that discussion).

[sunjayBhatia]

cc @youngnick since you mentioned this was something you wanted to include in your work on Policy Attachment

[youngnick]

If PolicyTargetReference is expanded to include a section name to target subsections of a resource, this may mean this discussion is moot and that custom Filters are not super useful.

I think that we probably need to talk more about the user experience implications of (basically) deprioritizing custom Filters. The big advantage of custom Filters is that they are inline in the HTTPRoute object, and so they are pretty clearly associated with it, so the status thing applies less for them on an individual basis.

However, if you allow custom filters that are then modified or disabled by Policy objects, then that is a nightmare for UX, because hidden side-effects everywhere.

I'm still thinking more about this.