Policy Attachment: Current PolicyTargetReference API limitations

[sunjayBhatia]

The PolicyTargetReference API currently lets us specify a whole resource to target with configuration in a custom Policy resource.

Using this for example to set a rate limit policy, we can apply Policies to different resources e.g. Gateways and HTTPRoutes to allow an operator and app owner to set defaults/overrides as they see fit. This example spike in Contour shows this with some example configuration YAMLs: projectcontour/contour#4776

However, it still feels like the PolicyTargetReference API is limited in a few scenarios that cause users to have to split configuration between multiple resources. This can in some cases just cause clutter or extra places for users to look for configuration updates/status, but in others actually mean the API doesn't work as intended.

Applying a Policy to an individual route rule
Since PolicyTargetReference only allows us to currently apply a Policy to whole resource we have to split route rules between different HTTPRoutes to target an individual route. The YAML provided in the Contour spike shows this. The /api/needs-ratelimit path match rule must be separated into its own HTTPRoute with a policy targeting only that resource. While the API is flexible enough to allow us to achieve the intended goal, it comes at the cost of duplication of the form (as I understand it) Gateway API is intended to do away with.

Applying a Policy to a whole virtualhost
Say we want to apply a Policy to foo-prod.com and a different Policy to foo-dev.com. This could be an authorization Policy, rate limit Policy, etc. This could be done in a few ways, depending on how your organization sets up configuration:

1. Gateway with multiple Listeners, one for foo-prod.com and one for foo-dev.com, HTTPRoutes can have a ParentReference to the whole Gateway or specific Listeners and don't necessarily need to specify a hostname.
2. Gateway Listeners in this case don't matter, could have a "general" (no hostname or wildcard possibly) Listener that HTTPRoutes for both virtualhosts attach to or specific Listeners. All route rules under foo-prod.com belong to HTTPRoute(s) that have that hostname configured, and the same with foo-dev.com.

For the first option, to apply separate Policies to foo-prod.com and foo-dev.com at the Gateway level, we would need to separate the Listeners into different Gateway resources. This has implications for many implementations, since the idea of merging Gateways is not super well defined. For example Contour does not merge Gateways at the moment and each Gateway requires a separate Contour controller and data-plane.

For the second option, we could either:

• Apply Policies directly to individual HTTPRoutes. If there are multiple for one virtualhost can become unwieldy and also means we sort of lose the Gateway owner persona override/defaulting control.
• Organize HTTPRoutes for individual virtualhosts in different namespaces and apply Policies to the Namespaces.

Solutions
With the ability to apply policies to sections of a resource, these limitations would appear to be mitigated. We would be able to target sections of a Gateway or Route to apply a Policy to. This would mean we don't restrict how organizations have to organize their resources as we would offer maximum flexibility.

Discussion questions/topics

• Do the above examples make sense?
• Are there other similar workflows we can provide as examples?
• Are there other ways of using Policy Attachment that allow us to achieve the intended goals above?

[shaneutt]

Do the above examples make sense? Are there other similar workflows we can provide as examples?

I personally think the point is well made. There wasn't any end-user feedback that played into this desire for match targets, was there?

Are there other ways of using Policy Attachment that allow us to achieve the intended goals above?

It would seem to me the main one that you already touched on is to make separate routes, as this helps with the downsides of the SectionName approach pointed out in the GEP.

---

I think one of the things I worry most about policy attachment is how opaque it can feel from the end-user perspective, and yet how incredibly relevant it can be to their experience. At least with whole object attachment, it can be fairly simple to understand where a policy applies. With SectionName attachment, it becomes a little bit harder. The difficulty comes from what a user has to do to understand the policies affecting their routes.

The way we're heading right now, a single HTTPRoute could have a number of policies from all different directions, and yet looking at the HTTPRoute itself it would be really difficult to know that without compiling all the policies and looking for those attachments, THEN visually inspecting the precedence for overrides and defaults.

For instance, if you have a policy which provides fields A, B then attach it to a Gateway with a override for B, and then attach two other instance of that policy directly to the HTTPRoute, one with a SectionName targeting a specific match and providing an A and a B value, and the other without a SectionName and only providing a B, as an end-user it becomes a real mess to understand how these policies are actually affecting your route, and that's only worsened by the fact that you now have to understand which individual sections those policies are affecting, not to mention remembering you have to account for defaults and overrides and we need to add new precedence rules for SectionName vs non-SectionName applications 😵.

There's a growing part of me that really wants to see the "effective policy" for any *Route be compiled by controllers and placed right in the .status of the resource, letting the computer do all that reference work and the user can just look at see "these are the effective policies applying to my route, and to specific sections of the route". I would be curious to hear your thoughts on that as it pertains to the complexity of adding more fine-grained policy targeting mechanisms.

(by the way, this is a really great topic thanks for bringing it up)

[sunjayBhatia]

There's a growing part of me that really wants to see the "effective policy" for any *Route be compiled by controllers and placed right in the .status of the resource, letting the computer do all that reference work and the user can just look at see "these are the effective policies applying to my route, and to specific sections of the route". I would be curious to hear your thoughts on that as it pertains to the complexity of adding more fine-grained policy targeting mechanisms.

yeah, been thinking about how to make things observable, which is worth talking about here, since adding the ability to target subsections of a resource adds more complexity to an already complex idea in general

I agree something in status of the resources involved would be nice to see, some discussion of that is in this doc however and seems from the GEP process this was not desirable? https://gateway-api.sigs.k8s.io/references/policy-attachment/#status

The kubectl plugin idea also is document here too: https://gateway-api.sigs.k8s.io/references/policy-attachment/#kubectl-plugin

We could also conceivably have some sort of UI component like a plugin for something like Octant or other technology

I can spend some time looking at some of these ideas too to see how it might look in practice, I'm not sure anyone has implemented either yet

From what I've seen, the only possibly live implementation is in Linkerd, and I haven't seen any sort of summary view in their implementation: https://linkerd.io/2.12/reference/authorization-policy/#authorizationpolicy

[sunjayBhatia]

There wasn't any end-user feedback that played into this desire for match targets, was there?

Nope not from us at Contour, we're still in the design process for creating project guidelines on Policy CRD creation

[maleck13]

Just to add another reference, we have a small project called Kuadrant, that is attempting to model rate limiting and auth as policy attachment resources using our own Auth and Rate Limiting services. Quite early on we found there was difficulty with policy attachment and HTTPRoute in the way described above (wanting to target a specific rule).
You can see how we are iterating and discussing some of these problems here:
Kuadrant/architecture#8 (currently in order to target a rule, we ask the end user to duplicate that rule in our policy).
We have also identified that it is vital for anyone using policy attachment to be able to provide a way to visualise the "effective policy".
Finally, we have not yet come up with an API model that allows us to confidently add in the default and override sections of the spec although we feel we are starting to get closer.