decision: Segmenting API Gateway features into PolicyAttachment CRDs

[arkodg]

Description:

Adding options for how API Gateway features such as ratelimiting can be expressed as PolicyAttachment CRDs. Please dont focus on the names, just added placeholders.

1. one generic policy CRD with features as top level fields

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: Policy
 .....
 spec:
   authentication:
   .....
   ratelimiting:
   ....

2. one generic policy CRD with networking (shapes traffic) and security (authenticates & authorizes traffic) as top fields

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: Policy
 .....
 spec:
   security:
     authentication:
     ....
   networking:
     ratelimiting:
     .....

3. 2 Policy CRDs - one for networking and the other for security

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: SecurityPolicy
 ....
 ....
 spec:
   authentication:

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: NetworkingPolicy
 ....
 spec:
   ratelimiting:
   ....

4. 1 CRD per API Gateway feature

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: Authentication
 ....
 spec:
   jwt:
   ....

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: Ratelimiting
 ....
 spec:
   ....

[arkodg]

@Xunzhuo @danehans @youngnick @skriss @LukeShu @AliceProxy can you please vote on this issue

[Xunzhuo]

I prefer 3 or 4, they have more independence and loose coupling between features.

[arkodg]

I vote for 2. or 3. and here's why

• 1 or 2 CRDs are easier to maintain for the project than multiple
• The user can express their intent in 1 instantiated resource (useful when team is small) or multiple ( useful when multiple teams author intent)

[Alice-Lilith]

I think 3 would be preferable over 1 and 2 although I prefer 4 the most. Allows for clear separation of resources depending on what the user is trying to accomplish. Prioritize keeping things self contained. While you could group two features into a single group like networking / security I feel that when a user has the goal of setting up a ratelimt they shouldn't need to worry about what other resources exist and then check to see if they have ratelimits configured on them and need to be edited vs. extended versus creating a new resource.

IE: I'm a cluster admin and I know that I want to setup authentication, I shouldn't have to use kubectl for some other resource and then check which of those resources configure authentication vs. a different security feature. I should just be able to quickly list all the resources in the cluster by the feature I'm trying to configure and get a rough idea from the info via kubectl get that I dont need to dig into any thing more in depth to figure out what my cluster config is.

Makes it easier for us to iterate over each individual feature separately. Not sure that the further grouping of a bunch of different things together is really benefiting the user but I'd like to hear any counter arguments.

IE: kubectl get ratelimits let's me roughly see the ratelimiting config for the cluster and then I can dive into a specific one if needed based on the previous info.

Vs. kubectl get policy I can see there are a bunch of policies configured for the cluster but I have to go into more depth with each of them to figure out what all is being configured.

Just to make my vote more clear:
4 > 3 > 2 > 1 (4 being the most desirable and 1 being the least)

[sunjayBhatia]

Also, see the Policy Boilerplate section which uses MUST language around the default and override fields for Policies. The hierarchical nature and defaulting/override behavior is a core concept of the Policy resource. Using more focused resources IMO leads to a lot less complexity for implementers and users when taking this into account.

[arkodg]

Policy Boilerplate

I think the complexity of merging individual policies is the same for 3 and 4

3 just gives the flexibility of merging personas into one (a small independant team) or allowing
individual personas to author and instantiate many policy resources

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: SecurityPolicy
 ....
 ....
 spec:
   authentication:
   .....
   cors:
   .....
   default:
     ......  
     authentication:
     .....
     cors:
     ....

[arkodg]

I think 3 would be preferable over 1 and 2 although I prefer 4 the most. Allows for clear separation of resources depending on what the user is trying to accomplish. Prioritize keeping things self contained. While you could group two features into a single group like networking / security I feel that when a user has the goal of setting up a ratelimt they shouldn't need to worry about what other resources exist and then check to see if they have ratelimits configured on them and need to be edited vs. extended versus creating a new resource.

IE: I'm a cluster admin and I know that I want to setup authentication, I shouldn't have to use kubectl for some other resource and then check which of those resources configure authentication vs. a different security feature. I should just be able to quickly list all the resources in the cluster by the feature I'm trying to configure and get a rough idea from the info via kubectl get that I dont need to dig into any thing more in depth to figure out what my cluster config is.

Makes it easier for us to iterate over each individual feature separately. Not sure that the further grouping of a bunch of different things together is really benefiting the user but I'd like to hear any counter arguments.

IE: kubectl get ratelimits let's me roughly see the ratelimiting config for the cluster and then I can dive into a specific one if needed based on the previous info.

Vs. kubectl get policy I can see there are a bunch of policies configured for the cluster but I have to go into more depth with each of them to figure out what all is being configured.

thanks for raising the argument around retrieving current intent.

• from a production point of view, the intent will version controlled / git and checked before being edited
• from a live state perspective, I can understand the case when the user only wants to retrieve the ratelimit block having a command like kubectl get ratelimit is easier than kubectl get trafficpolicy, but there are also cases (and I think this is more of a common case then edge case) where this same user owns and cares about multiple other traffic/networking properties such as loadbalancing and prefers all features being clubbed into the same resource.
  • there's also the case where the user doesnt remember the names of all the CRDs, and wants to remember the bare minimum to achieve their goal, so they might not be able to / or want to remember the exact names of each CRD such as ratelimiting, loadbalancing, waf, cors

[danehans]

xref #675 as @sunjayBhatia has spent considerable time researching this topic for Contour. As I stated here, I suggest EG focus on extensions that can be implemented as implementation-specific filters. EG can add Policy Attachment support at a future date to support overrides and defaulting.

[danehans]

+1 on 4

[lizan]

I would also +1 on 4 (non-maintainer), that gives each feature independence and they can evolve in their own pace as well.

[zinuga]

+1 for 2 and 3 as they align with well with networking and security persona

[danehans]

2 and 3 as they align with well with networking and security persona

Note that 4 meets this requirement while supporting more granular RBAC.

[arkodg]

xref #675 as @sunjayBhatia has spent considerable time researching this topic for Contour. As I stated here, I suggest EG focus on extensions that can be implemented as implementation-specific filters. EG can add Policy Attachment support at a future date to support overrides and defaulting.

• for features like auth it might need to be enforced, so policy fits better for that case imho
• the deprecation for policy and extension filter will look similar once the features have been graduated into GAPI, so no difference from that perspective

[danehans]

for features like auth it might need to be enforced, so policy fits better for that case IMHO

True, but AuthN "enforcement" should not be conflated with providing EG users with a request authentication mechanism. I agree that "enforcement" should be achieved through GWAPI Policy Attachment but as a separate backlogged feature.

the deprecation for policy and extension filter will look similar once the features have been graduated into GAPI, so no difference from that perspective

IMHO GWAPI Policy Attachment is still evolving (see xrefs) while implementation-specific filters are far more mature and ready to implement today.

xrefs:

• https://docs.google.com/document/d/1p_kmAOmqebX8zyHnXW5I58cD8ZhNX4QNSa_gw8uAYxw/edit?usp=sharing
• Policy Attachment: Priority/order of generated config when multiple Policies applied to a resource kubernetes-sigs/gateway-api#1498

[arkodg]

thanks everyone for sharing your votes and comments, realized that chatting about PolicyAttachment might be too premature, and will require us to first agree on which extension points we should be using and when, hoping we can use #675 for aligning together about which extension type to use.

[youngnick]

So, seems like I missed the discussion here. I'm a strong +1 for option 4, even with the cost it will impose on EG development, because (in addition to @AliceProxy and @sunjayBhatia's excellent arguments):

• I think the Policy objects should be called _Envoy_RatelimitingPolicy, _Envoy_AuthNPolicy, _Envoy_TimeoutPolicy, etc. We are the official Envoy Gateway API implementation, and work we do to define Envoy-specific policies can then be useful for other projects that are Envoy based. Using an Envoy prefix makes this a bit clearer.
• As I think I mentioned before, it's much easier to take a single object and discuss it upstream, since each object is tightly focussed.

That said, I think that having custom Filters or other extensions that can be configured in a hierarchical way using Policy objects seems like the most flexible option. (I'm going to remove the restriction on this in the Gateway API docs very soon).

[zirain]

+1 on 4

[arkodg]

thanks everyone for sharing your preference, the majority of the maintainers have voted for option 4 (1 CRD per API Gateway feature)

[arkodg]

@youngnick reg your comment about naming, my opinion is that we should not add any Envoyisms into the name or the feature API. The ideal goal would be to build platform agnostic APIs that can be moved into the Gateway API repository in the future. Any Envoyisms should be part of the advanced mode Envoy API

[youngnick]

Thanks @arkodg, but I disagree. Almost everything we do for Ratelimiting, Auth etc will be heavily influenced by how Envoy configures things anyway, so we should lean into making ours the Envoy versions, and then taking those to the community and seeing what can be made generic.

Rob and I spent days looking into how to write a generic Timeout policy across implementations, and it is impossible. Each datapath implementation thinks of timeouts very differently, and they are a very simple use case by comparison to ratelimiting etc.

Doing Envoy-ism versions first lets us both show how the Policy framework can be used, and gives our users (and the users of other Envoy datapath implementations) quicker access to this functionality. It also prevents name collisions later, in the happy case that we can make a generic version.

[danehans]

^ would also be consistent with existing EG API naming, EnvoyGateway and EnvoyProxy.

[danehans]

If we do move forward with PolicyAttachment, separate CRDs should be used.