Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RateLimitPolicy v2 #8

Merged
merged 17 commits into from
Mar 22, 2023
Merged

RateLimitPolicy v2 #8

merged 17 commits into from
Mar 22, 2023

Conversation

guicassolato
Copy link
Contributor

No description provided.

@guicassolato guicassolato self-assigned this Feb 2, 2023
@guicassolato guicassolato added the RFC Request For Comments label Feb 2, 2023
@guicassolato guicassolato force-pushed the rfc/rlp-cleanup branch 4 times, most recently from d04d63a to 0cc37bd Compare February 6, 2023 11:03
@guicassolato guicassolato changed the title RateLimitPolicy cleanup RateLimitPolicy v2 Feb 9, 2023
- name: writers
selectors:
# this is a condition
- selector: request.http.method
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So is the difference here that one has qualifiers and one doesn't. It is is an interesting idea, it did take me a minute to understand that by reading it

Copy link
Contributor Author

@guicassolato guicassolato Feb 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maleck13, I imagine you refer to distinction between what is a "condition" and what is a "counter qualifier". While the former includes an operator and value to be matched, the latter is just the selector.

This was one of the suggestions that came up during one of our tech discussion sections. I made sure to include other possible (including more explicit) variations here in the doc: https://github.com/Kuadrant/architecture/blob/rfc/rlp-cleanup/rfcs/0000-rlp-cleanup.md#possible-variations-for-the-selectors-conditions-and-counter-qualifiers

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it for the conditions, just not 100% sold on the counters. When I read selector here it makes sense that they are defining when a limit should be applied but using it for defining the counter feels a bit odd, not that I couldn't be convinced.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guicassolato I might have missed but are these selectors an and operation when there are multiple?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are, @maleck13

- limit: 100
duration: 12
unit: hour
weight: 1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am unfamiliar with the weight concept when talking about rate limting?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maleck13, it could also be called "increment by".

This is a concept in parts inherited from 3scale and, at the same time, a placeholder to kick off the discussion around a more sophisticated rate limit system where the increments are not necessarily linear to the number of hits. I'm thinking ways to increment the counter based upon variable factors such as the size of the payload, the estimated computation effort to process the request, etc – to name a few.

At the very least, in its simplest form, the option to modify the weight (or increment) of a limit can be a way for users to play with the rates without resetting the counter. It opens up implementation options as well for what's sometimes called "dynamic rate limit".

In time: I now wonder if "increment" isn't indeed a better name for the thing, so to avoid the confusion with something else, like weighting between limits (e.g. to infer priority).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah increment seems more explicit. I def think this would be an optional field with a default of 1 though

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question for me is also do we have a solid usecase at this point? It is easy to add properties but difficult to remove (breaking compatibility). So perhaps it is something we internalise within kuadrant and then potentially express externally when we see the use cases arise for adjusting the counter?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about delta optional field which defaults to 1?

Copy link
Member

@alexsnaps alexsnaps Mar 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unsure whether we should add spec.limits.increment at this stage, but won't fight it neither… otoh, I wonder if a disabled or something flag could be useful? i.e. not using a magic value as a limit (e.g. 0)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed from the RFC.

As agreed, the increment feature is out of scope. It deserves its own proposal to be discussed separately.

- selector: request.http.method
operator: eq
value: GET
- selector: request.http.path
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the simplicity here. When this is set, it would be good to validate that this path selector matches a rule in the HTTPRoute WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's possible for everything, but in general yes. For things like headers or checking if a path prefix is contained within another, etc, such validation can be challenging.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think validating the path and method would be a good place to start.

Copy link
Collaborator

@maleck13 maleck13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general I really like these improvement to the API. I don't want to comment to much on what the implementation might look like as I am not close enough to that component.

Some things worth a little more debate:

  1. do we want to stick with selectors for the counter qualifier. I feel this could potentially cause some confustion
  2. do we want to start with the weight property or drop that from this version for now until we have more concrete usecases
  3. how does this design relate to concept of overrides and defaults

Generally a +1 from me though

@maleck13
Copy link
Collaborator

@alexsnaps @guicassolato one concern I have is that (assuming we want to stick with policy attachment) adding that to the API, even if we went only with default for now would be another new version of the API. Are we content to accept that as where we are?

Copy link
Contributor

@eguzki eguzki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I like the abstraction of conditions and variables into selectors with/without values to be matched. 👍
  • From the design it is assumed (maybe I am wrong) that the rate limiting service (Limitador) will always be called. I miss some way to specify when I do not want to call Limitador. It is usually based on HTTP method, HTTP path and domain name. 👎
  • .spec.limits[].name may not have direct impact on the rate limiting config, but I think it can be useful for reporting state or stats. 👍
  • Envoy RLS proto allows hits_addend field to specify the "weight" (I would call it hits_addend) https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ratelimit/v3/rls.proto.html. That can be implemented in the wasm-shim 👍

How would be the spec for the following use case? I want rate limiting of X request per minute (or whatever duration) if the request has the header "X-TO-BE-RL", no matter what is the value of the header?. Or the opposite, no rate limit if the request has the header "X-NOT-TO-BE-RL", no matter what is the value of the header.

## Motivation
[motivation]: #motivation

The [`RateLimitPolicy`](https://pkg.go.dev/github.com/kuadrant/kuadrant-operator@v0.2.1/api/v1beta1#RateLimitPolicy) API (v1beta1), particularly its [`RateLimit`](https://pkg.go.dev/github.com/kuadrant/kuadrant-operator@v0.2.1/api/v1beta1#RateLimit) type used in `ratelimitpolicy.spec.rateLimits`, designed in part to fit the underlying implementation based on the Envoy [Rate limit](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/rate_limit_filter) filter, has been proven to be (i) _unnecessarily complex_, as well as (ii) possibly _limiting for the extension of the API_ either beyond this specific implementation and/or for supporting use cases of rate limiting not contemplated in the original design, such as comprising the concept of _weights_.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

possibly limiting for the extension of the API either beyond this specific implementation and/or for supporting use cases of rate limiting not contemplated in the original design, such as comprising the concept of weights.

Not sure I understand this, specifically the example of weights. The motivation of the initial design is to not abstract the Envoy rate limiting API which is as far as Kuadrant can go on rate limiting. So, any abstraction on top of the Envoy's rate limit api will add limitations. Those limitations should be the toll for a better UX.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That example was before I learned about Envoy's hits_addend.

Nevertheless, my point here was that there may be things we want to support that are between the Kuadrant control plane (i.e. what's stated in the RLP) and Limitador (i.e. the RL service's capabilities and how it is configured), and that do not necessarily have a 1:1 matching to any specific feature or field provided by Envoy. Obviously, within the constraints imposed by the protocol, we need to find a way to support such things, but the way they are presented to the user do not have to be shaped by the protocol. IWO, abstracting the implementation details and create our own language for how to state RL rules and intended behaviour.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this ref here about the Envoy RateLimitRequest field hits_addend: envoyproxy/envoy#12969

- limit: 100
duration: 12
unit: hour
weight: 1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about delta optional field which defaults to 1?

@guicassolato
Copy link
Contributor Author

  • From the design it is assumed (maybe I am wrong) that the rate limiting service (Limitador) will always be called. I miss some way to specify when I do not want to call Limitador. It is usually based on HTTP method, HTTP path and domain name.

That would be the union of all the selectors. For every selector in the policy, we add a "configuration rule" (how it was previously called). IOW, if you have a selector based on a value (e.g. the HTTP method) and that value is present in the proxy before it triggers the RL filter (always the case for the HTTP method), then the proxy will set a descriptor with the value. This is a way to avoid setting rules to trigger the RL filter based on values that are never actually used in the policy, at the same time it validates that what's used in the policy makes sense in the context.

@guicassolato
Copy link
Contributor Author

I did not know about that. Thanks for pointing out!

@maleck13
Copy link
Collaborator

image

I prefer C with second choice A

@guicassolato guicassolato force-pushed the rfc/rlp-cleanup branch 12 times, most recently from e475e23 to 659bb62 Compare February 17, 2023 12:22
@guicassolato
Copy link
Contributor Author

This has been discussed in the Kuadrant tech call yesterday, but I'll leave here a comment for the record. It's about targeting individual HTTPRouteMatches vs targeting a whole HTTPRouteRule.

I think only in a world of 1:1 relation between matching rules (HTTPRouteMatch) and backends (backendRef) we'd be able enforce targeting atomic HTTPRouteRules. Unfortunately I don't think that's how people naturally write routing rules. (Maybe it's me?) I think people tend to group matching rules by backend.

Maybe it's a matter of educating people, or establishing a new "best practice"? By declaring highly granular HTTPRouteRules, to the point of 1 HTTPRouteRule == 1 HTTPRouteMatch, it might be more verbose, but it gives more flexibility, less coupling.

Coupling is also a very important aspect at play here. People should not have to write HTTPRoutes based (only or ever) on how they will then define their RLPs. Otherwise one thing is coupled to the other. Plus, add KAP (and other types of policies) here and an organisation of an HTTPRoute once thought for RL is now broken.

Besides, the RLP API must allow one to write limits that target multiple HTTPRouteRules as a way to share a counter across different backends. (Or perhaps there are other ways?)

Add there two requirements together, i.e. how people write HTTPRoutes (translated to the necessity of occasionally targeting partial rules) and possibility to target more than one rule/backend, and targeting sets of HTTPRouteMatches hopefully becomes more acceptable and something we can live with for now. Or if people do move towards HTTPRouteRule == 1 HTTPRouteMatch, then it won't matter anyway.

@eguzki
Copy link
Contributor

eguzki commented Mar 3, 2023

IMO a RLP should target a HTTPRouteRule. It makes easier for me to reason about rate limiting and auth. It is about colouring routes.

If the RLP targeted a match, the scenario would be much more complicated, kind of "multicolour" routes using the color analogy.

@guicassolato
Copy link
Contributor Author

guicassolato commented Mar 3, 2023

IMO a RLP should target a HTTPRouteRule. It makes easier for me to reason about rate limiting and auth. It is about colouring routes.

If the RLP targeted a match, the scenario would be much more complicated, kind of "multicolour" routes using the color analogy.

@eguzki, unless users write 1 HTTPRouteMatch per HTTPRouteRule, what you suggest would lead to coupling. And if users write 1 HTTPRouteMatch per HTTPRouteRule, then targeting one kind or the other semantically means the exact same thing. Therefore, enforcing atomic targeting of HTTPRouteRules means forcing users into adopting the 1 HTTPRouteMatch == 1 HTTPRouteRule pattern or they will be coupling HTTPRoutes to RLPs.

@eguzki
Copy link
Contributor

eguzki commented Mar 3, 2023

will be coupling HTTPRoutes to RLPs.

I do not see this. I see rate limiting assigned (colouring) a given route. Just like Envoy's API for rate limiting. The rate limit config is at the route level, not at the matcher level.

@guicassolato
Copy link
Contributor Author

Coupling HTTPRoutes to RLPs or 1 HTTPRouteRule == 1 HTTPRouteMatch pattern is a logical conclusion.

I guess what you're saying is that Envoy's RL API does not allow us to implement the proposal due to not offering RL at the level of the match, but one level higher. This is a good point.

In Example 2, those two independent gateway actions are only possible because of the wasm shim, but using Envoy's RL API, it wouldn't exist one rule for GET /toys* and another for POST /toys* for us to inject the two separate sets of descriptor action configurations, simply because in the HTTPRoute it's defined as one single HTTPRouteRule.

I can agree with that. Then the only way out (without going "all-in" with our wasm shim to solve this) is by forcing users to move toward 1 HTTPRouteRule == 1 HTTPRouteMatch.

We change this proposal to atomic targeting of HTTPRouteRules, users will be forced to occasionally redefine their HTTPRouteRules within their HTTPRoutes to be able to attach limits to them, then again for other types of policies, until they realise it's better breaking HTTPRouteRules to no more than one HTTPRouteMatch each so policies can be more fine-grained.

@eguzki
Copy link
Contributor

eguzki commented Mar 3, 2023

lise it's better breaking HTTPRouteRules to no more than one HTTPRouteMatch each so policies can b

I am struggling to understand this. Just to clarify: By no means I want to force GwAPI users to 1 HTTPRouteRule == 1 HTTPRouteMatch.

What I am proposing is that rates, counters and when apply to entire route.. not for every route where there is a given match.

Maybe matches should be renamed to route_matches.

@guicassolato
Copy link
Contributor Author

I am struggling to understand this. Just to clarify: By no means I want to force GwAPI users to 1 HTTPRouteRule == 1 HTTPRouteMatch.

What I am proposing is that rates, counters and when apply to entire route.. not for every route where there is a given match.

Let's run a didactic example. By induction, we start with the simplest HTTPRoute with no matches. (Non-canonical representation to save a few lines of code here.)

kind: HTTPRoute
spec:
  rules:
  - backendRefs: [toystore] # /

Now we want a RLP of 100rps on method POST. First, we edit the HTTPRoute:

kind: HTTPRoute
spec:
  rules:
  - backendRefs: [toystore] # /
  - backendRefs: [toystore] # POST /
    matches:
    - method: POST

Then we can write the RLP:

kind: RateLimitPolicy
spec:
  limits:
  - rules:
    - matches: # or "first HTTPRouteRule whose entire set of matches is identical to this"
      - method: POST
    rates:
    - limit: 100
      duration: second

Now we want 500rps for method GET. Here we go again with another HTTPRouteRule in the HTTPRoute:

kind: HTTPRoute
spec:
  rules:
  - backendRefs: [toystore] # /
  - backendRefs: [toystore] # POST /
    matches:
    - method: POST
  - backendRefs: [toystore] # GET /
    matches:
    - method: GET

And edited RLP:

kind: RateLimitPolicy
spec:
  limits:
  - rules:
    - matches:
      - method: POST
    rates:
    - limit: 100
      duration: second
  - rules:
    - matches:
      - method: GET
    rates:
    - limit: 500
      duration: second

Let's keep going... 50rps for path /expensive, only methods GET, POST, PUT.

HTTPRoute:

kind: HTTPRoute
spec:
  rules:
  - backendRefs: [toystore] # /
  - backendRefs: [toystore] # POST /
    matches:
    - method: POST
  - backendRefs: [toystore] # GET /
    matches:
    - method: GET
  - backendRefs: [toystore] # GET|POST|PUT /expensive
    matches:
    - path:
        type: PathPrefix
        value: /expensive
      method: GET
    - path:
        type: PathPrefix
        value: /expensive
      method: POST
    - path:
        type: PathPrefix
        value: /expensive
      method: PUT

RLP:

kind: RateLimitPolicy
spec:
  limits:
  - rules:
    - matches:
      - method: POST
    rates:
    - limit: 100
      duration: second
  - rules:
    - matches:
      - method: GET
    rates:
    - limit: 500
      duration: second
  - rules:
    - matches:
      - path:
          type: PathPrefix
          value: /expensive
        method: GET
      - path:
          type: PathPrefix
          value: /expensive
        method: POST
      - path:
          type: PathPrefix
          value: /expensive
        method: PUT
    rates:
    - limit: 50
      duration: second

Hopefully it's getting clear at this point how the HTTPRoute is being shaped according to the needs of the RLP.

Now we want auth (KAP) for /expensive, methods POST, PUT and DELETE.

We cannot add DELETE to the GET|POST|PUT /expensive HTTPRouteRule because that would require extending the 50rps limit beyond desired (i.e. only methods GET, POST and PUT).

We cannot target the GET|POST|PUT /expensive HTTPRouteRule in the KAP because that would extend the auth rule to method the GET method.

Solution: we break the GET|POST|PUT /expensive HTTPRouteRule into two, one only for GET and another for POST|PUT, and we add a new HTTPRouteRule for DELETE. We make the RLP target 2 rules (GET /expensive and POST|PUT /expensive) and we make the KAP target also 2 rules (POST|PUT /expensive and DELETE /expensive).

HTTPRoute:

kind: HTTPRoute
spec:
  rules:
  - backendRefs: [toystore] # /
  - backendRefs: [toystore] # POST /
    matches:
    - method: POST
  - backendRefs: [toystore] # GET /
    matches:
    - method: GET
  - backendRefs: [toystore] # GET /expensive
    matches:
    - path:
        type: PathPrefix
        value: /expensive
      method: GET
  - backendRefs: [toystore] # POST|PUT /expensive
    matches:
    - path:
        type: PathPrefix
        value: /expensive
      method: POST
    - path:
        type: PathPrefix
        value: /expensive
      method: PUT
  - backendRefs: [toystore] # DELETE /expensive
    matches:
    - path:
        type: PathPrefix
        value: /expensive
      method: DELETE

RLP:

kind: RateLimitPolicy
spec:
  limits:
  - rules:
    - matches:
      - method: POST
    rates:
    - limit: 100
      duration: second
  - rules:
    - matches:
      - method: GET
    rates:
    - limit: 500
      duration: second
  - rules:
    - matches:
      - path:
          type: PathPrefix
          value: /expensive
        method: GET
    - matches:
      - path:
          type: PathPrefix
          value: /expensive
        method: POST
      - path:
          type: PathPrefix
          value: /expensive
        method: PUT
    rates:
    - limit: 50
      duration: second

KAP:

kind: AuthPolicy
spec:
  auth:
  - rules:
    - matches:
      - path:
          type: PathPrefix
          value: /expensive
        method: POST
      - path:
          type: PathPrefix
          value: /expensive
        method: PUT
    - matches:
      - path:
          type: PathPrefix
          value: /expensive
        method: DELETE
    rates:
    - limit: 50
      duration: second

What if there's another policy for POST|DELETE /expensive?

@eguzki
Copy link
Contributor

eguzki commented Mar 3, 2023

I think I see what you mean. Thanks for the more elaborated example @guicassolato

it's getting clear at this point how the HTTPRoute is being shaped according to the needs of the RLP.

I would like to avoid that by design. Routing is one thing and another is rate limiting.

HAving a single route and multiple limit objects can be accomplished with specific when fields. Right?

In the example

apiVersion: kuadrant.io/v2beta1
kind: RateLimitPolicy
metadata:
  name: toystore-per-endpoint-per-user
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  limits:
    - name: readers
      matches: 
        - path:
            type: PathPrefix
            value: "/toys"
          method: GET
        - path:
            type: PathPrefix
            value: "/toys"
          method: POST
      rates:
        - limit: 50
          duration: 1
          unit: minute
      counters:
        - auth.identity.username
      when:
        - selector:  context.request.http.method
          operator: eq
          value: GET
       - selector:  context.request.http.path
          operator: eq
          value: /toys
        - selector: auth.identity.group
          operator: neq
          value: admin
    - name: writers
      matches:
        - path:
            type: PathPrefix
            value: "/toys"
          method: GET
        - path:
            type: PathPrefix
            value: "/toys"
          method: POST
      rates:
        - limit: 5
          duration: 1
          unit: minute
        - limit: 100
          duration: 12
          unit: hour
      counters:
        - auth.identity.username
      when:
        - selector:  context.request.http.method
          operator: eq
          value: POST
       - selector:  context.request.http.path
          operator: eq
          value: /toys
        - selector: auth.identity.group
          operator: neq
          value: admin

Is that good enough? not too much DRY?

@alexsnaps
Copy link
Member

alexsnaps commented Mar 3, 2023

We are yet again reopening the same can of worms, but... tl;dr:

  • I'm arguing for having routing route reflect the policies' topology, yes;
  • We can decouple them, but we end up with the set intersection problem - which makes it almost impossible to reason about defaults & overrides;
  • We can partially decouple them, which I thought was the current agreement, so that a Policy can partially apply to a single HTTPRouteMatch within an HTTPRouteRule, so that you would "only" have to have the matcher (which are OR'ed within an HTTPRouteRule) for e.g. the methods in other to match individual methods with distinct policies within a single HTTPRouteRule - but that way, with ordering as precedence, it'd be clear with part of the policy (e.g. a limit) applies to say GETs.

So yes, this is verbose. Requires tweaking HTTPRouteRules to accommodate for particular use-cases, even as they arise (e.g. the DDoS), which means possible "talking" between roles (e.g. cluster admin and application devs)... This is far from perfect... but, and let me reiterate - I am not sarcastic, I really mean it - we can absolutely shelve this again while someone works on an alternate proposal. While this remains the "best" first step I could come up with, it does by no mean mean that there isn't a better approach staring us in the face. I'd be more than happy if that'd be the case.

So with the 3rd bullet point, the routing becomes:
kind: HTTPRoute
spec:
  rules:
  - backendRefs: [toystore]
    matches:
    - method: POST
    - method: GET
  - backendRefs: [toystore] 
    matches:
    - path:
        type: PathPrefix
        value: /expensive
     method: GET
    - path:
        type: PathPrefix
        value: /expensive
     method: POST
    - path:
        type: PathPrefix
        value: /expensive
     method: PUT
    - path:
        type: PathPrefix
        value: /expensive
     method: DELETE

You can now model the different policies using these individual matchers and have different RLP for each of the 4 methods on /expensive, while using yet different ones for /. But not for /foobar.

@guicassolato
Copy link
Contributor Author

Routing is one thing and another is rate limiting.

I think the reasons why we attach RL (or auth) to routes and route rules are:

  1. Because it's easier to reason to the user about what's going on if we can point to specific routes/route rules – e.g. "hey, you requested this limit at this route of yours, it's applied"; and the more we can say "it's applied only to this route and nothing more, nothing less", the better. (It's not 100% possible, but, to an extent, it is.)
    • All this information needs to go somehow in the status block of the RLP; it's better if can do that with the fewer level of extra details as possible. E.g. "it's route X but only when it matches condition Y"... And then there's another limit also to "route X" but subcondition "Z". Of course we can say, "limit 1 for route X when condition Y" and "limit 2 for route X when condition Z", but we hardly say something about "route X but not condition Y nor Z"... And these are only 2 limits. Arguably, in a complex RLP, to report all cases to the user, it's easier to tell the user to go read read the policy and figure out by themselves what's being applied.
    • We ourselves will need to be able to work those matches out later to solve for defaults and overrides, which I know is a different topic, but might have its roots in what we decide here.
  2. Because we want to reduce the overhead of having different languages to express effectively the same thing: "when X matches, apply Y". Reducing this overhead means less work to the user when writing the policy, when reading the policy, when keeping consistency between targeted network resource and the policy.
  3. Because, wasm shim excluded, the routes are what it's given to us to configure triggers for the RL service and the ext-authz service. It's only reasonable that we make sure those route rules exist.

HAving a single route and multiple limit objects can be accomplished with specific when fields. Right?

Now you're driving us in the opposite direction than 1 HTTPRouteRule == 1 HTTPRouteMatch. Taken to the limit, we could have just the simplest HTTPRoute with no matches at all and solve everything with when conditions.

In fact, that's exactly what would happen – again exemplifying by induction – if you add to your example above a KAP for POST|DELETE /toys with a strict constraint by the user saying "I will not break my HTTPRoute into finer HTTPRouteRules any further". Well, then, except the HTTPRouteRules needed for different backends, the user is probably in a better place with no HTTPRouteRules at all in the first place! Without them, we let all requests go to Limitador and to Authorino and let these ones decide based on "soft" conditions. Users can figure out by themselves what's being applied, and we work out defaults and overrides some other way to figure out if it's supposed to be the same route/same target or not.


To sum up, I think neither you're proposing only using when conditions for filtering, nor I'm saying we should get rid of them. I'm just saying that targeting HTTPRouteRules means that, as a user, you will often be faced with the tradeoff between quality status reporting, less language translation overhead, reasonable defaults and overrides on one hand and whatever benefit you can get from the immutability of your HTTPRoutes on the other. The more you lean to the former, the more you drive in the direction of 1 HTTPRouteRule, 1 HTTPRouteMatch.

Copy link
Contributor

@didierofrivia didierofrivia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Gui for the extensive work dedicated in this RFC, specially on examples and the digested comparison with the current RLP... and also the effort to match the KAP as well.

As I had the opportunity to reflect on the proposal both in our meetings and this RFC, some important points to highlight IMHO are the following:

Pros:

  • Super user intuitive, a great enhance on API UX
  • Having a 1-1 match between spec.limits.matches and httproute.spec.rules.matches ease the overhead when computing the status, will help to define a better Kuadrant Policies Status conditions/states #9
  • Abstracting the most out of Envoy specifics is a great step for any future different gateway implementation.
  • The granularity of the limits in time enables a finer rates definition.

Cons:

  • Verbosity defining limits per HTTPRouteMatch
  • Raised the need of keeping in sync the RLP with the Route matches...
  • ... and reflecting this in a clear way in their status.

Even tho we have some negatives implementing the proposal, the benefits are clear and will provide an uniform and comprehensive way of crafting Kuadrant policies. It's also worth mentioning that it will enable us to move forward and have a better understanding once it's started to be in use... real feedback from users will be valuable and we always can redefine it.

Great work!

Copy link
Contributor

@didierofrivia didierofrivia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎸

…the HTTPRouteRules to be bound to a limit + Limit's name becomes a first-class citizen
counters:
- auth.identity.username
- auth.identity.username
triggers:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not quite sure why we moved away from rules here... is it because of the hostname now being part of this?

Comment on lines 192 to 200
- matches: # matches the 1st HTTPRouteRule (i.e. GET or POST to /toys*)
- path:
type: PathPrefix
value: "/toys"
method: GET
- path:
type: PathPrefix
value: "/toys"
method: POST
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would:

      - matches: 
        - path:
            type: PathPrefix
            value: "/toys"

also matches the 1st HTTPRouteRule (i.e. GET or POST to /toys*) ? It would right, as this match is a subset of the two matchers in the route... am I understanding this right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking on atomic HTTPRouteMatches at first, so no. But maybe yes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, if we say "yes", then

- matches: 
  - path:
       type: PathPrefix
       value: "/toys"

would match any method, header and query string currently (or futurely) specified as well in the HTTPRouteMatches – not only GET or POST.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh ok, I misread that then... You mean any complete match will target the rule, so that if I have match A & match B to Rule 1, I don't need to target A & B, but A or B suffices... but either needs to be complete, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean any subset of the HTTPRouteMatches, not any subset of a HTTPRouteMatch.

In the example,

- matches:
  - path:
      type: PathPrefix
      value: "/toys"
    method: GET
  - path:
      type: PathPrefix
      value: "/toys"
    method: POST

are 2 HTTPRouteMatches.

The set of all subsets contains 22 = 2 × 2 = 4 elements:

  • [{}]
  • [{ path: { type: PathPrefix, value: /toys }, method: GET }]
  • [{ path: { type: PathPrefix, value: /toys }, method: POST }]
  • [{ path: { type: PathPrefix, value: /toys }, method: GET }, { path: { type: PathPrefix, value: /toys }, method: POST }]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if I have match A & match B to Rule 1, I don't need to target A & B, but A or B suffices... but either needs to be complete, right?

Correct.

If you have:

- matches:
  - path:
      type: PathPrefix
      value: "/toys"
    method: GET
  - path:
      type: PathPrefix
      value: "/toys"
    method: POST

Then just

- matches:
  - path:
      type: PathPrefix
      value: "/toys"
    method: GET

or just

- matches:
  - path:
      type: PathPrefix
      value: "/toys"
    method: POST

would suffice.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-read this and now need to step back and change what I said before.

To be possible for policy owners to define triggers in the policy that are strictly about what's intended to be protected, we have to allow for a subset of an HTTPRouteMatch. The contrary (i.e. forcing only atomic HTTPRouteMatches in the triggers) would cause the user sometimes having to add to the triggers more than intended, which is exactly what we wanted to avoid when we decided to move away from defining triggers in terms of full HTTPRouteRules.

With that, based on the example from before, one who writes the following trigger matches:

triggers:
- matches:
  - path:
      type: PathPrefix
      value: "/toys"

would have written a valid trigger and binding the limit definition to the HTTPRouteRule:

rules:
- matches:
  - path:
      type: PathPrefix
      value: "/toys"
    method: GET
  - path:
      type: PathPrefix
      value: "/toys"
    method: POST

What's put in the triggers nonetheless still have to be explicitly stated in the HTTPRouteRules. E.g. the following trigger:

triggers:
- matches:
  - path:
      type: Exact
      value: "/toys/special"

would be an invalid trigger even if it's technically a subset of the HTTPRouteRule above.


This is the opposite of what I had said. I will make sure the text is clear about that.

Thanks @alexsnaps!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Example 2 modified to make it somewhat clearer that a subset of an HTTPRouteMatch (as long as explicitly stated in the HTTPRoute) is also acceptable.

…ns for one that does not involve HTTP attributes used in 'soft' conditions
…ely stated within the trigger matches, but only a portion of it will do (if that portion is enough to scope the limit)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
RFC Request For Comments target/current
Projects
No open projects
Status: Done
Development

Successfully merging this pull request may close these issues.

6 participants