Document limitations on multiple routes with identical host name

[guicassolato]

Due to the importance of host names in the data plane for selecting the policies to enforce at the requests, Kuadrant only provides support for targeting HTTPRoutes that do not declare nor inherit host names identically declared or inherited by other routes. This includes routes that are directly targeted with a policy as well as routes that inherit policies from the gateway, which must also not declare host names that identically conflict with other routes.

To be able to support routes with identical host names (declared or inherited from the gateway), the policy lookup would have to include as well other attributes of the HTTP request (path, headers, etc), allowing for a perfect mapping between route honored by the gateway and policy set targeting the route (directly or indirectly.) Arguably, the lookup key might be needed to expand even beyond HTTP attributes.

Keeping it at the abstract level, the key used to index and lookup for the set of policy rules to enforce should deterministically map 1:1 to the subset of the network traffic to protect by those rules. Such subset of the network would not be the same as any other subset desired to keep unprotected or protected under different policy rules.

Often, the subset of the network is defined in the form of a HTTPRoute object, possibly narrowed to a subset of its host names and internal HTTP matching rules. In general, with more kinds of network resources supported in the targetRef of Kuadrant policies in the future, anything that delimits a subset of the traffic, such as a service name, a pod selector, a namespace, etc, would qualify as a unique identifier to index and to trigger policy rules in the data plane.

While a more robust deterministic mapping between any generically defined subset of the network and its corresponding policies to enforce is not implemented, routes with identical host names cannot be properly supported.

The problem of conflicting host names affects especially the enforcement of AuthPolicy and RateLimitPolicy resources, not having been reported for the DNSPolicy and TLSPolicy so far. To exemplify, consider the following topology:

            gateway <--- authpolicy-2
            (*.io)     ratelimitpolicy-2
           /      \
     route-a      route-b
     (app.io)     (app.io)
      /foo          /bar
        ^
        |
  authpolicy-1
ratelimitpolicy-1

On requests to app.io/foo, both authpolicy-1 and ratelimitpolicy-1 are enforced as expected. Whereas on requests to app.io/bar, unexpectedly, neither authpolicy-2 nor ratelimitpolicy-2 will be enforced.

This is because, from both the auth service's point of view and the wasm-shim's point of view, app.io is linked to authpolicy-1 and to ratelimitpolicy-1 respectively. /foo works only as a secondary condition in those decision points, after authpolicy-1 and ratelimitpolicy-1 have already been selected. By not falling back to authpolicy-2/ratelimitpolicy-2 (linked to *.io) on requests to /bar, no policy is enforced and the request is deemed authorized/unlimited.

This problem would not undergo with more control over the request that is sent to the protection services. Therefore, it is, to an extend, an easier problem to solve for rate limiting today than it is for auth, due to greater control over the request that is sent to Limitador (performed by the wasm-shim), compared to no control at all over the request to Authorino (limited by the implementation of Istio AuthorizationPolicy).

While Kuadrant does not have full control over the routing that triggers the request to ext-authz and no control at all over the payload that is sent to Authorino, the indexation and triggering of auth policies is based only on the host names. Other HTTP attributes (path, headers, verb, etc) can only determine whether a particular policy already selected by host name will be enforced or ignored. This second level of decision is implemented in the form of "top-level conditions". Nevertheless, a police, once selected, if ignored, does not cause other secondary policies to be searched and triggered alternatively.

Consistent behavior is implemented in the wasm-shim, which also uses the host as lookup key to find a configured RateLimitPolicy. Then, the other conditions are employed at a second layer, to decide whether to call Limitador. But similarly to what happens in Authorino, if the next conditions do not match, no alternate RateLimitPolicy is searched.

Multiple approaches are currently under consideration to overcome this limitation of Kuadrant, with some known caveats and different levels of effort and complexity to implement. These include:

1. Modifications in the lookup system of the auth service to enable falling back to more generic policies in case of top-level conditions mismatch and analogous work in the wasm-shim for rate limiting → can only cover for a portion of the known use cases, redundant with existing features of the auth service, 2 sprints of work (Authorino and wasm-shim) to do the exact same thing
2. Modifications in the wasm-shim to support indexing beyond host names and replacement of Istio AuthorizationPolicy API for triggering external authorization by own implementation in the wasm-shim as well, thus making possible to leverage the auth service's already existing alternative to host names → most viable option in the long-term, though with increased dependency on wasm for integration; still flawed against non-deterministic decision at the gateway regarding which route to honor if not based on HTTP attributes
3. Modifications in the wasm-shim to support indexing beyond host names and work together with Istio to add support for Envoy per-route external authorization configs in the Istio AuthorizationPolicy API → hard to implement in Istio; likely to face the same challenge later on with other Gateway API implementation (e.g. Envoy Gateway); could solve for auth but not completely for rate limiting
4. Merging of policies into single bigger internal configs, whose original rules are wrapped under more specific conditions → risk of conflicting rules and conditions, may require changes in the APIs (unique names for policy rules across policies)

[guicassolato]

Edited the description to fix about consistent behavior between auth and RL.

[trepel]

hello, I expanded this scenario by applying policies on route-b as well. Applying authpolicy-3 on route-b it fails with AuthScheme is not ready yet. The authpolicy-1 that has been applied on route-a wins it seems. However, it is different with Ratelimit policies. The ratelimitpolicy-3 applies successfully on route-b and effectively causes that the ratelimitpolicy-1 is ignored. Not sure what the behavior should be in such scenario but that inconsistency is not ideal.

[guicassolato]

hello, I expanded this scenario by applying policies on route-b as well. Applying authpolicy-3 on route-b it fails with AuthScheme is not ready yet. The authpolicy-1 that has been applied on route-a wins it seems. However, it is different with Ratelimit policies. The ratelimitpolicy-3 applies successfully on route-b and effectively causes that the ratelimitpolicy-1 is ignored. Not sure what the behavior should be in such scenario but that inconsistency is not ideal.

We should look into this. ratelimitpolicy-3 should fail just like authpolicy-3.

[guicassolato]

hello, I expanded this scenario by applying policies on route-b as well. Applying authpolicy-3 on route-b it fails with AuthScheme is not ready yet. The authpolicy-1 that has been applied on route-a wins it seems. However, it is different with Ratelimit policies. The ratelimitpolicy-3 applies successfully on route-b and effectively causes that the ratelimitpolicy-1 is ignored. Not sure what the behavior should be in such scenario but that inconsistency is not ideal.

@trepel, I've confirmed what you said.

Indeed, the behavior for the RateLimitPolicy is the opposite of the behavior with the AuthPolicy for the case of policies targeting each route. Whereas for the case I exemplified in the description (i.e. 1 gateway policy and 1 route policy), AuthPolicy and RateLimitPolicies behave the same way, i.e. enforcing <x>policy-1 on route-a and no policy at all on route-b.

I'll make sure to cover that in the docs.

Thanks!