Ideas around reporting Policy configuration in Route Status

[sunjayBhatia]

These ideas could be used by the upstream project itself to provide a mechanism for implementers of the Gateway API.
This is mostly an exercise in seeing if it is feasible to do something in this space, and what solutions could look like.

The suggestions here may be combined or used separately where appropriate.

---

Add the "effective policy configuration" to the RouteParentStatus type.
This is an effort to signify the applied policies flow down from individual Gateway hierarchies, especially given Routes may be attached to different Gateways.
We could go with an approach that does not include this information under RouteParentStatus, but would need to then duplicate some of the fields from this struct, namely controllerName to ensure we know which controller handled the Policy and set the configuration.

type RouteParentStatus struct {
	...

	// EffectivePolicyConfigurations is a list of policies in effect on this
	// Route.
	//
	// +optional
	EffectivePolicyConfigurations []RouteEffectivePolicyConfiguration `json:"effectivePolicyConfigurations"`
}

type RouteEffectivePolicyConfiguration struct {
	PolicyType PolicyGroupKind `json:"policyType"`

	SectionName SectionName `json:"sectionName"`

	PolicyValue string `json:"policyValue"`
}

// PolicyGroupKind indicates the group and kind of a Policy resource.
type PolicyGroupKind struct {
	// Group is the group of the Policy.
	Group Group `json:"group,omitempty"`

	// Kind is the kind of the Route.
	Kind Kind `json:"kind"`
}

A possible resulting Status example:

...
status:
  parents:
  - conditions:
    - lastTransitionTime: "2022-11-07T20:27:56Z"
      message: Accepted HTTPRoute
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: projectcontour.io/projectcontour/contour
    effectivePolicyConfigurations:
    - policyType:
        group: projectcontour.io
        kind: RateLimitPolicy
      policyValue: '{"requests":10,"unit":"minute","responseStatusCode":503}'
      sectionName: unsupportedfornow
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: example
      namespace: demo

This follows this spike and included example YAML for HTTPRoutes and RateLimitPolicies.
It uses this spike on the Gateway API.
Here we've also added the sectionName field which could be used if PolicyTargetReference supports it in the future and the value would the name of the HTTPRouteRule this final Policy applies to.
The policyValue is just marshalled JSON from the RateLimitPolicy for simplicity in this example.
The values here represent the combination of a Policy applied to a Gateway and the Policy applied to the HTTPRoute this Status is set on.

---

Rather than using a structured type to represent the kind of Policy that is in effect, we could use a type usable as a map key.
This would ensure one effective Policy is set per kind.
See Contour spike and Gateway API changes

type RouteEffectivePolicyConfiguration struct {
	PolicyType string `json:"policyType"`
  ...
}

Resulting in Status:

status:
  parents:
  - ...
    controllerName: projectcontour.io/projectcontour/contour
    effectivePolicyConfigurations:
    - policyType: projectcontour.io/RateLimitPolicy
      policyValue: '{"requests":10,"unit":"minute","responseStatusCode":503}'
      sectionName: unsupportedfornow
    - policyType: projectcontour.io/AuthorizationPolicy
      policyValue: ...
      sectionName: unsupportedfornow
    ...

---

Expressing the actual value of the Policy settings in Status is a bit complicated.
One way could be to introduce a PolicySettings list, that would allow us to show individual field settings and what Policy resource they came from.
We can again use the field name as a unique key to use as a map key for some validation.

type RouteEffectivePolicyConfiguration struct {
	...

	// +listType=map
	// +listMapKey=field
	PolicySettings []PolicySetting `json:"policySettings"`
}

type PolicySetting struct {
	Field string `json:"field"`

	Value string `json:"value"`

	PolicyName string `json:"policyName"`
}

See example Status below.

status:
  parents:
  - conditions:
    - lastTransitionTime: "2022-11-11T01:44:12Z"
      message: Accepted HTTPRoute
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: projectcontour.io/projectcontour/contour
    effectivePolicyConfigurations:
    - policySettings:
      - field: local.requests
        policyName: local-ratelimit-config-route
        value: "2"
      - field: local.unit
        policyName: local-ratelimit-config-route
        value: minute
      - field: local.responseStatusCode
        policyName: local-ratelimit-config-gw
        value: "503"
    ...

Here we can see a field has been set by the Policy targeting the Gateway and others set by the Policy targeting the HTTPRoute itself.

This approach as a few upsides, since it offers a lot of visibility into fields, values, and what resources they were configured in.
However, it is a bit complicated and can definitely become unwieldy if a Policy resource grows to contain many fields or any significant nested structure.

See Contour spike and Gateway API spike.

---

Instead of expressing the values specified by a hierarchy of Policies in Status directly, we could instead have the controller implementing the Route and applying Policies simply add a reference to a new Policy generated with the resulting values.

We would add to the RouteParentStatus (or other location):

type RouteParentStatus struct {
  ...

	EffectivePolicyConfigurationReferences []LocalObjectReference `json:"effectivePolicyConfigRefs"`
}

These would be a list of references to Policy objects that were generated to express what Policy values are actually in effect, given the defaults and overrides applied.
There could be some validation that the group/kind/name of these weren't repeated.

Some example status:

status:
  parents:
  - conditions:
    - lastTransitionTime: "2022-11-11T01:44:12Z"
      message: Accepted HTTPRoute
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: projectcontour.io/projectcontour/contour
    effectivePolicyConfigRefs:
    - group: projectcontour.io
      kind: RatelimitPolicy
      name: generated-name-for-effective-policy

One thing to handle here is what the targetRef in this generated Policy would be set to, since this shouldn't actually be applied to any resource.

Downsides:

• Have to give RBAC to the Gateway controller to create/update a new resource
• Effective configuration is not in the Route itself, have to go look at another object

---

The above option becomes a little more attractive, if the upstream APIs supply a PolicyAttachment type similar to ReferenceGrant that connects some implementation specific Policy resource and the Gateway API resource (or other) that it targets.

The generated Status would not reference a Gateway API PolicyAttachment resource but rather the implementation specific Policy.

type PolicyAttachment struct {
  metav1.TypeMeta   `json:",inline"`
  metav1.ObjectMeta `json:"metadata,omitempty"`

  Spec PolicyAttachmentSpec `json:"spec"`

  Status PolicyAttachmentStatus `json:"status"`
}

type PolicyAttachmentSpec struct {
	// TargetRef identifies an API object to apply policy to.
	TargetRef PolicyTargetReference `json:"targetRef"`

  Default LocalObjectReference `json:"default"`

  Override LocalObjectReference `json:"override"`
}

An instance of a PolicyAttachment might look like:

kind: PolicyAttachment
apiVersion: gateway.networking.k8s.io/v1alpha1
metadata:
  name: ratelimit-policy-attachment
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: echo
  default:
    group: projectcontour.io
    kind: RatelimitPolicy
    name: default-policy
  override:
    group: projectcontour.io
    kind: RatelimitPolicy
    name: override-policy 

This idea has other implications and might be something to explore as it gives some structure to implementers of the API.

[jm96441n]

Could this also be extended to HTTPRoute extension refs? As far as I know extension refs can suffer from similar issues to display effective configuration.

[youngnick]

Yes, I think that once we have some ideas that work for Policy attachment, we should look at using similar ideas for ExtensionRefs as well.