Weighted BackendRefs not showing expected outcome

[ardikabs]

Description:
I've got the intention to implement weighted backendRef for canary purposes.
Following through this documentation https://gateway.envoyproxy.io/latest/user/http-traffic-splitting/#weighted-backendrefs, didn't show an expected outcome for the weighted, it means given the weight ratio between services, such as backend A set the weight to 8, then backend B set the weight to 2, the result of Envoy Proxy config (both RDS and LDS) doesn't show the expected implementation, which should be 8:2 ratio.

Expectation:
The expectation is similar to the doc mentioned which is

The HTTPRoute below will configure the gateway to send 80% of the traffic to the backend service, and 20% to the backend-2 service.

Repro steps:
Assuming the backend services are ready as in backend and backend-2 with 1 replica.

Then apply below HTTPRoute

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-headers
spec:
  parentRefs:
  - name: eg
  hostnames:
  - backends.example
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - group: ""
      kind: Service
      name: backend
      port: 3000
      weight: 8
    - group: ""
      kind: Service
      name: backend-2
      port: 3000
      weight: 2

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Environment:
Latest / v0.6.0

Logs:

• RDS:

dynamicRouteConfigs:
- lastUpdated: "2023-11-09T15:14:02.825Z"
  routeConfig:
    '@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
    ignorePortInHostMatching: true
    name: envoy-gateway-system/eg/http
    virtualHosts:
    - domains:
      - backends.example
      name: envoy-gateway-system/eg/http/backends_example
      routes:
      - match:
          prefix: /
        name: httproute/envoy-gateway-system/http-headers/rule/0/match/0/backends_example
        route:
          cluster: httproute/envoy-gateway-system/http-headers/rule/0
  versionInfo: c0ca1f138a0b4e661716ad9b97eae569732c3b77b83d93ca93dbdc7c25daa9a9

• CDS:

dynamicActiveClusters:
- cluster:
    '@type': type.googleapis.com/envoy.config.cluster.v3.Cluster
    commonLbConfig:
      localityWeightedLbConfig: {}
    connectTimeout: 10s
    dnsLookupFamily: V4_ONLY
    edsClusterConfig:
      edsConfig:
        ads: {}
        resourceApiVersion: V3
      serviceName: httproute/envoy-gateway-system/http-headers/rule/0
    lbPolicy: LEAST_REQUEST
    name: httproute/envoy-gateway-system/http-headers/rule/0
    outlierDetection: {}
    perConnectionBufferLimitBytes: 32768
    type: EDS
  lastUpdated: "2023-11-09T14:01:30.025Z"
  versionInfo: a153eda209a91c54127227c8ecf047cbc4db498d0c54c6bc0c916cd793c1f9a3

• EDS/CLA:

dynamicEndpointConfigs:
- endpointConfig:
    '@type': type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment
    clusterName: httproute/envoy-gateway-system/http-headers/rule/0
    endpoints:
    - lbEndpoints:
      - endpoint:
          address:
            socketAddress:
              address: 10.252.0.57
              portValue: 3000
          healthCheckConfig: {}
        healthStatus: HEALTHY
        loadBalancingWeight: 1
      - endpoint:
          address:
            socketAddress:
              address: 10.252.0.20
              portValue: 3000
          healthCheckConfig: {}
        healthStatus: HEALTHY
        loadBalancingWeight: 1
      loadBalancingWeight: 2
      locality: {}
    policy:
      overprovisioningFactor: 140

[lemonlinger]

It seems we should use Weighted Cluster to distribute traffic among various backend services. Each weighted cluster corresponds to a specific backend service.

Here is an example of Envoy config to use weighted clusters.

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 10000 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route:
                  weighted_clusters:
                    clusters:
                    - name: cluster_1
                      weight: 2
                    - name: cluster_2
                      weight: 8
          http_filters:
          - name: envoy.filters.http.router
  clusters:
  - name: cluster_1
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: cluster_1
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: backend1
                port_value: 8080
  - name: cluster_2
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: cluster_2
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: backend2
                port_value: 8080

[ardikabs]

Apart from Weighted BackendRef, I observed EG always creates corresponding backends to each HTTPRoute with a dedicated cluster, even though it refers to the same backend.
For instance, if we define 2 different HTTPRoutes for the same backend, EG will create 2 different clusters, it means, when in the future we introduce a circuit breaker functionality (#2125), it requires to be enabled on each xRoute that uses the same backend.

@arkodg , Is it also intended behavior from EG? Because I am under the impression that the backend used among xRoute will share the similar behavior later.

[arkodg]

@lemonlinger we dont use Weighted Cluster for traffic splitting.
Check out the test case which highlights the weights - input & output and also here
We need to debug why this works offline but not at runtime

@ardikabs hope we can focus on weighted backendref issue in this issue, to answer your question around mapping Service to xds Cluster is not 1:1 but its per HTTPRoute Rule per Service to allow different BackendTrafficPolicy to target different route rules

[ardikabs]

Hi @arkodg , i think the unexpected behavior is coming from Envoy itself. I tried to log locally for the ClusterLoadAssigment result in Envoy Gateway, like below, which is aligned with the EG test case.

{
  "cluster_name": "httproute/envoy-gateway-system/http-headers/rule/0",
  "endpoints": [
    {
      "locality": {},
      "lb_endpoints": [
        {
          "HostIdentifier": {
            "Endpoint": {
              "address": {
                "Address": {
                  "SocketAddress": {
                    "address": "10.252.0.57",
                    "PortSpecifier": {
                      "PortValue": 3000
                    }
                  }
                }
              }
            }
          },
          "load_balancing_weight": {
            "value": 1
          }
        },
        {
          "HostIdentifier": {
            "Endpoint": {
              "address": {
                "Address": {
                  "SocketAddress": {
                    "address": "10.252.0.67",
                    "PortSpecifier": {
                      "PortValue": 3000
                    }
                  }
                }
              }
            }
          },
          "load_balancing_weight": {
            "value": 1
          }
        },
        {
          "HostIdentifier": {
            "Endpoint": {
              "address": {
                "Address": {
                  "SocketAddress": {
                    "address": "10.252.0.66",
                    "PortSpecifier": {
                      "PortValue": 3000
                    }
                  }
                }
              }
            }
          },
          "load_balancing_weight": {
            "value": 1
          }
        }
      ],
      "LbConfig": null,
      "load_balancing_weight": {
        "value": 2
      }
    },
    {
      "locality": {},
      "lb_endpoints": [
        {
          "HostIdentifier": {
            "Endpoint": {
              "address": {
                "Address": {
                  "SocketAddress": {
                    "address": "10.252.0.20",
                    "PortSpecifier": {
                      "PortValue": 3000
                    }
                  }
                }
              }
            }
          },
          "load_balancing_weight": {
            "value": 1
          }
        }
      ],
      "LbConfig": null,
      "load_balancing_weight": {
        "value": 8
      }
    }
  ]
}

But the retrieved ClusterLoadAssigment in the Envoy Proxy, show the following result:

dynamicEndpointConfigs:
- endpointConfig:
    '@type': type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment
    clusterName: httproute/envoy-gateway-system/http-headers/rule/0
    endpoints:
    - lbEndpoints:
      - endpoint:
          address:
            socketAddress:
              address: 10.252.0.57
              portValue: 3000
          healthCheckConfig: {}
        healthStatus: HEALTHY
        loadBalancingWeight: 1
      - endpoint:
          address:
            socketAddress:
              address: 10.252.0.67
              portValue: 3000
          healthCheckConfig: {}
        healthStatus: HEALTHY
        loadBalancingWeight: 1
      - endpoint:
          address:
            socketAddress:
              address: 10.252.0.66
              portValue: 3000
          healthCheckConfig: {}
        healthStatus: HEALTHY
        loadBalancingWeight: 1
      - endpoint:
          address:
            socketAddress:
              address: 10.252.0.20
              portValue: 3000
          healthCheckConfig: {}
        healthStatus: HEALTHY
        loadBalancingWeight: 1
      loadBalancingWeight: 8
      locality: {}
    policy:
      overprovisioningFactor: 140

[zhaohuabing]

Looks like the two LocalityLbEndpoints have been merged by Envoy because the locality is the same (null).

A quick & dirty fix:

              locality := &endpointv3.LocalityLbEndpoints{
			Locality: &corev3.Locality{
				Region: "region" + strconv.Itoa(i),  // Assign different values to the Region for two LocalityLbEndpoints
			},
			LbEndpoints: endpoints,
			Priority:    0,
		}

{
  "envoy-gateway-system": {
    "envoy-default-eg-e41e7b31-b5547d87b-7gml6": {
      "@type": "type.googleapis.com/envoy.admin.v3.EndpointsConfigDump",
      "dynamicEndpointConfigs": [
        {
          "endpointConfig": {
            "@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
            "clusterName": "httproute/default/http-headers/rule/0",
            "endpoints": [
              {
                "lbEndpoints": [
                  {
                    "endpoint": {
                      "address": {
                        "socketAddress": {
                          "address": "10.244.0.8",
                          "portValue": 3000
                        }
                      },
                      "healthCheckConfig": {}
                    },
                    "healthStatus": "HEALTHY",
                    "loadBalancingWeight": 1
                  }
                ],
                "loadBalancingWeight": 8,
                "locality": {
                  "region": "region0"
                }
              },
              {
                "lbEndpoints": [
                  {
                    "endpoint": {
                      "address": {
                        "socketAddress": {
                          "address": "10.244.0.10",
                          "portValue": 2000
                        }
                      },
                      "healthCheckConfig": {}
                    },
                    "healthStatus": "HEALTHY",
                    "loadBalancingWeight": 1
                  }
                ],
                "loadBalancingWeight": 2,
                "locality": {
                  "region": "region1"
                }
              }
            ],
            "policy": {
              "overprovisioningFactor": 140
            }
          }
        }
      ]
    }
  }
}

[arkodg]

thanks for debugging this one @zhaohuabing !
I'd vote to set locality to <name>/setting/<index> so it would look like httproute/default/http-headers/rule/0/setting/0 to guarantee uniqueness

[ardikabs]

can we cherry-pick the fix into the recent version (v0.6.0), @arkodg?

[arkodg]

@ardikabs we will do patch releases once we GA (v1.0), hopefully soon, for now, requesting you to use v0.0.0-latest if you really need this change, the next minor release should have this change