Epic: Contour Benchmarking and Performance Improvements

[sunjayBhatia]

Starting an epic on this topic to ensure work is tracked and we get organized

While we have worked to ensure Contour code performs well via optimizations in feature implementation, code reviews, and fixing performance regressions reported by production users, we do not yet have a dedicated set of tests or benchmarks that demonstrate how we expect Contour to perform in production (under some specific conditions we test against).

When we say "performance" here we're mostly talking about measuring how long Contour takes to reconcile kubernetes resources into Envoy configuration, as Contour the software component is primarily the xDS server and control plane for Envoy. A secondary performance concern may be benchmarking requests throughput in Envoy. Contour does not directly participate in the data plane however it does configure Envoy and we do provide deployment guidelines for Envoy.

Ultimately we don't want to provide "absolute" performance numbers as users may have wildly varying environments, but we could establish benchmarks of the form: Given a cluster with specific characteristics and workloads running in this cluster (that resembles a plausible production cluster), users should expect Contour to make apps exposed by Ingress/HTTPProxy/HTTPRoute resources in X time, Y percentage of the time

We also would like to make performance improvements and be able to assess how changes to Contour internals will affect Contour performance. Building an automated test suite will help us perform these sorts of experiments more easily. This work dovetails with improvements we would like to Contour xDS code for example.

List of items/issues that this work will encompass:

• Implement benchmarking test suite
  • script for deploying test env: test/scripts: Add script to deploy gcp benchmarking cluster #4247
  • (in progress) Implement test for reported issue: Performance Regression v1.8.1 => v1.18.1 #4058, initially to be run manually: test/e2e/bench: Add bench test suite #4251
  • Set up GCP account for running benchmark suite in CI
  • Run test suite in CI (daily)
  • Add test for steady-state Contour (given N Ingress resources, updating M per minute, new resources should reconcile quickly, and changes should reconcile quickly)
  • Add or modify tests for different resource limits for Contour/Envoy
  • Incorporate more tests possibly from Tighter feedback loop with knative-sandbox/net-contour #2863
  • Incorporate test around leader election, how SLO is affected when Contour goes down
• xDS changes assessment
  • refactor Contour startup to ensure non-racy xDS server initialization (see Use envoy xds server type in daily e2e tests #4100 (comment)) - completed by refactor to controller-runtime runnables and "leader election notifier" pattern
  • run e2e tests against go-control-plane xDS server in CI
  • run benchmark test suite against go-control-plane xDS server in CI
  • make go-control-plane the default xDS server
  • Delta xDS improvements

[github-actions]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack