grpc-xds

This repository contains Go and Java implementations of xDS control planes for gRPC services running on Kubernetes.

The goal of the control plane implementations is to provide a practical understanding of xDS, beyond the API specifications and the pithy examples.

This repository also contains accompanying sample applications implemented in Go and Java. The sample applications implement the helloworld.Greeter gRPC service.

The scripts and manifests in this repository enable running the xDS control planes and the sample gRPC applications on either Google Kubernetes Engine (GKE) clusters, or on local kind Kubernetes clusters.

Features and branches

The following branches incrementally add features to the control plane implementations:

• grpc-xds/v0.1.x: Basic control plane implementations, with all clients receiving the same xDS resource snapshot.
• grpc-xds/v0.2.x: Adds data plane mTLS support, based on gRFC A29: xDS-Based Security for gRPC Clients and Servers.
• grpc-xds/v0.3.x: Adds support for locality-weighted load balancing across workloads running on multiple Kubernetes clusters in different regions.
• grpc-xds/v0.4.x: Adds workload identity based on SPIFFE, client authorization based on gRFC A41: xDS RBAC Support, and server authorization based on gRFC A29: xDS-Based Security for gRPC Clients and Servers
• grpc-xds/v0.5.x: Adds support for running Envoy instances as gateway proxies for traffic entering the mesh (north-south traffic), and documents Traffic Director configuration and deployment.

Repository structure

The control-plane-go and control-plane-java directories contain the sample xDS control plane management server implementations.

The greeter-go and greeter-java directories contain Go and Java implementations of sample gRPC servers that implement the helloworld.Greeter service. Each implementation can be deployed either as a "leaf" service, or as an "intermediary" service.

The leaf service (greeter-leaf) responds with its host name and the cloud service provider zone of the underlying compute node.

The intermediary service (greeter-intermediary) forwards the request to the next hop, which is another helloworld.Greeter service implementation. Upon receiving a response from the next hop (defined by the NEXT_HOP environment variable), the intermediary service appends its own host name and zone, before returning the response to the client.

The hack directory contains shell scripts for deploying to multiple Kubernetes clusters.

The k8s directory contains Kubernetes manifests and patches that are used by both the Go and Java implementations of the xDS control plane and the sample gRPC applications. Patches that are specific to the Go and Java implementations can be found in k8s subdirectories of control-plane-(go|java) and greeter-(go|java). Skaffold renders the complete manifests using Kustomize.

The k8s/troubleshoot directory contains Kubernetes manifests that you can use to deploy a Pod with networking and gRPC troubleshooting tools to your cluster.

Prerequisites

To run the samples, you need one or more Kubernetes clusters. The sample works great with Google Kubernetes Engine (GKE) and Artifact Registry, but you can also use local clusters using kind.

The docs directory contains instructions on both creating GKE clusters and an Artifact Registry container image repository, or creating multi-node Kubernetes clusters using kind, and on creating certificate authorities (CAs) for issuing workload TLS certificates.

If you want to build and deploy the Java implementation of the xDS control plane and sample gRPC application, you need Java 17.

If you want to build and deploy the Go implementation of the xDS control plane and sample gRPC applications, you need Go 1.23 or later.

You also need the following tools:

• kubectl at a version that is supported by the Kubernetes control plane version that you use (kube-apiserver specifically)
• Kustomize v4.5.5 or later
• Skaffold v2.10.1 or later
• gRPCurl v1.9.1 or later
• yq v4.41.1 or later

To make it easier to switch between kubeconfig contexts for multiple Kubernetes clusters, you can also optionally install kubectx.

If you use macOS, you can use brew to install kubectl, Kustomize, Skaffold, gRPCurl, yq, kind, and kubectx:

brew update
brew install kubectl kustomize skaffold grpcurl yq kind kubectx

Follow the steps in the documents Verify local development setup using Go or Verify local development setup using Java to ensure that your cluster and the prerequisite tools are set up correctly.

Local Kubernetes cluster setup using kind

Follow the the documentation on create multi-node kind Kubernetes clusters with fake zone labels (topology.kubernetes.io/zone), and set up a root certificate authority using cert-manager to issue workload TLS certificates.

GKE and Artifact Registry setup

1. Follow the documentation on creating GKE clusters, an Artifact Registry container image repository, and certificate authorities in Certificate Authority Service (CA Service) for issuing workload TLS certificates.

2. Create and export an environment variable called SKAFFOLD_DEFAULT_REPO to point to your container image registry:

   export SKAFFOLD_DEFAULT_REPO=$AR_LOCATION-docker.pkg.dev/$PROJECT_ID/$AR_REPOSITORY

   Replace the following:

   • $AR_LOCATION: the location of your Artifact Registry container image repository, e.g., us-central1 or us.
   • $PROJECT_ID: your Google Cloud project ID.
   • $AR_REPOSITORY: the name of your Artifact Registry container image repository, e.g., grpc-xds.

Running the xDS control plane and sample gRPC applications

1. Build the container images for the xDS control plane and the sample gRPC applications, render the Kubernetes resource manifests, apply them to the Kubernetes cluster, and set up port forwarding.

   Leave Skaffold running so that port forwarding keeps working.

   Choose one of the run options below, and choose whether you want to deploy the Go (go) or Java (java) implementations.

   1. Run the xDS control plane and the sample applications greeter-intermediary and greeter-leaf on one Kubernetes cluster:

      make run-[go|java]

   2. Run the sample applications greeter-intermediary and greeter-leaf on two Kubernetes clusters, and the xDS control plane on one of the Kubernetes clusters:

      make run-[go|java]-multi-cluster

      In this configuration, the sample applications running on the two Kubernetes clusters all connect to the same xDS control plane.

   3. Run the xDS control plane and the sample applications greeter-intermediary and greeter-leaf on two Kubernetes cluster, with xDS federation enabled:

      make run-[go|java]-federation

      In this configuration, the sample applications receive xDS configuration from the xDS control plane in their respective Kubernetes clusters.

      The greeter-intermediary sample application running on one of the Kubernetes clusters sets up a gRPC channel to the greeter-leaf sample application running on the other Kubernetes cluster, by retrieving xDS resources from the xDS control plane running on the other Kubernetes cluster.

   4. Run the sample applications greeter-intermediary and greeter-leaf in one Kubernetes cluster without using xDS:

      cd greeter-[go|java]
      make run-no-xds

      In this configuration, the greeter-intermediary sample application discovers and load balances connections to the greeter-leaf sample application using Kubernetes cluster DNS and cluster IP.

   You may see messages similar to the following during deployment:

   xds:deployment/greeter-leaf: FailedMount: MountVolume.SetUp failed for volume "workload-certs" : secret "greeter-leaf-cert" not found

   You can safely ignore these messages as long as the deployment proceeds, and you eventually see the message Deployments stabilized in __ seconds.

2. In a new terminal, tail the xDS control plane logs:

   make tail-control-plane

3. In another new terminal, tail the greeter-intermediary logs from the Kubernetes cluster referenced by your current kubeconfig context:

   make tail-greeter-intermediary

4. In yet another new terminal, tail the greeter-leaf logs from the Kubernetes cluster referenced by your current kubeconfig context:

   make tail-greeter-leaf

5. In - you guessed it - a new terminal, create a private key and TLS certificate for your developer workstation. You do this by creating a temporary Kubernetes Pod with a workload TLS certificate and private key, copy this certificate and private key to your developer workstation, and deleting the temporary Pod:

   make host-certs

   The workload certificate, CA certificate, and private key will be copied to the certs directory.

6. Send a request to the greeter-leaf server in one of the clusters, using mTLS and the DNS resolver:

   make request-leaf-mtls

   If you use GKE workload TLS certificates, bypass certificate verification, as gRPCurl cannot verify the SPIFFE ID in the gRPC server's workload TLS certificates:

   make request-leaf-mtls-noverify

7. Send a request to the greeter-intermediary server in one of the clusters, using mTLS and the DNS resolver:

   make request-mtls

   If you use GKE workload TLS certificates, bypass certificate verification, as gRPCurl cannot verify the SPIFFE ID in the gRPC server's workload TLS certificates:

   make request-mtls-noverify

8. Observe the xDS control plane logs as you scale the greeter Deployments. For instance, you can scale the greeter-leaf Deployment in one of the clusters:

   kubectl scale deployment/greeter-leaf --namespace=xds --replicas=2

9. To explore resources across both clusters, use either kubectx, or the kubectl config current-context, kubectl config get-contexts, and kubectl config use-context commands.

10. Delete the xDS control plane and greeter Deployment, Service, and ConfigMap resources in both Kubernetes clusters:

    make delete-apps

    This command does not delete the xds namespaces or other resources, such as the GKE workload TLS certificate configuration resources, cert-manager, or the bastion Pod.

See the Makefile for examples of other commands you can run.

Development

Deploy to one cluster only, and set up a local file watch that automatically rebuilds and redeploys the xDS control plane and gRPC applications on code changes:

make dev-[go|java]

Remote debugging

Deploy to one cluster only, and set up remote debugging by exposing and port-forwarding to delve (for Go) or the JDWP agent (for Java):

make debug-[go|java]

Troubleshooting

1. Create a bastion Pod in one of the Kubernetes clusters with various tools available to troubleshoot issues.

   make run-bastion

   This takes a few minutes, as an init container installs a number of tools.

   The bastion Pod is configured to only have access to the API server of the Kubernetes cluster where the Pod is deployed. To troubleshoot two clusters, create a bastion Pod in each cluster.

2. Open an interactive shell in the Pod's container:

   make troubleshoot

Some troubleshooting commands:

• View the Kubernetes Pod IP addresses:

  kubectl get pods --namespace=xds --output=wide

• View the operating xDS configuration of an xDS-enabled gRPC server using Client Status Discovery Service (CSDS):

  grpcurl -plaintext POD_IP:50052 envoy.service.status.v3.ClientStatusDiscoveryService/FetchClientStatus | yq --prettyPrint

  Replace POD_IP with the IP address of the Kubernetes Pod.

• List the ACK'ed xDS resources of a gRPC server using grpcdebug:

  grpcdebug POD_IP:50052 xds status

• View the operating xDS configuration of a gRPC server using grpcdebug:

  grpcdebug POD_IP:50052 xds config | yq --prettyPrint

• View the Listener Discovery Service (LDS) configuration only of a gRPC server using grpcdebug:

  grpcdebug POD_IP:50052 xds config --type LDS | yq --input-format=json --prettyPrint

  Replace LDS with other xDS services to view other ACK'ed xDS resources.

• Set the GRPC_GO_LOG_SEVERITY_LEVEL and GRPC_GO_LOG_VERBOSITY_LEVEL environment variables to see addtional log messages from gRPCurl's interaction with the xDS control plane management server:

  export GRPC_GO_LOG_SEVERITY_LEVEL=info
  export GRPC_GO_LOG_VERBOSITY_LEVEL=99

• Send a request to the greeter-leaf service using mTLS and xDS:

  grpcurl \
    -cacert /var/run/secrets/workload-spiffe-credentials/ca_certificates.pem \
    -cert /var/run/secrets/workload-spiffe-credentials/certificates.pem \
    -key /var/run/secrets/workload-spiffe-credentials/private_key.pem \
    -d '{"name": "World"}' \
    -import-path /opt/protos \
    -insecure \
    -proto google/rpc/error_details.proto \
    -proto helloworld/greeter.proto \
    xds:///greeter-leaf \
    helloworld.Greeter/SayHello

• Send a request to the greeter-intermediary service using mTLS and xDS:

  grpcurl \
    -cacert /var/run/secrets/workload-spiffe-credentials/ca_certificates.pem \
    -cert /var/run/secrets/workload-spiffe-credentials/certificates.pem \
    -key /var/run/secrets/workload-spiffe-credentials/private_key.pem \
    -d '{"name": "World"}' \
    -import-path /opt/protos \
    -insecure \
    -proto google/rpc/error_details.proto \
    -proto helloworld/greeter.proto \
    xds:///greeter-intermediary \
    helloworld.Greeter/SayHello

• View the xDS bootstrap configuration file of the bastion Pod:

  jq < $GRPC_XDS_BOOTSTRAP

Kubernetes references

• API Concepts
• Headless Services
• EndpointSlices
• Namespace Sameness

xDS references

• On the state of Envoy Proxy control planes
• xDS API Overview
• xDS REST and gRPC protocol
• xDS Features in gRPC
• Aggregated Discovery Service (ADS)
• Load Reporting Service (LRS)
• Open Request Cost Aggregation (ORCA)
• gRFC A14: gRPC Channelz
• gRFC A27: xDS-Based Global Load Balancing
• gRFC A28: gRPC xDS traffic splitting and routing
• gRFC A29: xDS-Based Security for gRPC Clients and Servers
• gRFC A30: xDS v3 Support
• gRFC A31: gRPC xDS Timeout Support and Config Selector Design
• gRFC A32: gRPC xDS circuit breaking
• gRFC A33: Client-Side Fault Injection
• gRFC A36: xDS-Enabled Servers
• gRFC A37: xDS Aggregate and Logical DNS Clusters
• gRFC A39: xDS HTTP Filter Support
• gRFC A40: xDS Configuration Dump via Client Status Discovery Service in gRPC
• gRFC A41: xDS RBAC Support
• Envoy Role Based Access Control (RBAC) HTTP Filter
• gRFC A42: xDS Ring Hash LB Policy
• gRFC A44: gRPC xDS Retry Support
• gRFC A47: xDS Federation
• gRFC A48: xDS Least Request LB Policy
• gRFC A50: gRPC xDS Outlier Detection Support
• gRFC A51: Custom Backend Metrics Support
• gRFC A52: gRPC xDS Custom Load Balancer Configuration
• gRFC A53: Option for Ignoring xDS Resource Deletion
• gRFC A55: xDS-Based Stateful Session Affinity for Proxyless gRPC
• gRFC A57: XdsClient Failure Mode Behavior
• gRFC A58: weighted_round_robin LB policy
• gRFC A65: mTLS Credentials in xDS Bootstrap File
• xRFC TP1: xdstp:// structured resource naming, caching and federation support
• xRFC TP2: Dynamically Generated Cacheable xDS Resources

Google Cloud API references

• Compute Engine API
• Network Services API (a.k.a. Service Routing API)
• Network Security API
• Common Expression Language (CEL) matcher language reference

Disclaimer

This is not an officially supported Google product.