local-kubernetes-playground

Software engineers at ETHR previously used a variation of the following project as a playground for software development, automation testing, research, and for demonstrating proof-of-concepts.

This playground was the motivation behind establishing x-ethr and its related open-source repositories.

Important

The following project requires an expansive amount of knowledge around development, kubernetes, and overall systems. While the guide can be followed step-by-step to produce a fully functioning cluster, there are requirements that would otherwise be challenging for beginners to 1. understand, 2. setup, 3. debug.

If requirements are correctly met, the entirety of this project can be deployed in under five minutes by simply following the usage section.

Users of local-kubernetes-playground will involve themselves in the following disciplines:

• Software Engineering
• DevOps
• Systems Administration
• GitOps
• Databases
• Security

Example

The Playground's Deployed Service Mesh

Requirements

Important

Usage, requirements, and documentation was vetted on a Mac Studio, M1 Max 2022 on MacOS, Sonoma 14.5. Other systems are likely subject to incompatibilities.

System

• MacOS with Administrative Privileges
• go
• cloud-provider-kind
• Homebrew
• Kind
• kubectl
• Docker Desktop
• istioctl
• ethr-cli
• flux
• psql

Optional(s)

• OpenLens - Kubernetes UI Dashboard

Usage

Note

During the first minute or two, there may be a few warnings that surface. Due to Kubernetes reconciliation, all errors should resolve by minute three or four.

1. Install kind.

   go install sigs.k8s.io/kind@latest
   sudo install "$(go env --json | jq -r ".GOPATH")/bin/kind" /usr/local/bin

2. Create a cluster via kind.

   kind create cluster --config "configuration.yaml"
   kubectl config set-context "$(printf "%s-kind" "kind")"

3. Unable node(s).

   kubectl label node kind-control-plane node.kubernetes.io/exclude-from-external-load-balancers- 

4. Setup a local load-balancer (within its own private terminal session).

   go install sigs.k8s.io/cloud-provider-kind@latest
   sudo install "$(go env --json | jq -r ".GOPATH")/bin/cloud-provider-kind" /usr/local/bin
   cloud-provider-kind -v 9

5. Verify connectivity to the cluster.
   • If using OpenLens, select the kind-kind context.
6. Bootstrap.

   flux bootstrap github --repository "https://github.com/x-ethr/cluster-management" \
       --owner "x-ethr" \
       --private "false" \
       --personal "false" \
       --path "clusters/local" \
       --verbose

   • Requires a valid GitHub personal access token set as environment variables: GITHUB_TOKEN.
   • For users outside the x-ethr organization, fork, import, or copy the https://github.com/x-ethr/cluster-management repository; or use a customized Flux GitOps project.
7. Sync local cluster repository's vendors.

   git submodule update --remote --recursive

8. Add kustomization.yaml to new cluster directory (only applicable during first-time cluster setup).

   cat << EOF > ./vendors/cluster-management/clusters/local/kustomization.yaml
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources: []
   EOF

9. Optionally, update the Kustomization.flux-system.spec.interval (changes each time a local cluster is bootstrapped).
10. Push local changes to vendors submodules.

    git submodule foreach "git add . && git commit --message \"Git Submodule Update(s)\" && git push -u origin HEAD:main" 

11. Start the local registry.

    bash ./scripts/registry.bash

12. Wait for the various resources to reconcile successfully.

Service-Mesh

The following command will port-forward the gateway's configured port 80 and expose it on localhost:8080.

kubectl port-forward --namespace development services/gateway 8080:80

Network Traffic

In order to view tracing and network traffic, issue the following command(s):

for i in $(seq 1 250); do
    curl "http://localhost:8080/v1/test-service-1"
    curl "http://localhost:8080/v1/test-service-2"
    curl "http://localhost:8080/v1/test-service-2/alpha"
    
    curl "http://localhost:8080/v1/authentication"
done

Kiali

The following command will expose the kiali service and open a browser to its dashboard.

istioctl dashboard kiali

Tracing (Jaeger)

The following command will expose the jaeger service and open a browser to its dashboard.

istioctl dashboard jaeger

Istio & istoctl

Useful istoctl command(s)

kubectl -n istio-system logs --since=1h istiod-6bc5bc58b4-wvhmc --follow

Redis

Useful kubectl command(s)

Logging

kubectl --namespace caching logs --since=10m services/redis --follow

Useful redis-cli command(s)

redis-cli

Add consumer to consumer group

xadd demo-stream * name john email jdoe@test.com
xadd demo-stream * tom tom@test.com

Contributions

Please see the Contributing Guide file for additional details.

Debugging

Ingress LB Address

kubectl get --namespace istio-system svc/ingress-gateway -o=jsonpath='{.status.loadBalancer.ingress[0].ip}'

(upstream_reset_before_response_started{connection_termination})

Restart the Istio API Gateway Deployment

External Reference(s)

• Official Schema Store
  • OpenAPI 3.1
  • OpenAPI 3.0
• AWS EKS, Crossplane, Flux Sample
  • Blog Reference
• Istio By Example
• Distributed Tracing
• Slog Guide
• Tokens & Microservice(s)
• Golang Reverse Proxy
• JWKS
• CoreOS OIDC
• Example of Microservice Communication
• Redis Message Broker
• Kubernetes & Redis
• Redis Streams