Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gateway broken on K3d #7637

Closed
1 task
loekd opened this issue May 27, 2024 · 6 comments · Fixed by radius-project/samples#1683, radius-project/recipes#71 or radius-project/docs#1193
Closed
1 task

Gateway broken on K3d #7637

loekd opened this issue May 27, 2024 · 6 comments · Fixed by radius-project/samples#1683, radius-project/recipes#71 or radius-project/docs#1193
Assignees
@willdavsmith @brooke-hamilton
Labels
bug Something is broken or not working as expected important This item is a high priority Issue we intend to address as soon as possible

Comments

@loekd
Copy link

loekd commented May 27, 2024 •
edited
Loading

Steps to reproduce

  • Run the project as a Dev Container, using a prepared k3d Kubernetes environment. (or a Codespace)
    • There's a DevContainer in the provided repo.
  • Deploy this file
  • Use curl to fetch resources at port 80 from the nginx Service (works):
    • image
  • Use curl to try and fetch resources from one of the api services (does not work) 
    curl http://localhost/blue/api/color/
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.25.5</center>
</body>
</html>

curl http://localhost/api/color/
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.25.5</center>
</body>
</html>
    • The nginx Service receives the request, not the blue or green API.

Observed behavior

Gateway does not work.

Desired behavior

Gateway resource should work in the recommended way of Getting Started.

Workaround

Use docker desktop K8s or AKS. Gateway does work over there.

Docker Desktop:

curl  http://localhost/api/color   
green

curl  http://localhost/blue/api/color/
blue

 curl  http://localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

rad Version

RELEASE   VERSION   BICEP     COMMIT
0.34.0    v0.34.0   0.34.0    0fd82e7eaa9388fead4ea76cb9137ba2a225a236

Operating system

Windows11 running WSL Dev Container based on mcr.microsoft.com/devcontainers/base:bullseye

Additional context

No response

Would you like to support us?

  • Yes, I would like to support you

AB#12414
@loekd loekd added the bug Something is broken or not working as expected label May 27, 2024
@radius-triage-bot
Copy link

👋 @loekd Thanks for filing this bug report.

A project maintainer will review this report and get back to you soon. If you'd like immediate help troubleshooting, please visit our Discord server.

For more information on our triage process please visit our triage overview

@loekd
Copy link
Author

loekd commented May 27, 2024

  • Noticed that in docker-desktop, I get responses from Envoy, not Nginx.
  • Created the cluster without the default ingress: k3d cluster create -p '8081:80@loadbalancer' --k3s-arg '--disable=traefik@server:0'

@sylvainsf sylvainsf added the important This item is a high priority Issue we intend to address as soon as possible label May 30, 2024
@radius-triage-bot
Copy link

We've prioritized work on this issue. Please subscribe to this issue for notifications, we'll provide updates as we make progress.

We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.

For more information on our triage process please visit our triage overview

@brooke-hamilton
Copy link
Contributor

/assign @brooke-hamilton

@brooke-hamilton
Copy link
Contributor

@loekd thank you for opening this issue! 🚀

I can reproduce this on radius version 0.36.0 and k3d version v5.7.3 (current versions).

Steps to reproduce

  1. Clone the repo, launch VS Code and open the root folder of the repo, and run the dev container.
  2. When the container is running, open a new terminal window.
  3. Install radius (or build radius from source): wget -q "https://raw.githubusercontent.com/radius-project/radius/main/deploy/install.sh" -O - | /bin/bash
  4. Install k3d: wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
  5. Create a new k3d cluster: k3d cluster create -p "8081:80@loadbalancer" --k3s-arg "--disable=traefik@server:0"
  6. Install Radius on Kubernetes: rad install kubernetes --set rp.publicEndpointOverride=localhost:8081
  7. Run the kubectl command to list all pods: kubectl get pods -A

Expected result

All pods are running

Actual result

The contour-envoy pod is stuck in pending state.

$ kubectl get pods -A
NAMESPACE       NAME                                      READY   STATUS    RESTARTS   AGE
kube-system     coredns-576bfc4dc7-5xcwh                  1/1     Running   0          2m53s
kube-system     local-path-provisioner-6795b5f9d8-h9qxz   1/1     Running   0          2m53s
kube-system     metrics-server-557ff575fb-qnhdp           1/1     Running   0          2m53s
kube-system     svclb-contour-envoy-e3d65e8f-mkx54        2/2     Running   0          113s
radius-system   applications-rp-5847558fc6-ffdzm          1/1     Running   0          2m32s
radius-system   bicep-de-6d9654cd69-gkg82                 1/1     Running   0          2m32s
radius-system   contour-contour-6978b48f5d-fhndb          1/1     Running   0          113s
radius-system   contour-envoy-f88bx                       0/2     Pending   0          113s
radius-system   controller-7948677468-gk6ck               1/1     Running   0          2m32s
radius-system   dashboard-65cb99dbd4-nwbn9                1/1     Running   0          2m32s
radius-system   ucp-86b4fc44cd-7w7fs                      1/1     Running   0          2m32s

I'm working on a fix.

@brooke-hamilton
Copy link
Contributor

brooke-hamilton commented Aug 10, 2024 •
edited
Loading

The root cause is that k3d by default includes servicelb as a feature using klipper-lb, which conflicts with the load balancer deployed by Radius.

The fix is to disable servicelb when creating the k3d cluster by adding --k3s-arg "--disable=servicelb@server:*" to the create command, which will allow the load balancer deployed by Radius to manage traffic.

k3d cluster create -p "8081:80@loadbalancer" --k3s-arg "--disable=traefik@server:*" --k3s-arg "--disable=servicelb@server:*"

The Radius docs need to be updated with this information. I will submit a PR to the docs repo and close this issue when the PR is merged.

Additional information

Radius uses contour as a load balancer. You can see this by examining the deployments in the radius-system namespace.

$ kubectl get deployments -n radius-system
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
applications-rp   1/1     1            1           77s
bicep-de          1/1     1            1           77s
contour-contour   1/1     1            1           35s
controller        1/1     1            1           77s
dashboard         1/1     1            1           77s
ucp               1/1     1            1           77s

@loekd thanks again for creating this issue.
@willdavsmith thank you for pointing me to servicelb.

This was referenced Aug 10, 2024
Closed
Merged
Merged
Merged
@brooke-hamilton brooke-hamilton mentioned this issue Sep 24, 2024
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@willdavsmith willdavsmith

@brooke-hamilton brooke-hamilton

Labels
bug Something is broken or not working as expected important This item is a high priority Issue we intend to address as soon as possible
Projects
None yet
Milestone
No milestone
4 participants
@sylvainsf @loekd @willdavsmith @brooke-hamilton