Running Radius on AKS Automatic causes Node failures

[loekd]

Steps to reproduce

• Deploy Radius to AKS Automatic
• Deploy workload
• Scale workload to many instances

Observed behavior

• After some time, nodes from the node pool will crash

• They recover after some time

Desired behavior

• No node crashes

Workaround

• No workaround
• Does not happen when running workloads without Radius

rad Version

RELEASE   VERSION   BICEP     COMMIT
0.34.0    v0.34.0   0.34.0    0fd82e7eaa9388fead4ea76cb9137ba2a225a236

Operating system

Ubuntu (default node)

Additional context

No response

Would you like to support us?

• Yes, I would like to support you

AB#12487

[radius-triage-bot]

👋 @loekd Thanks for filing this bug report.

A project maintainer will review this report and get back to you soon. If you'd like immediate help troubleshooting, please visit our Discord server.

For more information on our triage process please visit our triage overview

[radius-triage-bot]

👍 We've reviewed this issue and have agreed to add it to our backlog. Please subscribe to this issue for notifications, we'll provide updates when we pick it up.

We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.

For more information on our triage process please visit our triage overview

[radius-triage-bot]

We've prioritized work on this issue. Please subscribe to this issue for notifications, we'll provide updates as we make progress.

We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.

For more information on our triage process please visit our triage overview

[rynowak]

The likely cause of this is that the UCP APIServer Extension went down. We're using a Kubernetes extensibility point that's pretty heavyweight, and can cause issues like this. The reasons why we chose that approach aren't really true any more and we should migrate away.

A better approach would be for us to port-forward to the control-plane instead of exposing it through the API Server. This would still require the user to have Kubernetes credentials (that's good) but would also be more flexible because users could expose the control-plane in whatever manner they like.

[brooke-hamilton]

/assign @brooke-hamilton

[rynowak]

A better approach would be for us to port-forward to the control-plane instead of exposing it through the API Server. This would still require the user to have Kubernetes credentials (that's good) but would also be more flexible because users could expose the control-plane in whatever manner they like.

@brooke-hamilton - if you're interested in learning more about this approach, this is what the ArgoCD CLI does. Argo's control-plane is running inside the cluster, and (by default) they port-forward so the CLI can talk to it. https://github.com/argoproj/argo-cd/blob/master/pkg/apiclient/apiclient.go#L206

[brooke-hamilton]

I opened this issue as related. Azure/AKS#4513.

[brooke-hamilton]

I believe this behavior is not related to Radius specifically. I was able to reproduce node crashes by deploying a single (non-Radius) pod to an AKS auto cluster and scaling it up to 100 instances. @loekd please comment if there is any other context we should consider. Thank you for reporting the issue! 🚀