Skip to content

Conversation

ytimocin
Copy link
Contributor

@ytimocin ytimocin commented Mar 21, 2025

Description

Problem

The bicep-extensibility library was updated to require an encoded kubeConfig string in the request payload:
ExtensibilityOperationRequestExtensions.cs#L66

However, in our current recipes, we set kubeConfig to an empty string:
Search reference

This results in the extensibility provider not having the required Kubernetes config context to execute operations. We need to ensure that kubeConfig is properly set in the request after the Deployment Engine (DE) receives it from UCP.

Solution

We handle this differently depending on the runtime environment:

Production scenarios (DE running in a cluster):

Use Kubernetes in-cluster configuration via KubernetesClientConfiguration.InClusterConfig:
https://github.com/kubernetes-client/csharp/blob/master/src/KubernetesClient/KubernetesClientConfiguration.InCluster.cs#L41

Local development scenarios (DE running outside a cluster):

  • Load kubeConfig from the user’s local kubeconfig file.
  • If the active Radius workspace does not match the current Kubernetes context in the kubeconfig, we throw an error to avoid accidental mis-targeting.
  • This is to ensure that the local development environment is configured intentionally and predictably.

Implementation Notes

We introduce the following environment variables to support this logic:

  • RAD_LOCAL_DEVELOPMENT (required):

Must be set to true when running the control plane locally. This flag determines whether to use local vs. in-cluster kubeconfig resolution.

  • KUBECONFIG (optional):

Can be set to override the default kubeconfig file location (~/.kube/config).

Type of change

  • This pull request is a minor refactor, code cleanup, test improvement, or other maintenance task and doesn't change the functionality of Radius (issue link optional).

Fixes: #issue_number

Contributor checklist

Please verify that the PR meets the following requirements, where applicable:

  • An overview of proposed schema changes is included in a linked GitHub issue.
    • Yes
    • Not applicable
  • A design document PR is created in the design-notes repository, if new APIs are being introduced.
    • Yes
    • Not applicable
  • The design document has been reviewed and approved by Radius maintainers/approvers.
    • Yes
    • Not applicable
  • A PR for the samples repository is created, if existing samples are affected by the changes in this PR.
    • Yes
    • Not applicable
  • A PR for the documentation repository is created, if the changes in this PR affect the documentation or any user facing updates are made.
    • Yes
    • Not applicable
  • A PR for the recipes repository is created, if existing recipes are affected by the changes in this PR.
    • Yes
    • Not applicable

@ytimocin ytimocin requested review from a team as code owners March 21, 2025 14:43
Copy link

codecov bot commented Mar 21, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 55.65%. Comparing base (202e099) to head (20d66ca).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #8932   +/-   ##
=======================================
  Coverage   55.65%   55.65%           
=======================================
  Files         599      599           
  Lines       40750    40750           
=======================================
+ Hits        22678    22681    +3     
+ Misses      16353    16352    -1     
+ Partials     1719     1717    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link

github-actions bot commented Mar 21, 2025

Unit Tests

3 659 tests  ±0   3 657 ✅ ±0   6m 54s ⏱️ +23s
  296 suites ±0       2 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 20d66ca. ± Comparison against base commit 202e099.

♻️ This comment has been updated with latest results.

@ytimocin ytimocin force-pushed the ytimocin/update-docs-and-launch-config branch from 76603ff to 2264d41 Compare March 21, 2025 17:43
@ytimocin ytimocin temporarily deployed to functional-tests March 21, 2025 17:43 — with GitHub Actions Inactive
@radius-functional-tests
Copy link

radius-functional-tests bot commented Mar 21, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 2264d41
Unique ID funcfadd169278
Image tag pr-funcfadd169278
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-funcfadd169278
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-funcfadd169278
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-funcfadd169278
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-funcfadd169278
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-funcfadd169278
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting ucp-cloud functional tests...
⌛ Starting corerp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

Copy link
Contributor

@kachawla kachawla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bicep-extensibility library was updated to require an encoded kubeConfig string in the request payload:
ExtensibilityOperationRequestExtensions.cs#L66

@ytimocin The linked file was last updated a year ago, so I'm curious why did we start running into this issue now. Could you please share more details on this?

@ytimocin ytimocin force-pushed the ytimocin/update-docs-and-launch-config branch from 2264d41 to 77b4566 Compare March 24, 2025 22:05
@ytimocin
Copy link
Contributor Author

@ytimocin The linked file was last updated a year ago, so I'm curious why did we start running into this issue now. Could you please share more details on this?

We were using bicep-extensibility as a submodule and the version we were using was at least 2 years old. Now, we have removed the submodules and started using Extensibility as a dependency and we are pointing to a very new version.

I hope that makes sense. Let me know if you have any other questions @kachawla.

@@ -193,7 +205,6 @@ You should be able to successfully the following commands from the Radius reposi

```sh
ls ../deployment-engine/src
ls ../deployment-engine/submodules/bicep-extensibility/src
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this not needed anymore?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we removed submodules from DE. It can be linked to the other comment you had above.

Comment on lines -226 to -231
Pull latest of the `radius-project/deployment-engine` project.
Run submodule update to update bicep extensibility support for extensible resources:

```bash
git submodule update --init --recursive
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question as above - why is this not needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Submodules are removed. We are only using dependencies now.

"RADIUSBACKENDURI": "http://localhost:9000"
"RADIUSBACKENDURI": "http://localhost:9000",
"KUBECONFIG": "${env:HOME}/.kube/config",
"RAD_LOCAL_DEVELOPMENT": "true"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this env variable (RAD_LOCAL_DEVELOPMENT) consumed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was going to be used as a way to understand if a local development was in progress or not. After our discussion with @brooke-hamilton I decided to remove these added flags. Thanks for your review!

@ytimocin ytimocin force-pushed the ytimocin/update-docs-and-launch-config branch from 77b4566 to 24cfd2f Compare March 24, 2025 22:40
Signed-off-by: ytimocin <ytimocin@microsoft.com>
@ytimocin ytimocin force-pushed the ytimocin/update-docs-and-launch-config branch from 24cfd2f to 20d66ca Compare March 24, 2025 22:42
@ytimocin ytimocin temporarily deployed to functional-tests March 24, 2025 22:42 — with GitHub Actions Inactive
@kachawla
Copy link
Contributor

@ytimocin The linked file was last updated a year ago, so I'm curious why did we start running into this issue now. Could you please share more details on this?

We were using bicep-extensibility as a submodule and the version we were using was at least 2 years old. Now, we have removed the submodules and started using Extensibility as a dependency and we are pointing to a very new version.

I hope that makes sense. Let me know if you have any other questions @kachawla.

Makes sense. Thanks for sharing the details.

@radius-functional-tests
Copy link

radius-functional-tests bot commented Mar 24, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 20d66ca
Unique ID func0e7821a781
Image tag pr-func0e7821a781
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func0e7821a781
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func0e7821a781
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func0e7821a781
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func0e7821a781
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func0e7821a781
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

@ytimocin ytimocin merged commit 1a1a30f into main Mar 25, 2025
32 checks passed
@ytimocin ytimocin deleted the ytimocin/update-docs-and-launch-config branch March 25, 2025 00:15
Copy link
Contributor

@kachawla kachawla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ytimocin One more question (commented on the wrong PR earlier) - should Radius documentation be updated to guide users about different options for kube config?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants