rad deploy can run for hours without timing out

[willtsai]

Steps to reproduce

1. Configure an application deployment workflow that uses Radius.
2. Run rad deploy as part of the workflow.
3. Encounter a deployment operation that does not complete normally.
4. Observe that rad deploy continues waiting without an apparent timeout.

Example workflow job: https://github.com/willtsai/aks-store-demo/actions/runs/30588288843/job/91024711426

Observed behavior

The rad deploy command appears to wait indefinitely when the deployment does not complete. In the example GitHub Actions run, the deployment workflow continued for more than two hours before eventually failing. This ties up the workflow runner and delays actionable failure feedback.

Desired behavior

rad deploy should enforce a reasonable timeout, or provide a configurable timeout option, for deployment operations. When the timeout is reached, the command should exit with a non-zero status and a clear error identifying the timed-out deployment or resource. The workflow should not remain blocked for multiple hours by default.

Workaround

No workaround identified other than configuring an external timeout at the workflow step or job level.

rad Version

Not captured in the referenced workflow run.

Operating system

GitHub-hosted Actions runner; exact operating system and architecture were not captured.

Additional context

Example failure: https://github.com/willtsai/aks-store-demo/actions/runs/30588288843/job/91024711426

The referenced job details were not accessible through the available API, so the exact command output and final error are not included here.

[radius-triage]

👍 We've reviewed this issue and have agreed to add it to our backlog. Please subscribe to this issue for notifications, we'll provide updates when we pick it up.

We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.

For more information on our triage process please visit our triage overview

[radius-triage]

👋 @willtsai Thanks for filing this bug report.

A project maintainer will review this report and get back to you soon. If you'd like immediate help troubleshooting, please visit our Discord server.

For more information on our triage process please visit our triage overview

[willtsai]

another run that took 1+ hrs - https://github.com/willtsai/aks-store-demo/actions/runs/30581318817

[willtsai]

Investigate what caused the container image build part of the deployment workflow to take so long (1+ hrs) to fail and propose a fix for this to fail faster.

[sylvainsf]

Root cause analysis: where the multi-hour duration comes from

I researched two real runs of this app (willtsai/aks-store-demo, commit c9c1ca5), a comparable app, the in-cluster builder configuration, and the recipe that selects the build platform. The wall-clock is dominated by container image builds that run outside the Radius deployment boundary: cross-architecture builds under QEMU emulation, on a builder whose concurrency is deliberately capped, running on the CI runner's own resources. This writeup lays out the evidence and the boundary. It intentionally proposes no changes to Radius itself.

The two runs

Run	Duration	How it ended
30588288843	2h 4m	Radius.Compute/containerImages builds reached the deployment engine's per-resource operation timeout
30581318817	1h 24m	one image build crashed (cc SIGSEGV under emulation); the deploy returned once all operations were terminal

Every pre-deploy step (control-plane setup, state restore, environment creation) finished in under three minutes combined. Run rad commands is ~99% of the run, and inside it the radius-env.bicep deploy takes about 11 seconds while the app deploy takes the remaining 80 to 120 minutes.

What actually fills the time: emulated cross-arch builds

The failing image in run 2 is unambiguous:

> [linux/arm64 chef 2/3] RUN cargo install cargo-chef --locked
#14 397.4 error: linking with `cc` failed: signal: 11 (SIGSEGV) (core dumped)
process "/dev/.buildkit_qemu_emulator /bin/sh -c cargo install cargo-chef --locked" ... exit code: 101

That is the application's own multi-stage Rust Dockerfile (FROM rust:1.96 AS chef) being compiled for linux/arm64 through buildkit_qemu_emulator on an amd64 runner. Every native toolchain step (cargo, cc, the .NET builds in the other images) runs under emulation, roughly an order of magnitude slower than native, and the SIGSEGV is a known class of QEMU emulation instability, not an app bug (six sibling images built fine on the same run).

Why arm64 gets built at all, and why a near-identical app avoids it

The build platform is chosen by the Radius.Compute/containerImages recipe in resource-types-contrib. Its default is dual-arch. In Compute/containerImages/recipes/kubernetes/bicep/kubernetes-containerimages.bicep:

platforms: build.?platforms ?? [
  'linux/amd64'
  'linux/arm64'
]

If the app's build block omits platforms, the recipe builds for both amd64 and arm64 and passes that to BuildKit (build.sh: --opt "platform=$PLATFORMS"). The in-cluster BuildKit runs on the amd64 runner, so the amd64 half builds natively and the arm64 half runs under QEMU emulation.

That single field is the entire difference between the two apps:

• aks-store-demo: all 7 containerImages set only build.source, no platforms, so they inherit the dual-arch default and pay the arm64-under-QEMU cost. 80 to 120 minutes.
• eShop (Reshrahim/eShop), same recipe, same builder, 9 images: every containerImages sets platforms: ['linux/amd64'] explicitly. Single arch, matches the runner, no emulation. A successful deploy (run 29973541624) completed its Run rad commands step in about five minutes for nine images.

Same builder, more images, minutes instead of hours. The one material difference is that aks-store-demo builds arm64 under emulation and eShop does not, and that difference comes down to one line in each app's app.bicep.

Dual-arch is a legitimate requirement, not a mistake. Plenty of users target mixed or arm64 node pools (for example AKS arm64 pools) and genuinely need multi-arch images, so the recipe's dual-arch default is a reasonable portability choice, not a bug. The expensive part is specifically emulated cross-arch builds: producing arm64 on an amd64-only builder. A user who needs arm64 output still wants it; the question is whether it is produced natively (on an arm64 builder/node) or by emulation. Only the emulated path is slow, and choosing that path is a build-platform decision, not a Radius one.

The builder concurrency limit, measured

The in-cluster BuildKit that backs Radius.Compute/containerImages is configured by the Radius chart with max-parallelism = 1 by default (deploy/Chart/values.yaml -> dynamicrp.buildkit.maxParallelism, rendered into buildkitd.toml), and a sidecar resource cap of 2 CPU / 4Gi. The Repo Radius extension installs the control plane with only --set dynamicrp.buildkit.enabled=true and does not raise either value, so for these runs the image builds are serialized one worker step at a time, inside a 4Gi sidecar. Seven images, each a full multi-stage native-toolchain build, and for aks-store-demo each also doing an emulated arm64 pass, run essentially back to back.

That serialization is a property of the build backend's configuration, not of the app's dependency graph. In this app.bicep the seven containerImages each declare only dependsOn: [ registryCreds ], so they are independent siblings that Radius schedules concurrently; BuildKit then runs them one worker-step at a time because that is how the builder is sized.

Why raising the cap is not a free fix (runner pressure)

The obvious lever, raising maxParallelism, is a tradeoff bounded by the CI runner:

• The BuildKit sidecar runs inside the ephemeral control-plane cluster on the GitHub Actions runner itself, so build CPU and memory come out of the runner's fixed budget, capped further by the 4Gi sidecar limit.
• Running more native-toolchain compiles in parallel (doubly so emulated ones) multiplies memory and CPU pressure against that cap. BuildKit OOM under concurrent image builds is exactly what [Bug]: BuildKitd doesn't scale for multiple container image builds in dynamic-rp #12542 documented. maxParallelism=1 is a deliberate conservative default for that reason.
• Critically, consumers of the Radius extension may be running on standard, un-upgraded GitHub-hosted runners with fixed, limited CPU and memory. On such a runner, raising build parallelism to shorten wall-clock would push the sidecar into OOM/thrash rather than help. Any "just build them in parallel" mitigation has to account for the runner the user actually brings; it is not safe to assume larger runners.

So the two forces are in tension: emulation makes each build slow, and the runner-bounded 4Gi/serialized builder is what keeps memory pressure survivable. You cannot relieve the first by relaxing the second without a bigger runner.

Why a failure still takes the full time

Standard incremental-deployment semantics (the model ARM/Bicep use): independent resources run concurrently and the deployment reports a result only once all operations reach a terminal state. So even a build that fails early does not short-circuit the deploy; the run waits for the serialized queue of remaining builds to drain. This is a property of the deployment model, executed correctly, not a defect.

The boundary

• Radius: builds the dependency graph from app.bicep, schedules independent resources concurrently and dependent ones in order, passes build.platforms through to the recipe, and reports a result when all operations are terminal. It did this correctly in both runs.
• Not Radius: the choice of build platform (the recipe default vs the app's platforms field) and the resulting QEMU emulation, the app's Dockerfiles/base images/toolchains, the BuildKit concurrency and resource sizing (chart/extension install values), and the CI runner those builds actually execute on.

Mitigations, all outside Radius

• Match the build platform to the builder. An app that only needs amd64 can set platforms: ['linux/amd64'] (as eShop does) and avoid emulation entirely. An app that genuinely needs arm64 output should build it natively — on an arm64 builder or node pool — rather than by emulation on an amd64 runner. This is incompatible with the Repo Radius worker.
• Have the AI Extension set platform. The AI extension can set platform on containerImages always to amd64.
• Bound the run at the harness. The issue's own workaround, a timeout-minutes on the workflow job or step, is the right place to cap wall-clock, since the deploy is driven by a CI workflow that owns its time budget.