chore: adjust retries to fail faster

[Despire]

Completely removed retries for failed autoscaled operations and reduced the retries in terraformer and other tasks to speed up the failure process.

Reordered tasks to unblock adding a new nodepool if scale-up continuously fails and blocks other tasks.

Summary by CodeRabbit

• New Features

  • Added an explicit option to disable retries for kubectl operations.

• Improvements

  • Reduced retry attempts for several command paths to make operations fail faster when appropriate.
  • Reworked nodepool addition scheduling for more accurate control-plane handling and related restarts.
  • Adjusted autoscaled nodepool recovery behavior for quicker reset on scale-up failures.
  • Updated component image tags.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 37ef0fae-5e94-4ce2-be2f-a5b158106059

📥 Commits

Reviewing files that changed from the base of the PR and between 9ba3850 and f045424.

📒 Files selected for processing (2)

• manifests/claudie/kustomization.yaml
• manifests/testing-framework/kustomization.yaml

✅ Files skipped from review due to trivial changes (1)

• manifests/claudie/kustomization.yaml

---

Walkthrough

Adds an explicit kubectl.NoRetries constant, replaces magic -1 retry values with it across services, tunes retry counts for Cilium and Tofu, lowers an autoscaler failure threshold, refactors nodepool addition flows, and updates kustomize image tags.

Changes

Retry behavior standardization and nodepool scheduling refactor

Layer / File(s)	Summary
Introduce NoRetries constant internal/kubectl/kubectl.go	Exported constant NoRetries = -1 replaces implicit magic number usage for disabling retry behavior in kubectl configuration.
Standardize kubectl NoRetries usage services/kube-eleven/internal/worker/service/task_reconcile_infrastructure.go, services/kuber/internal/worker/service/task_delete_nodes.go, services/kuber/internal/worker/service/task_patch_kubeadm_cm.go, services/manager/internal/service/healthcheck.go, services/manager/internal/service/managementcluster/delete_kubeconfig.go, services/manager/internal/service/managementcluster/delete_metadata.go, services/manager/internal/service/managementcluster/internal/autoscaler/cluster_autoscaler.go, services/manager/internal/service/managementcluster/secret.go	Ten kubectl client configurations across kube-eleven, kuber, and manager services are updated to use kubectl.NoRetries instead of -1 for explicit retry disabling.
Cilium rollout restart retry budget adjustment services/kuber/internal/worker/service/task_cilium_rollout_restart.go	Cilium daemonset rollout restart reduces retry attempts from 5 to 3.
Tofu command retry count reduction services/terraformer/internal/worker/service/internal/tofu/terraform.go	Maximum retry count for Tofu/terraformer commands is reduced from 3 to 1 across command retry helpers.
Autoscaler scale-up failure counter threshold services/manager/internal/service/reconciliation.go	Autoscaled nodepool TargetSize reset now triggers when scale-up failure counter > 0 instead of > 2.
Nodepool addition and partial-addition flow refactoring services/manager/internal/service/reconciliate_kubernetes.go	diff.Added and diff.PartiallyAdded handling reordered and split with distinct labels; per-nodepool prep may append kubeadm certSAN patch and Cilium restart for control-plane additions, longhorn CA enablement added conditionally, longhorn storage classes reconciled after adds, and task deltas/descriptions adjusted to include node counts.
Manifest image tag updates manifests/claudie/kustomization.yaml, manifests/testing-framework/kustomization.yaml	Bump ghcr.io/berops/claudie/* and testing-framework image tags to cd82725-4239.

Possibly related PRs

• berops/claudie#2069: Related changes to autoscaled nodepool scale-up failure handling in reconciliation.go.
• berops/claudie#2038: Related to isControlNode and node deletion flow that uses kubectl retry configuration.

Suggested labels

test-set-autoscaling

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adjusting retry configurations across multiple services to fail faster, which aligns with the PR's core objective of reducing retries and speeding up failure detection.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch chore/tweak-retries

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[bernardhalas]

LGTM, thanks