Allow clusterless jobs

Fournos_Design_Document.md

@@ -117,11 +117,29 @@ All jobs flow through Kueue — there is one scheduling path with different cons
 | `cluster` + `hardware` (default: exclusive)          | `fournos.dev/cluster: cluster-1` + GPU request | Locks the cluster (100 cluster-slots) and requests specific GPUs.       |
 | `exclusive: false` + `cluster` + `hardware`          | `fournos.dev/cluster: cluster-1` + GPU request | Shared access (1 cluster-slot) with specific hardware on a specific cluster. |
 | `exclusive: false` + `hardware` (no cluster)         | *(none)*                                       | All flavors with enough GPU quota are eligible. Kueue picks first fit.  |
+| `clusterless: true` (no cluster)                     | *(none)*                                       | Bypasses Kueue entirely. Runs on hub cluster without target cluster access. |
 
 
 Each ResourceFlavor has `spec.nodeLabels: { fournos.dev/cluster: <name> }`. When `cluster` is specified, the operator sets a matching `nodeSelector` on the Workload's podSet template so Kueue constrains admission to that flavor.
 
-Because `exclusive` defaults to `true`, any job with `spec.cluster` automatically locks the assigned cluster — including jobs that also specify `spec.hardware`. To get shared access (multiple jobs on the same cluster), set `exclusive: false` explicitly. Jobs without `spec.cluster` must set `exclusive: false` (since exclusive mode requires a cluster target). Hardware is always required for non-exclusive jobs — only exclusive cluster-locked jobs may omit it.
+Because `exclusive` defaults to `true`, any job with `spec.cluster` automatically locks the assigned cluster — including jobs that also specify `spec.hardware`. To get shared access (multiple jobs on the same cluster), set `exclusive: false` explicitly. Jobs without `spec.cluster` must set `exclusive: false` (since exclusive mode requires a cluster target). Hardware is always required for non-exclusive jobs — only exclusive cluster-locked jobs and clusterless jobs (`clusterless: true`) may omit it.
+
+### Clusterless Jobs
+
+Clusterless jobs (`clusterless: true`) bypass Kueue entirely and run on the hub cluster without target cluster access. They skip the Pending phase and go directly from Resolving to Admitted.
+
+**Lifecycle:** `Resolving → Admitted → Running → Succeeded/Failed`
+
+**Benefits:**
+- No Kueue admission delay
+- No target cluster resource contention
+- Ideal for CI/validation workflows
+
+**Restrictions:**
+- Cannot combine with `cluster` specification
+- Must use `exclusive: false`
+- Cannot use `lockOnly: true`
+- No kubeconfig passed to execution environment
 
 ### Exclusive cluster locking
 

Makefile

@@ -8,7 +8,7 @@ FOURNOS_WORKLOAD_NAMESPACE     ?= fournos-local-dev
 FOURNOS_SECRETS_NAMESPACE      ?= psap-secrets
 
 .PHONY: lint format test docker-build docker-push \
-        install deploy dev-setup dev-run dev-teardown \
+        install deploy dev-setup dev-run dev-teardown wip-run \
         ci-setup ci-run ci-stop
 
 ##@ Code Quality
@@ -87,6 +87,14 @@ dev-run:
 	FOURNOS_RESOLVE_JOB_TEMPLATE=dev/mock-resolve/resolve_job.yaml \
 	$(VENV_BIN)python -m fournos
 
+wip-run:
+	FOURNOS_GC_INTERVAL_SEC=5 \
+	FOURNOS_CONTROLLER_NAMESPACE=$(FOURNOS_CONTROLLER_NAMESPACE) \
+	FOURNOS_WORKLOAD_NAMESPACE=psap-automation-wip \
+	FOURNOS_SECRETS_NAMESPACE=psap-secrets \
+	FOURNOS_RESOLVE_JOB_TEMPLATE=config/forge/resolve_job.yaml \
+	$(VENV_BIN)python -m fournos
+
 dev-teardown:
 	KIND_EXPERIMENTAL_PROVIDER=$(KIND_EXPERIMENTAL_PROVIDER) kind delete cluster --name $(KIND_CLUSTER_NAME)
 

README.md

@@ -87,17 +87,23 @@ oc delete FournosJob -n $FOURNOS_WORKLOAD_NAMESPACE <name>      # cleanup
 | `spec.priority` | no | Kueue WorkloadPriorityClass name |
 | `spec.secretRefs` | no | Vault-synced K8s Secret names (prefixed with `vault-`) to mount into the pipeline. Populated by the execution engine during the Resolving phase. The operator validates each name in `FOURNOS_SECRETS_NAMESPACE`, copies the secrets into the operator namespace, and mounts them as a projected volume at `/var/run/secrets/fournos/<entry-name>/`. |
 | `spec.exclusive` | no (default `true`) | If `true`, locks the target cluster so no other FournosJob can run there. Requires `spec.cluster`. Hardware is optional — when omitted the Workload only requests cluster-slot resources for locking. |
+| `spec.clusterless` | no (default `false`) | If `true`, runs without cluster access — no kubeconfig is passed to the execution environment. Cannot be combined with `cluster` or `exclusive: true`. Hardware specifications are optional — if omitted, the job runs on hub cluster resources without Kueue hardware scheduling. |
 | `spec.shutdown` | no | Shutdown action: `Stop` cancels gracefully (Tekton `CancelledRunFinally` — runs `finally` tasks); `Terminate` cancels immediately (Tekton `Cancelled` — skips `finally` tasks). Both wait for the PipelineRun to finish before releasing Kueue quota. |
 
-\* `spec.hardware` is required unless the job uses exclusive cluster locking
-(`exclusive: true` + `cluster`), in which case it may be omitted — the
-Workload only needs cluster-slot resources. Every job passes through the
-Resolving phase where the execution engine populates `spec.hardware` (if
-not already set) and `spec.secretRefs` directly on the FournosJob. Since `exclusive` defaults
-to `true`, any job with `spec.cluster` locks the cluster exclusively —
-including jobs that also specify `spec.hardware`. Set `exclusive: false` for
-shared access (hardware is then required). Jobs without `spec.cluster` must
-set `exclusive: false`.
+\* `spec.hardware` is optional for all jobs except those requiring explicit GPU 
+scheduling. Hardware specifications are handled as follows:
+- **Exclusive cluster jobs** (`exclusive: true` + `cluster`): Hardware may be 
+  omitted since the Workload only needs cluster-slot resources for locking.
+- **Non-exclusive jobs** (`exclusive: false`): Hardware is optional — when 
+  omitted, the Workload only requests cluster-slot resources for shared access.
+- **Clusterless jobs** (`clusterless: true`): Hardware is optional since jobs 
+  run on hub cluster resources without Kueue scheduling.
+Every job passes through the Resolving phase where the execution engine
+populates `spec.hardware` (if not already set) and `spec.secretRefs` directly
+on the FournosJob. Since `exclusive` defaults to `true`, any job with
+`spec.cluster` locks the cluster exclusively — including jobs that also specify
+`spec.hardware`. Set `exclusive: false` for shared access. Jobs without
+`spec.cluster` must set `exclusive: false`.
 
 ### Status
 
@@ -106,11 +112,53 @@ The operator writes status to `.status`:
 | Field | Description |
 |---|---|
 | `phase` | `Resolving` → `Pending` → `Admitted` → `Running` → `Succeeded` / `Failed` / `Stopping` → `Stopped` |
-| `cluster` | Cluster assigned by Kueue |
+| `cluster` | Cluster assigned by Kueue (or `[clusterless]` for clusterless jobs) |
 | `pipelineRun` | Name of the Tekton PipelineRun |
 | `dashboardURL` | Tekton Dashboard link (if configured) |
 | `message` | Error details on failure |
 
+## Clusterless Jobs
+
+Clusterless jobs (`clusterless: true`) run entirely on the hub cluster without
+access to target clusters. They bypass Kueue admission and skip the Pending
+phase for faster execution.
+
+### Examples
+
+**Basic clusterless job:**
+```yaml
+apiVersion: fournos.dev/v1
+kind: FournosJob
+metadata:
+  name: hub-only-job
+spec:
+  clusterless: true
+  exclusive: false  # Required
+  executionEngine:
+    forge:
+      project: "local/benchmark"
+      args: ["unit-test"]
+```
+
+### Clusterless Job Lifecycle
+
+```
+Resolving → Admitted → Running → Succeeded/Failed
+            ↑
+        Skips Pending (no Kueue admission)
+```
+
+**Benefits:**
+- ⚡ **Faster startup**: No Kueue admission delay
+- 🧹 **Resource efficient**: No target cluster resource blocking
+- 🎯 **Hub-focused**: Perfect for CI/validation workflows
+
+**Restrictions:**
+- Cannot combine with `cluster` specification
+- Must use `exclusive: false`
+- Cannot use `lockOnly: true`
+- No target cluster kubeconfig available to execution environment
+
 ## Local development
 
 Prerequisites: [Podman](https://podman.io/),

config/fournos-validation/workflows/tasks.yaml

@@ -6,6 +6,7 @@ spec:
   params:
     - name: kubeconfig-secret
       type: string
+      default: "fournos-clusterless-placeholder"
   steps:
     - name: check-connectivity
       image: bitnami/kubectl:latest
@@ -19,10 +20,17 @@ spec:
       script: |
         #!/usr/bin/env bash
         set -euo pipefail
+
+        if [[ "$(params.kubeconfig-secret)" == "fournos-clusterless-placeholder" ]]; then
+          echo "Running in clusterless mode - skipping cluster connectivity validation"
+          exit 0
+        fi
+
         echo "Validating connectivity to target cluster…"
         kubectl cluster-info
         echo "Target cluster is reachable."
   volumes:
     - name: kubeconfig
       secret:
         secretName: $(params.kubeconfig-secret)
+        optional: true

fournos/core/tekton.py

@@ -52,7 +52,7 @@ def create_pipeline_run(
         *,
         name: str,
         pipeline: str,
-        kubeconfig_secret: str,
+        kubeconfig_secret: str | None,
         resolved_secrets: list[ResolvedSecret],
         cluster: str,
         owner_ref: dict | None = None,
@@ -92,7 +92,12 @@ def create_pipeline_run(
                         "name": "fournos-workload-namespace",
                         "value": settings.workload_namespace,
                     },
-                    {"name": "kubeconfig-secret", "value": kubeconfig_secret},
+                    {
+                        "name": "kubeconfig-secret",
+                        "value": kubeconfig_secret
+                        if kubeconfig_secret
+                        else "fournos-clusterless-placeholder",
+                    },
                 ],
                 "workspaces": [
                     {

fournos/handlers/execution.py

@@ -153,26 +153,32 @@ def reconcile_admitted(spec, name, namespace, status, patch, body):
     conditions = list(status.get("conditions") or [])
 
     if pr is None:
-        cluster = status.get("cluster", "")
+        clusterless = spec.get("clusterless", False)
 
-        try:
-            kubeconfig_secret = ctx.registry.copy_kubeconfig_secret(
-                cluster, name, owner_ref(body)
-            )
-        except client.exceptions.ApiException as exc:
-            patch.status["phase"] = Phase.FAILED
-            patch.status["message"] = f"Failed to copy kubeconfig: {exc.reason}"
-            set_condition(
-                patch,
-                conditions,
-                COND_PIPELINE_RUN_READY,
-                "False",
-                "KubeconfigNotFound",
-                f"Failed to copy kubeconfig: {exc.reason}",
-            )
-            ctx.kueue.delete_workload(name)
-            logger.error("Job %s: kubeconfig copy failed: %s", name, exc)
-            return
+        if clusterless:
+            # For clusterless jobs, skip kubeconfig handling
+            kubeconfig_secret = None
+            cluster = ""
+        else:
+            cluster = status.get("cluster", "")
+            try:
+                kubeconfig_secret = ctx.registry.copy_kubeconfig_secret(
+                    cluster, name, owner_ref(body)
+                )
+            except client.exceptions.ApiException as exc:
+                patch.status["phase"] = Phase.FAILED
+                patch.status["message"] = f"Failed to copy kubeconfig: {exc.reason}"
+                set_condition(
+                    patch,
+                    conditions,
+                    COND_PIPELINE_RUN_READY,
+                    "False",
+                    "KubeconfigNotFound",
+                    f"Failed to copy kubeconfig: {exc.reason}",
+                )
+                ctx.kueue.delete_workload(name)
+                logger.error("Job %s: kubeconfig copy failed: %s", name, exc)
+                return
 
         secret_refs_raw = spec.get("secretRefs") or []
         try:
@@ -204,9 +210,12 @@ def reconcile_admitted(spec, name, namespace, status, patch, body):
                 cluster=cluster,
                 owner_ref=owner_ref(body),
             )
-            logger.info(
-                "Job %s: created PipelineRun for target cluster %s", name, cluster
-            )
+            if clusterless:
+                logger.info("Job %s: created PipelineRun in clusterless mode", name)
+            else:
+                logger.info(
+                    "Job %s: created PipelineRun for target cluster %s", name, cluster
+                )
         except client.exceptions.ApiException as exc:
             if exc.status != 409:
                 patch.status["phase"] = Phase.FAILED

fournos/handlers/lifecycle.py

@@ -51,6 +51,7 @@ def on_create(spec, name, namespace, status, patch, body):
     cluster = spec.get("cluster")
     exclusive = spec["exclusive"]
     lock_only = spec.get("lockOnly", False)
+    clusterless = spec.get("clusterless", False)
 
     if lock_only and not cluster:
         patch.status["phase"] = Phase.FAILED
@@ -64,6 +65,22 @@ def on_create(spec, name, namespace, status, patch, body):
         )
         return
 
+    if clusterless:
+        for cond, msg in [
+            (lock_only, "clusterless: true cannot be combined with lockOnly: true"),
+            (exclusive, "clusterless: true requires exclusive: false"),
+            (
+                cluster,
+                "clusterless: true cannot be combined with cluster specification",
+            ),
+        ]:
+            if cond:
+                patch.status["phase"] = Phase.FAILED
+                patch.status["message"] = msg
+                return
+        patch.status["phase"] = Phase.RESOLVING
+        patch.status["message"] = "Resolving job requirements"
+        return
     if exclusive and not cluster:
         patch.status["phase"] = Phase.FAILED
         patch.status["message"] = "exclusive: true requires 'cluster' to be set"

fournos/handlers/resolving.py

@@ -176,16 +176,14 @@ def _resolve_hardware(
 
     if not gpu_type or not gpu_count:
         if not exclusive_lock:
-            _resolve_failed(
-                patch,
-                conditions,
+            # Hardware is optional for non-exclusive jobs (including clusterless)
+            logger.info(
+                "Job %s: non-exclusive job without hardware — "
+                "Workload will only request cluster-slot resources",
                 name,
-                "No hardware requirements: spec.hardware not populated "
-                "after resolution",
-                reason="NoHardware",
-                cond_message="No hardware requirements found",
             )
-            return None, None
+            return None, 0
+
         logger.warning(
             "Job %s: exclusive cluster lock without hardware — "
             "Workload will only request cluster-slot resources "
@@ -333,6 +331,23 @@ def reconcile_resolving(spec, name, status, patch, body):
 
     if not _validate_secret_refs(spec, name, conditions, patch):
         return
-    _create_workload_and_transition(
-        spec, name, conditions, patch, body, gpu_type, gpu_count
-    )
+
+    clusterless = spec.get("clusterless", False)
+    if clusterless:
+        # Skip Kueue entirely for clusterless jobs - go directly to Admitted phase
+        patch.status["phase"] = Phase.ADMITTED
+        patch.status["cluster"] = "[clusterless]"
+        patch.status["message"] = "Clusterless job ready for execution"
+        set_condition(
+            patch,
+            conditions,
+            COND_WORKLOAD_ADMITTED,
+            "True",
+            "ClusterlessAdmitted",
+            "Clusterless job bypassed Kueue admission",
+        )
+        logger.info("Job %s: clusterless job bypassing Kueue, phase=Admitted", name)
+    else:
+        _create_workload_and_transition(
+            spec, name, conditions, patch, body, gpu_type, gpu_count
+        )

fournos/operator.py

@@ -43,6 +43,7 @@
 def startup(**_):
     logging.getLogger("fournos").setLevel(settings.log_level.upper())
     logging.getLogger("aiohttp.access").setLevel(logging.WARNING)
+    logging.getLogger("kopf.objects").setLevel(logging.WARNING)
 
     for line in BANNER.strip().splitlines():
         logger.info(line)

manifests/crd.yaml

@@ -115,6 +115,15 @@ spec:
                     Requires 'cluster' to be set.  Hardware is optional — when
                     omitted the Workload only requests cluster-slot resources
                     for locking.
+                clusterless:
+                  type: boolean
+                  default: false
+                  description: >-
+                    Run without cluster access - no kubeconfig is passed to the
+                    execution environment. Cannot be combined with 'cluster' or
+                    'exclusive: true'. Hardware specifications are optional -
+                    if omitted, the job runs on hub cluster resources without
+                    Kueue hardware scheduling.
                 lockOnly:
                   type: boolean
                   default: false

tests/forge/deploy/orchestration/ci.py

@@ -76,7 +76,7 @@ def main(ctx, project_source):
 
 @main.command()
 @click.pass_context
-@ci_lib.safe_ci_command
+@ci_lib.safe_ci_entrypoint
 def deploy(ctx):
     """Complete FOURNOS deployment with cleanup (clean slate deployment)."""
     return fournos_deploy.cleanup_and_deploy()