Harden task concurrency with a fixed worker pool and bounded job queue #472

Goal

Refactor the native zed task runtime so local concurrency is explicitly resource-bounded by construction. The current implementation already bounds parallel task groups by --jobs and uses a Mutex + Condvar command limiter, but it still creates scoped worker threads per parallel chunk rather than reusing a fixed worker pool.

We should move to a reusable scheduler that makes it impossible for task-graph growth to translate into uncontrolled local thread/process growth.

Current behavior

• TaskRunOptions.jobs is a positive concurrency budget; default is 1.
• zed task run ... --jobs N bounds concurrent task command processes.
• parallel task groups use thread::scope and spawn at most N scoped threads per chunk.
• CommandLimiter uses Mutex<usize> + Condvar to cap concurrent command execution.
• task execution identity/state is synchronized with Mutex<BTreeMap<...>> + Condvar, so duplicate in-flight task identities wait rather than independently re-run.

This is bounded today, but it is not yet a persistent worker-pool architecture.

Proposed architecture

Introduce one scheduler per task-run invocation with:

• a fixed number of reusable Rust worker threads;
• a bounded in-memory queue of runnable jobs;
• explicit backpressure when the queue is full;
• Mutex + Condvar (or an equivalently well-audited bounded primitive) around queue state;
• no ad hoc/naked thread::spawn in task execution paths outside the scheduler, except narrowly documented exceptional cases;
• a separate child-process concurrency budget/permit layer so worker count and expensive subprocess count remain independent resource controls;
• graceful scheduler shutdown and join semantics;
• deterministic error propagation/cancellation behavior;
• no deadlock when nested tasks enqueue additional work;
• queue capacity derived from worker count (for example a small multiple of jobs) rather than unbounded graph size.

Resource policy

The scheduler should make the following invariants testable:

1. live worker threads <= configured worker count;
2. live child processes <= configured process budget;
3. queued jobs <= configured queue capacity;
4. a large dependency graph cannot create O(graph_size) threads;
5. producers block/yield/fail in a defined way when the bounded queue is saturated;
6. duplicate task identities remain deduplicated and wait on the existing execution-state contract;
7. all workers terminate before TaskRuntime::run returns.

We should preserve explicit --jobs N behavior. If/when defaults become CPU-derived, use std::thread::available_parallelism() with a conservative fallback and keep the behavior documented and testable.

Implementation notes

Likely homes:

• src/task_runtime.rs: scheduler, queue, worker lifecycle, integration with dependency/group execution;
• src/task_cli.rs: preserve positive --jobs validation and user-facing contract;
• docs/task-runtime.md: document thread/process/queue bounds and backpressure semantics.

Prefer a small internal scheduler abstraction over bringing in a heavyweight async runtime solely for this purpose.

Acceptance criteria

• fixed-size reusable worker pool implemented;
• bounded queue implemented with explicit saturation/backpressure semantics;
• command/subprocess limit remains independently enforced;
• parallel groups and nested task invocation use the scheduler instead of per-chunk worker spawning;
• stress test with a large synthetic task graph proves worker-thread count stays bounded;
• stress test proves queue capacity is enforced;
• test proves child-process concurrency never exceeds its configured budget;
• test covers nested task submission without deadlock;
• test covers worker panic/error propagation and scheduler shutdown;
• docs updated with concurrency/resource invariants;
• repository policy prevents future unbounded task-path thread spawning.

A companion organization-policy issue will track the reusable concurrency rule for Zed repositories.