Describe the enhancement
This enhancement covers a series of targeted improvements to the Rust-based scheduler components to address scaling bottlenecks, race conditions, and architectural complexity.
1. Lock contention in HostCache::hosts_index
Where: src/host_cache/cache.rs:60 : RwLock<BTreeMap<CoreKey, BTreeMap<MemoryKey, HashSet>>>.
Problem: Every check_out takes a read lock to scan, but on a successful match upgrades to a write lock at line 262 to remove from the index. Every check_in takes a write lock. With concurrent job processing, all operations serialize through this lock.
Fix options:
- Sharded mutex per CoreKey bucket (e.g., parking_lot::RwLock<BTreeMap<CoreKey, Mutex>>).
- scc::TreeIndex for lock-free reads and optimistic writes.
2. Layer permit race
Where: src/pipeline/matcher.rs:116-156
Problem: Layers are queried before permits are requested, leading to wasted DB work when multiple schedulers race for the same layer.
Fix:
- Use SELECT ... FOR UPDATE SKIP LOCKED on the layer row. This pushes deduplication into Postgres and allows the removal of the LayerPermitService actor.
3. Redundant outer transaction
Where: src/pipeline/dispatcher/actor.rs:96-120
Problem: Handler opens a transaction that wraps gRPC calls and advisory locks, while inner per-frame transactions do the actual work.
Fix: Remove the outer transaction. Move the host advisory lock into a tiny transaction or into the per-proc transaction to ensure it is released immediately after use.
4. reserved_hosts 10s TTL
Where: src/host_cache/actor.rs:53-67
Problem: Dispatches can take >10s if RQD is slow. If the reservation expires, the host is marked available, leading to duplicate dispatch attempts and rollbacks.
Fix: Drop the short timer. Use a strict check_out/check_in pair. Use a much longer safety TTL (e.g., 5 min) only for cleaning up leaked tasks.
5. Drop the actor layer
Where: HostCacheService, LayerPermitService, and RqdDispatcherService.
Problem: These services use Actix actors despite having no state mutation requirements that justify the overhead of mailboxes, heap-allocated message envelopes, and round-trip latency.
Fix: Refactor to standard #[derive(Clone)] structs with Arc-wrapped internals. Use tokio::spawn for background tasks.
6. panic! on host-cache DB failure
Where: src/pipeline/matcher.rs:402
Problem: Transient query failures take down the entire process, trashing in-flight dispatches.
Fix: Implement a circuit breaker pattern (Healthy/Degraded states) with exponential backoff. Exit cleanly only after sustained failure.
7. Dead metrics
Where: pipeline/matcher.rs:46-47 and cluster.rs:36
Problem: Atomic counters like HOSTS_ATTEMPTED are updated but never exported to Prometheus.
Fix: Properly register these as Prometheus counters in metrics/mod.rs or delete them if unused.
Describe the enhancement
This enhancement covers a series of targeted improvements to the Rust-based scheduler components to address scaling bottlenecks, race conditions, and architectural complexity.
1. Lock contention in HostCache::hosts_index
Where: src/host_cache/cache.rs:60 : RwLock<BTreeMap<CoreKey, BTreeMap<MemoryKey, HashSet>>>.
Problem: Every check_out takes a read lock to scan, but on a successful match upgrades to a write lock at line 262 to remove from the index. Every check_in takes a write lock. With concurrent job processing, all operations serialize through this lock.
Fix options:
2. Layer permit race
Where: src/pipeline/matcher.rs:116-156
Problem: Layers are queried before permits are requested, leading to wasted DB work when multiple schedulers race for the same layer.
Fix:
3. Redundant outer transaction
Where: src/pipeline/dispatcher/actor.rs:96-120
Problem: Handler opens a transaction that wraps gRPC calls and advisory locks, while inner per-frame transactions do the actual work.
Fix: Remove the outer transaction. Move the host advisory lock into a tiny transaction or into the per-proc transaction to ensure it is released immediately after use.
4. reserved_hosts 10s TTL
Where: src/host_cache/actor.rs:53-67
Problem: Dispatches can take >10s if RQD is slow. If the reservation expires, the host is marked available, leading to duplicate dispatch attempts and rollbacks.
Fix: Drop the short timer. Use a strict check_out/check_in pair. Use a much longer safety TTL (e.g., 5 min) only for cleaning up leaked tasks.
5. Drop the actor layer
Where: HostCacheService, LayerPermitService, and RqdDispatcherService.
Problem: These services use Actix actors despite having no state mutation requirements that justify the overhead of mailboxes, heap-allocated message envelopes, and round-trip latency.
Fix: Refactor to standard #[derive(Clone)] structs with Arc-wrapped internals. Use tokio::spawn for background tasks.
6. panic! on host-cache DB failure
Where: src/pipeline/matcher.rs:402
Problem: Transient query failures take down the entire process, trashing in-flight dispatches.
Fix: Implement a circuit breaker pattern (Healthy/Degraded states) with exponential backoff. Exit cleanly only after sustained failure.
7. Dead metrics
Where: pipeline/matcher.rs:46-47 and cluster.rs:36
Problem: Atomic counters like HOSTS_ATTEMPTED are updated but never exported to Prometheus.
Fix: Properly register these as Prometheus counters in metrics/mod.rs or delete them if unused.