Skip to content

feat(bench): add cancellation cascade benchmarks (Phase 2)#1117

Open
taras wants to merge 5 commits intov4from
bench-cancellation-cascade
Open

feat(bench): add cancellation cascade benchmarks (Phase 2)#1117
taras wants to merge 5 commits intov4from
bench-cancellation-cascade

Conversation

@taras
Copy link
Member

@taras taras commented Feb 18, 2026

Motivation

Implements Phase 2 of the benchmark analysis plan (issue #1112): "Cost of Correctness" benchmarks that demonstrate Effection's value proposition for cancellation guarantees.

The goal is to show potential adopters:

  1. How Effection's structured concurrency automatically handles cancellation
  2. The performance cost vs manual approaches (the "cost of correctness")
  3. That all implementations correctly clean up resources (leaked = 0)

Approach

New Cancellation Benchmark Scenario

Spawns N concurrent worker tasks, each with nested sub-tasks up to depth D. Each task allocates a tracked "resource" and registers cleanup. Then cancels all tasks and verifies cleanup.

Four Implementations Compared

Library Approach Ergonomics
effection-structured Automatic cancellation via scoped() exit Best - no manual wiring
async+abort Manual AbortController threading Verbose, error-prone
effect Effect.js fiber interruption Good, but heavier runtime
rxjs Subscription teardown (idiomatic takeUntil + finalize) Fast, but manual management

Benchmark Results

Small Scale (10 tasks × 5 depth = 50 resources)

Library Avg (ms) Correctness
effection-structured 2.3 ✅ 0 leaked
async+abort 0.5 ✅ 0 leaked
effect 4.7 ✅ 0 leaked
rxjs 0.3 ✅ 0 leaked

Medium Scale (50 tasks × 10 depth = 500 resources)

Library Avg (ms) Correctness
effection-structured 21.9 ✅ 0 leaked
async+abort 4.2 ✅ 0 leaked
effect 45.0 ✅ 0 leaked
rxjs 1.2 ✅ 0 leaked

Large Scale (100 tasks × 20 depth = 2000 resources)

Library Avg (ms) Correctness
effection-structured 85.0 ✅ 0 leaked
async+abort 59.2 ✅ 0 leaked
effect 297.9 ✅ 0 leaked
rxjs 6.8 ✅ 0 leaked

Key Findings

  1. Effection beats Effect.js for cancellation: 2-3.5x faster at all scales
  2. Cost of correctness: ~5x overhead vs manual abort at small scale, converging to ~1.5x at large scale
  3. All implementations verify cleanup: leaked = 0 across all runs
  4. RxJS is fastest but requires manual subscription management

Implementation Details

Core Files

  • tasks/bench/scenarios/cancellation/tracker.ts - Resource tracker + barrier with timeout
  • tasks/bench/scenarios/cancellation/scenario.ts - Cancellation scenario helper
  • tasks/bench/scenarios/cancellation/*.cancellation.ts - Four implementations
  • tasks/bench.ts - CLI with --tasks/-t option, correctness stats, regex validation
  • tasks/bench/types.ts - BenchmarkKind, CorrectnessStats, ScenarioEntry
  • tasks/bench/scenarios.ts - Typed scenario registry

Harness Reliability Improvements

  • Barrier timeout: Prevents hangs if workers fail before reaching barrier (30s default)
  • Worker messageerror handling: Surfaces serialization failures as hard errors
  • Regex validation: User-friendly errors for invalid --include/--exclude patterns
  • Typed scenario registry: Eliminates fragile filename-based classification
  • Kind validation: Worker response validation prevents silent mismatches

Usage

# Run only cancellation benchmarks
deno task bench --include cancellation -t 50 -d 10

# Full suite with JSON output
deno task bench --json -t 50 -d 100

# Test invalid regex handling (should fail gracefully)
deno task bench --include "+" --repeat 1
# error: Invalid --include pattern "+": Invalid regular expression: /+/: Nothing to repeat

Commits in this PR

  • feat(bench): add cancellation cascade benchmarks (Phase 2) - Core implementation
  • fix: address PR feedback - TS2345 fix, remove unused Deferred
  • style: fix formatting - deno fmt
  • refactor(bench): typed scenario registry and improved correctness - Eliminate filename regex, O(1) aggregation, idiomatic RxJS
  • fix(bench): improve harness reliability - Barrier timeout, messageerror handling, regex validation

Closes part of #1112 (Phase 2)

Implements 'Cost of Correctness' cancellation benchmarks comparing:
- effection-structured: automatic cancellation via scope exit
- async+abort: manual AbortController threading
- effect: Effect.js fiber interruption
- rxjs: subscription teardown with takeUntil

Key additions:
- Resource tracker for correctness verification (allocated vs released)
- Barrier for deterministic task synchronization
- CLI --tasks/-t option for concurrent task count
- Correctness stats in table and JSON output

All implementations verify cleanup correctness (leaked = 0).
@pkg-pr-new
Copy link

pkg-pr-new bot commented Feb 18, 2026

Open in StackBlitz

npm i https://pkg.pr.new/thefrontside/effection@1117

commit: df0568d

- Fix TS2345 in scenario.ts by widening channel type to accept LegacyBenchmarkOptions
- Remove unused Deferred.make from effect.cancellation.ts
Addresses PR feedback:

1. Typed scenario registry (eliminates fragile filename regex):
   - scenarios.ts now exports ScenarioEntry[] with explicit kind
   - bench.ts uses entry.kind instead of filename matching
   - Results grouped by kind from registry, not result.name.match()

2. Worker handshake validation (prevents silent deadlock):
   - runBenchmark validates event.kind matches expected
   - Fails fast with clear error on mismatch

3. O(1) correctness aggregation (reduces harness overhead):
   - Added tracker.addCounts(allocated, released) method
   - Replaced O(n) for-loop with O(1) bulk addition

4. Idiomatic RxJS takeUntil pattern:
   - Refactored to use pipe(takeUntil(cancel$), finalize(...))
   - Comments now match implementation
   - Uses merge() for concurrent worker subscription
1. Prevent cancellation benchmark hangs when barrier never reached:
   - Add BarrierTimeoutError class for clear error identification
   - Add timeout parameter to createBarrier() (default: 30s)
   - Add abort() method for external error injection
   - Barrier rejects all pending waiters on timeout/abort

2. Surface worker messageerror events as hard failures:
   - Add spawn listener for worker.messageerrors in runBenchmark()
   - Throws with actionable context on serialization failure

3. Make include/exclude regex handling user-friendly:
   - Add validateRegex() helper with friendly error messages
   - Validate patterns early before any benchmark work
   - Use exit(1) for clean shutdown on validation errors
   - No stack trace noise for user input errors
@cowboyd
Copy link
Member

cowboyd commented Feb 18, 2026

do we need to include LoC here to demonstrate the complexity of the asynchronous/abort code? One could look at these result and determine that you're better off because async/abort is faster and also correct (0 leaks). We need a benchmark that captures complexity.

Another area for benchmarking is with error handling

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants