feat(bench): add cancellation cascade benchmarks (Phase 2)#1117
Open
feat(bench): add cancellation cascade benchmarks (Phase 2)#1117
Conversation
Implements 'Cost of Correctness' cancellation benchmarks comparing: - effection-structured: automatic cancellation via scope exit - async+abort: manual AbortController threading - effect: Effect.js fiber interruption - rxjs: subscription teardown with takeUntil Key additions: - Resource tracker for correctness verification (allocated vs released) - Barrier for deterministic task synchronization - CLI --tasks/-t option for concurrent task count - Correctness stats in table and JSON output All implementations verify cleanup correctness (leaked = 0).
commit: |
- Fix TS2345 in scenario.ts by widening channel type to accept LegacyBenchmarkOptions - Remove unused Deferred.make from effect.cancellation.ts
Addresses PR feedback: 1. Typed scenario registry (eliminates fragile filename regex): - scenarios.ts now exports ScenarioEntry[] with explicit kind - bench.ts uses entry.kind instead of filename matching - Results grouped by kind from registry, not result.name.match() 2. Worker handshake validation (prevents silent deadlock): - runBenchmark validates event.kind matches expected - Fails fast with clear error on mismatch 3. O(1) correctness aggregation (reduces harness overhead): - Added tracker.addCounts(allocated, released) method - Replaced O(n) for-loop with O(1) bulk addition 4. Idiomatic RxJS takeUntil pattern: - Refactored to use pipe(takeUntil(cancel$), finalize(...)) - Comments now match implementation - Uses merge() for concurrent worker subscription
1. Prevent cancellation benchmark hangs when barrier never reached: - Add BarrierTimeoutError class for clear error identification - Add timeout parameter to createBarrier() (default: 30s) - Add abort() method for external error injection - Barrier rejects all pending waiters on timeout/abort 2. Surface worker messageerror events as hard failures: - Add spawn listener for worker.messageerrors in runBenchmark() - Throws with actionable context on serialization failure 3. Make include/exclude regex handling user-friendly: - Add validateRegex() helper with friendly error messages - Validate patterns early before any benchmark work - Use exit(1) for clean shutdown on validation errors - No stack trace noise for user input errors
Member
|
do we need to include LoC here to demonstrate the complexity of the asynchronous/abort code? One could look at these result and determine that you're better off because async/abort is faster and also correct (0 leaks). We need a benchmark that captures complexity. Another area for benchmarking is with error handling |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Implements Phase 2 of the benchmark analysis plan (issue #1112): "Cost of Correctness" benchmarks that demonstrate Effection's value proposition for cancellation guarantees.
The goal is to show potential adopters:
Approach
New Cancellation Benchmark Scenario
Spawns N concurrent worker tasks, each with nested sub-tasks up to depth D. Each task allocates a tracked "resource" and registers cleanup. Then cancels all tasks and verifies cleanup.
Four Implementations Compared
scoped()exittakeUntil+finalize)Benchmark Results
Small Scale (10 tasks × 5 depth = 50 resources)
Medium Scale (50 tasks × 10 depth = 500 resources)
Large Scale (100 tasks × 20 depth = 2000 resources)
Key Findings
Implementation Details
Core Files
tasks/bench/scenarios/cancellation/tracker.ts- Resource tracker + barrier with timeouttasks/bench/scenarios/cancellation/scenario.ts- Cancellation scenario helpertasks/bench/scenarios/cancellation/*.cancellation.ts- Four implementationstasks/bench.ts- CLI with--tasks/-toption, correctness stats, regex validationtasks/bench/types.ts-BenchmarkKind,CorrectnessStats,ScenarioEntrytasks/bench/scenarios.ts- Typed scenario registryHarness Reliability Improvements
--include/--excludepatternsUsage
Commits in this PR
feat(bench): add cancellation cascade benchmarks (Phase 2)- Core implementationfix: address PR feedback- TS2345 fix, remove unused Deferredstyle: fix formatting- deno fmtrefactor(bench): typed scenario registry and improved correctness- Eliminate filename regex, O(1) aggregation, idiomatic RxJSfix(bench): improve harness reliability- Barrier timeout, messageerror handling, regex validationCloses part of #1112 (Phase 2)