Skip to content

Conversation

@msrathore-db
Copy link
Collaborator

@msrathore-db msrathore-db commented Nov 17, 2025

🥞 Stacked PR

Use this link to review incremental changes.


Summary

Adds comprehensive test coverage for the straggler mitigation feature, validating both the implementation and edge cases.

E2E Tests (19 tests):

  • ✅ Slow download identification and cancellation
  • ✅ Fast downloads not marked as stragglers
  • ✅ Minimum completion quantile requirement
  • ✅ Sequential fallback activation after threshold
  • ✅ Sequential mode enforcement (one download at a time)
  • ✅ No stragglers detected in sequential mode
  • ✅ Sequential fallback applies only to current batch
  • ✅ Monitoring thread respects cancellation
  • ✅ Parallel mode respects max parallel downloads
  • ✅ Cancelled straggler retry logic
  • ✅ Mixed speed download scenarios
  • ✅ Clean shutdown during monitoring

Advanced Unit Tests:

  • ✅ Duplicate detection prevention across multiple cycles
  • ✅ CancellationTokenSource atomic replacement
  • ✅ Concurrent CTS cleanup
  • ✅ Cleanup behavior under exceptions
  • ✅ Shutdown cancellation respect
  • ✅ Counter overflow protection (long type)
  • ✅ Concurrent modification safety

Test Coverage:

  • Basic functionality validation
  • Edge case handling
  • Thread safety verification
  • Resource cleanup verification
  • Cancellation token management

CloudFetchDownloader Integration:
- Background monitoring thread (runs every 5 seconds)
- Per-file CancellationTokenSource for clean cancellation
- Automatic retry mechanism for cancelled stragglers
- Metrics tracking for all active downloads
- Sequential fallback mode support
- Thread-safe metrics and cancellation token management

Testing:
- 19 comprehensive E2E tests covering various scenarios
- Advanced unit tests for CTS management, cleanup, counter overflow, and concurrency
- Tests for slow download detection, sequential fallback, and cancellation respect

Builds on: stack/straggler-detector
@msrathore-db msrathore-db force-pushed the stack/straggler-integration branch from 2b653d7 to 790b0b4 Compare November 17, 2025 20:54
@msrathore-db msrathore-db changed the title feat(csharp): integrate straggler mitigation into CloudFetchDownloader test(csharp): add comprehensive tests for straggler mitigation Nov 17, 2025
@msrathore-db msrathore-db changed the base branch from stack/straggler-detector to stack/straggler-downloader-impl November 17, 2025 21:05
@msrathore-db
Copy link
Collaborator Author

Closing this PR. Recreating with correct branch reference.
Old head: stack/straggler-integration
New head: stack/straggler-tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants