-
Notifications
You must be signed in to change notification settings - Fork 5
feat: improve upload management: FIFO slicing + session-aware adaptiv… #195
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: feature/improve-upload-management
Are you sure you want to change the base?
feat: improve upload management: FIFO slicing + session-aware adaptiv… #195
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces centralized upload slicing management to enforce FIFO ordering and bounded concurrency, plus session-aware adaptive parameter reset functionality. The implementation aims to prevent parallel slicer contention and ensure deterministic upload behavior under load.
- Adds
UploadSlicingManager
with FIFO queue processing and configurable concurrency limits - Implements session-aware reset functionality in
AdaptiveUploadController
to clear state on session changes - Updates dependency injection and factory patterns to wire the new components
Reviewed Changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.
Show a summary per file
File | Description |
---|---|
src/ByteSync.Client/Services/Communications/Transfers/Uploading/UploadSlicingManager.cs |
New centralized slicing manager with Channel-based FIFO queue and bounded worker concurrency |
src/ByteSync.Client/Services/Communications/Transfers/Uploading/AdaptiveUploadController.cs |
Adds session service subscription for state reset and improved downscaling logic |
src/ByteSync.Client/Services/Communications/Transfers/Uploading/FileUploadProcessor.cs |
Integrates slicing manager to enqueue slicing operations instead of direct execution |
src/ByteSync.Client/Services/Communications/Transfers/Uploading/FileSlicer.cs |
Adds backpressure control based on pending slice count vs upload parallelism |
src/ByteSync.Client/DependencyInjection/Modules/SingletonsModule.cs |
Registers new components as singletons in DI container |
src/ByteSync.Client/Factories/FileUploadProcessorFactory.cs |
Updates factory to resolve and inject slicing manager |
tests/ByteSync.Client.Tests/Services/Communications/Transfers/Uploading/*Tests.cs |
Updates unit tests for new slicing manager integration and adaptive method usage |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
src/ByteSync.Client/Services/Communications/Transfers/Uploading/UploadSlicingManager.cs
Outdated
Show resolved
Hide resolved
src/ByteSync.Client/Services/Communications/Transfers/Uploading/UploadSlicingManager.cs
Outdated
Show resolved
Hide resolved
_workers.Add(Task.Run(async () => | ||
{ | ||
await foreach (var work in _queue.Reader.ReadAllAsync()) | ||
{ | ||
try | ||
{ | ||
await work(); | ||
} | ||
catch (Exception ex) | ||
{ | ||
_logger.LogError(ex, "Slicing worker error"); | ||
} | ||
} | ||
})); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Worker tasks are started with Task.Run
but never awaited or tracked for completion. If workers fail or the channel completes, they won't be restarted. Consider monitoring worker health and restarting failed workers.
Copilot uses AI. Check for mistakes.
src/ByteSync.Client/Services/Communications/Transfers/Uploading/AdaptiveUploadController.cs
Show resolved
Hide resolved
src/ByteSync.Client/Services/Communications/Transfers/Uploading/FileSlicer.cs
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good job 👍 Here are a few comments
src/ByteSync.Client/Services/Communications/Transfers/Uploading/FileUploadProcessor.cs
Outdated
Show resolved
Hide resolved
|
||
public interface IUploadSlicingManager | ||
{ | ||
Task Enqueue(Func<Task> startSlicingAsync); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note: the signature might be: Task<UploadProgressState> Enqueue(SharedFileDefinition sharedFileDefinition).
UploadSlicingManager would then instanciate the FileSlicer, the UploadProgressState and return the UploadProgressState.
src/ByteSync.Client/Services/Communications/Transfers/Uploading/AdaptiveUploadController.cs
Show resolved
Hide resolved
src/ByteSync.Client/Services/Communications/Transfers/Uploading/UploadSlicingManager.cs
Outdated
Show resolved
Hide resolved
src/ByteSync.Client/Services/Communications/Transfers/Uploading/UploadSlicingManager.cs
Show resolved
Hide resolved
src/ByteSync.Client/Services/Communications/Transfers/Uploading/FileSlicer.cs
Outdated
Show resolved
Hide resolved
|
Summary
Introduces a centralized slicing queue to enforce FIFO ordering and bounded concurrency across uploads, and resets adaptive upload parameters on session changes.
Key changes
UploadSlicingManager
: AddsIUploadSlicingManager
andUploadSlicingManager
using aChannel<Func<Task>>
, FIFO processing, and max concurrency of 2; ignores stale work after session resets.AdaptiveUploadController
subscribes toISessionService.SessionObservable
and fully resets chunk size, parallelism, and window.AdaptiveUploadController
andUploadSlicingManager
inSingletonsModule
; resolves and injectsIUploadSlicingManager
inFileUploadProcessorFactory
.FileUploadProcessor
enqueues slicing viaIUploadSlicingManager
instead of starting it directly to ensure ordered execution.SliceAndEncryptAdaptiveAsync
, injects and stubsIUploadSlicingManager
, and wiresISessionService
mock for the controller.Rationale
Notes
MAX_CONCURRENT_SLICES
is set to 2 and can be tuned later.