Resume transfers #12

codeSamuraii · 2025-09-22T20:34:41Z

File Transfer Backend Improvements

Switched from a Redis queue to Redis Streams for chunk management in the Store class, enabling more robust and resumable file transfers. Methods for chunk handling (put_chunk, get_next_chunk, get_chunk_by_range) now use streams instead of lists.
Added new models in lib/models.py (UploadProgress, DownloadProgress, ResumeInfo, and ClientState) to support progress tracking and client state management for resumable transfers.
Implemented methods in Store to save and retrieve upload/download progress and sender/receiver state, supporting transfer resumption and better error handling.
Improved cleanup logic in Store to reliably remove all transfer-related keys and prevent race conditions. [1] [2]

Testing and Dependency Updates

Refactored test client setup to use a new HTTPTestClient class with helper methods, improving test readability and maintainability. [1] [2] [3]
Updated test file generation logic for more accurate chunk sizing, and moved HTTP/WebSocket helpers to their respective client classes for better organization. [1] [2]

Codebase Cleanup

Removed unused callback and exception handling code from lib/callbacks.py to streamline the codebase.

- Switched from Redis lists to Redis Streams for chunk tracking - Added upload/download progress persistence and resumption - Implemented WebSocket /resume/{uid} endpoint for upload resumption - Added HTTP Range header support for partial downloads (206 responses) - Created resumption handler for managing transfer state - Updated JavaScript client to save/restore upload progress via localStorage - Added connection management for handling disconnections gracefully - Transfers now wait for peer reconnection within timeout window - All transfers are resumable by default (no configuration needed) Key components: - lib/store.py: Refactored to use Redis Streams with progress tracking - lib/stream_store.py: New module for stream operations (deprecated) - lib/transfer.py: Enhanced with resume_from and start_byte parameters - lib/resume.py: Handles resumption logic and state validation - lib/range_utils.py: Parses and validates HTTP Range headers - lib/connection.py: Manages peer states and reconnection windows - views/websockets.py: Added /resume endpoint for upload resumption - views/http.py: Enhanced with Range/partial content support - static/js/file-transfer.js: Added localStorage progress persistence 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

The stream_store.py module was created but never used - all functionality was integrated directly into store.py. Also removed backup file. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

Major changes: - Created Pydantic models (UploadProgress, DownloadProgress, TransferStates, ResumeInfo) for type safety and validation - Refactored Store class with specific methods instead of generic keyword-based approach - Simplified FileTransfer class by breaking down large methods into smaller, focused ones - Merged resume.py and connection.py functionality into new TransferManager class - Updated all references to use the new cleaner API Improvements: - Replaced enums with simple class constants for TransferStates - Added specific Redis methods (get_upload_progress, get_sender_state, etc.) - Improved code readability by removing deep nesting and complex conditionals - Removed unnecessary comments - code is now self-explanatory - Simplified test cases to focus on core functionality Note: Some range download tests need further work but core functionality is intact. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

views/websockets.py

- 416 in case of invalid Range header - Fast hash in upload key instead of ID - Fixing tests

Copilot

Pull Request Overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

views/websockets.py:1

The exception handler for waiting for receiver connection was removed, but the timeout handling remains. This could cause unhandled exceptions if errors other than TimeoutError occur during wait_for_client_connected().

import string

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-25T11:29:02Z

views/http.py

+    else:
+        await transfer.set_client_connected()


The else clause after the return statement is unnecessary. The code after the if block will only execute when the condition is false, making the else clause redundant.

Suggested change

else:

await transfer.set_client_connected()

await transfer.set_client_connected()

Copilot · 2025-09-25T11:29:03Z

static/js/file-transfer.js

+const BUFFER_THRESHOLD_MOBILE = CHUNK_SIZE_MOBILE * 16;
+const BUFFER_THRESHOLD_DESKTOP = CHUNK_SIZE_DESKTOP * 16;
+const MAX_HASH_SAMPLING = 2 * 1024**2;
+const DEBUG_LOGS = false;


DEBUG_LOGS is set to false in production code. Consider using an environment variable or build-time flag to control debug logging instead of hardcoding it.

Suggested change

const DEBUG_LOGS = false;

const DEBUG_LOGS = typeof window !== 'undefined' && window.DEBUG_LOGS === true;

Copilot · 2025-09-25T11:29:03Z

lib/transfer.py

-        except (ClientDisconnect, WebSocketDisconnect) as e:
-            self.error(f"△ Unexpected upload error: {e}")
-            await self.store.put_in_queue(self.DEAD_FLAG)
+                if chunk_count % 1 == 0 or bytes_uploaded % (16 * 1024) == 0:


The condition chunk_count % 1 == 0 will always be true since any integer modulo 1 equals 0. This makes the entire condition equivalent to just the second part. Either remove this redundant check or use a meaningful modulo value like chunk_count % 10 == 0.

Suggested change

if chunk_count % 1 == 0 or bytes_uploaded % (16 * 1024) == 0:

if chunk_count % 10 == 0 or bytes_uploaded % (16 * 1024) == 0:

Copilot · 2025-09-25T11:29:04Z

tests/helpers.py

 def generate_test_file(size_in_kb: int = 10) -> tuple[bytes, FileMetadata]:
    """Generates a test file with specified size in KB."""
-    chunk_generator = ((letter * 1024).encode() for letter in chain.from_iterable(repeat(ascii_letters)))
-    content = b''.join(next(chunk_generator) for _ in range(size_in_kb))
+    chunk_generator = ((str(letter) * 32).encode() for letter in chain.from_iterable(repeat(ascii_letters)))
+    content = b''.join(next(chunk_generator) for _ in range(size_in_kb * 1024 // 32))


The magic number 32 appears twice without explanation. Consider defining it as a constant with a descriptive name like CHUNK_PATTERN_SIZE = 32 to make the relationship between the two uses clearer.

Copilot

Pull Request Overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

static/js/file-transfer.js:1

The condition chunk_count % 1 == 0 will always be true for any integer, making this check redundant. Consider removing it or using a different threshold like chunk_count % 10 == 0 to save progress periodically.

const TRANSFER_ID_MAX_NUMBER = 1000;

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-25T15:01:36Z

static/js/file-transfer.js

+const BUFFER_THRESHOLD_MOBILE = CHUNK_SIZE_MOBILE * 16;
+const BUFFER_THRESHOLD_DESKTOP = CHUNK_SIZE_DESKTOP * 16;
+const MAX_HASH_SAMPLING = 2 * 1024**2;
+const DEBUG_LOGS = false;


DEBUG_LOGS is hardcoded to false. Consider making this configurable based on environment or build configuration to avoid accidentally disabling debugging in development.

Suggested change

const DEBUG_LOGS = false;

const DEBUG_LOGS = (

(typeof window !== 'undefined' && window.DEBUG_LOGS === true) ||

(typeof window !== 'undefined' && window.location && window.location.search.includes('debug=true'))

);

Copilot · 2025-09-25T15:01:36Z

lib/store.py

-        """Set an event flag for this transfer."""
-        event_key = self.key(event_name)
+        if last_id is not None:
+            min_id = f'({last_id.decode() if isinstance(last_id, bytes) else last_id}'


The f-string is missing a closing parenthesis. It should be f'({last_id.decode() if isinstance(last_id, bytes) else last_id})' to properly format the Redis XRANGE minimum ID.

Suggested change

min_id = f'({last_id.decode() if isinstance(last_id, bytes) else last_id}'

min_id = f'({last_id.decode() if isinstance(last_id, bytes) else last_id})'

The ( character means that the ID range is exclusive (doesn't include start and end IDs). There shouldn't be an end parenthesis since it's not text but a specific Redis syntax.

codeSamuraii and others added 5 commits September 20, 2025 12:25

Deleted WS download endpoint

adb2bd4

Remove unused stream_store.py module

a247ff7

The stream_store.py module was created but never used - all functionality was integrated directly into store.py. Also removed backup file. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

Big tests refacto

e77a8ca

codeSamuraii requested a review from Copilot September 22, 2025 20:34

github-advanced-security bot found potential problems Sep 22, 2025

View reviewed changes

views/websockets.py Dismissed Show dismissed Hide dismissed

This comment was marked as outdated.

Sign in to view

Review comments:

65cee30

- 416 in case of invalid Range header - Fast hash in upload key instead of ID - Fixing tests

codeSamuraii force-pushed the resume-transfers branch from 62f27fc to 65cee30 Compare September 23, 2025 22:01

codeSamuraii added 3 commits September 24, 2025 00:41

Refacto async JS

9af5bc6

Resumable uploads and downloads working

f2e01bd

Merge branch 'improve-async-js' into resume-transfers

800e5f1

codeSamuraii requested a review from Copilot September 25, 2025 11:27

Copilot AI reviewed Sep 25, 2025

View reviewed changes

Repository owner deleted a comment from Copilot AI Sep 25, 2025

codeSamuraii marked this pull request as ready for review September 25, 2025 14:59

codeSamuraii requested a review from Copilot September 25, 2025 15:00

Copilot AI reviewed Sep 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Resume transfers #12

Resume transfers #12

Uh oh!

codeSamuraii commented Sep 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 25, 2025

Uh oh!

Copilot AI Sep 25, 2025

Uh oh!

Copilot AI Sep 25, 2025

Uh oh!

Copilot AI Sep 25, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 25, 2025

Uh oh!

Copilot AI Sep 25, 2025

Uh oh!

codeSamuraii Sep 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	else:
	await transfer.set_client_connected()
	await transfer.set_client_connected()

	const DEBUG_LOGS = false;
	const DEBUG_LOGS = typeof window !== 'undefined' && window.DEBUG_LOGS === true;

	if chunk_count % 1 == 0 or bytes_uploaded % (16 * 1024) == 0:
	if chunk_count % 10 == 0 or bytes_uploaded % (16 * 1024) == 0:

-const DEBUG_LOGS = false;
+const DEBUG_LOGS = (
+    (typeof window !== 'undefined' && window.DEBUG_LOGS === true) ||
+    (typeof window !== 'undefined' && window.location && window.location.search.includes('debug=true'))
+);

	min_id = f'({last_id.decode() if isinstance(last_id, bytes) else last_id}'
	min_id = f'({last_id.decode() if isinstance(last_id, bytes) else last_id})'

Uh oh!

Resume transfers #12

Are you sure you want to change the base?

Resume transfers #12

Uh oh!

Conversation

codeSamuraii commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

codeSamuraii Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codeSamuraii commented Sep 22, 2025 •

edited

Loading