feat(proxy): Introduce an S3-aware proxy mode in dfdaemon

[YQ-Wang]

Description

Add an S3-aware proxy mode to dfdaemon that classifies incoming S3 requests and routes them
accordingly:

• GetObject (full and ranged): accelerated through Dragonfly P2P. If the caller's SigV4
  signature explicitly covers the Range header, dfdaemon preserves that original signed
  Range on the source request instead of rewriting it.
• HeadObject and ListObjectsV2: direct passthrough.
• Other S3 APIs: passthrough, not accelerated.

Key implementation details:

• New proxy::s3 module for request classification (virtual-hosted and path-style buckets,
  configurable host allowlist, automatic AWS endpoint detection).
• Task ID generation strips per-request SigV4 query parameters (X-Amz-Credential,
  X-Amz-Date, X-Amz-Expires, X-Amz-Signature, X-Amz-Security-Token,
  X-Amz-SignedHeaders, X-Amz-Algorithm) so different presigned URLs for the same object
  share the same P2P task.
• New proxy::header helpers for SigV4 signed-header detection and range preservation signaling
  via internal X-Dragonfly-Preserve-Original-Range-For-Source header.
• dfdaemon_download marks signed-range requests so downstream resource handling preserves the
  caller's original signed Range on source requests.
• resource::task generates deterministic FNV-1a piece numbers for signed-range pieces to avoid
  collisions with standard sequential piece numbering.
• resource::piece conditionally omits the Range header on source GET requests when the
  preserve-original-range flag is set, so the source returns the full signed range as-is.
• S3 proxy metrics (dragonfly_client_s3_proxy_request_total) for per-operation, per-route
  counters.
• Updated README, ci/dfdaemon.service comments, and doctest fixes in
  dragonfly-client-util (unrelated pre-existing breakage).

Related Issue

Motivation and Context

Organizations storing large ML models, datasets, or media files in S3 and distributing them
across many hosts cannot leverage Dragonfly P2P today without rewriting download logic to use
Dragonfly's API directly. With S3-aware proxy mode, those workloads are accelerated transparently:
applications only need to set HTTPS_PROXY and trust the proxy CA — no code changes required.

The proxy preserves caller-provided SigV4 headers and presigned URLs end-to-end. dfdaemon does
not discover AWS credentials or re-sign origin requests, keeping the security model unchanged.

Testing

Tested end-to-end against a real S3 object (s3://bucket-name/test_dragonfly/file.mp4,
~50 MB) through a locally deployed Dragonfly cluster (manager, scheduler, seed-client, client via
Docker Compose). All tests ran inside the client container with HTTPS_PROXY and AWS_CA_BUNDLE
configured to route traffic through the patched dfdaemon.

boto3 test matrix (all passed with correct proxy routing confirmed via dfdaemon logs):

Test	Method	Route
HeadObject	s3.head_object()	passthrough
ListObjectsV2	s3.list_objects_v2()	passthrough
GetObject (full)	s3.get_object()	P2P via dfdaemon
GetObject (small range)	s3.get_object(Range='bytes=0-1023')	P2P via dfdaemon
GetObject (nonzero range)	s3.get_object(Range='bytes=1048576-2097151')	P2P via dfdaemon
Multipart download	s3.download_file()	P2P via dfdaemon
Presigned HEAD	presigned URL + HTTP HEAD	passthrough
Presigned GET (full)	presigned URL + HTTP GET	P2P via dfdaemon
Presigned GET (ranged)	presigned URL + HTTP GET + Range header	P2P via dfdaemon

s5cmd tests (all passed):

Test	Command	Route
Default download (full-object signed range)	s5cmd cp s3://... .	P2P via dfdaemon
Forced multipart (multiple signed partial ranges)	s5cmd cp --part-size 5 --concurrency 3 s3://... .	P2P via dfdaemon

Presigned URL test: Tested a regional presigned URL (bucket-name.s3.us-east-1.amazonaws.com) for both full GET and ranged GET — both routed correctly.

Recursive folder download: s3://bucket-name/test_dragonfly/ with nested subdirectories — all objects downloaded with correct sizes and routing.

cargo test: Passed after fixing unrelated pre-existing doctest failures in dragonfly-client-util.

[codecov]

Codecov Report

❌ Patch coverage is 85.49451% with 132 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.39%. Comparing base (5400d49) to head (3de8859).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
dragonfly-client/src/proxy/mod.rs	84.49%	51 Missing ⚠️
dragonfly-client/src/proxy/s3.rs	91.98%	23 Missing ⚠️
dragonfly-client/src/resource/task.rs	75.00%	22 Missing ⚠️
dragonfly-client/src/resource/piece.rs	0.00%	17 Missing ⚠️
dragonfly-client/src/proxy/header.rs	92.19%	11 Missing ⚠️
dragonfly-client-metric/src/lib.rs	66.66%	5 Missing ⚠️
dragonfly-client-backend/src/http.rs	84.61%	2 Missing ⚠️
dragonfly-client/src/grpc/dfdaemon_download.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1779      +/-   ##
==========================================
+ Coverage   44.54%   46.39%   +1.84%     
==========================================
  Files          91       92       +1     
  Lines       25834    26813     +979     
==========================================
+ Hits        11508    12440     +932     
- Misses      14326    14373      +47     

Files with missing lines	Coverage Δ
dragonfly-client-config/src/dfdaemon.rs	89.51% <100.00%> (+0.19%)	⬆️
dragonfly-client-util/src/cgroups/mod.rs	0.00% <ø> (ø)
dragonfly-client-util/src/sysinfo/cpu.rs	6.19% <ø> (ø)
dragonfly-client/src/grpc/dfdaemon_download.rs	4.79% <0.00%> (ø)
dragonfly-client-backend/src/http.rs	96.70% <84.61%> (-0.27%)	⬇️
dragonfly-client-metric/src/lib.rs	72.35% <66.66%> (-0.10%)	⬇️
dragonfly-client/src/proxy/header.rs	88.07% <92.19%> (+2.15%)	⬆️
dragonfly-client/src/resource/piece.rs	37.59% <0.00%> (-1.27%)	⬇️
dragonfly-client/src/resource/task.rs	14.04% <75.00%> (+7.98%)	⬆️
dragonfly-client/src/proxy/s3.rs	91.98% <91.98%> (ø)
... and 1 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.