Feature request:
Support S3 clients that send SigV4-signed ranged GetObject requests through dfdaemon's HTTP proxy.
Some clients, such as s5cmd, download large S3 objects using multipart ranged GETs. These requests include the Range header in AWS SigV4 SignedHeaders. Today, dfdaemon's proxy may rewrite the Range header when fetching pieces from the source. Since Range is part of the canonical signed request, rewriting it causes S3 to reject the request with 403 SignatureDoesNotMatch.
The request is to make dfdaemon preserve the original Range header when it is explicitly covered by SigV4 authentication, while keeping the existing piece-splitting behavior for requests where Range is not signature-bound.
Use case:
A user wants to accelerate model checkpoint downloads from S3 in a Dragonfly cluster without changing application code. The application uses s5cmd to download large model files, for example:
HTTPS_PROXY=http://127.0.0.1:4001 \
s5cmd cp s3://my-bucket/models/large-checkpoint.safetensors /models/
s5cmd performs multipart downloads with requests like:
GET /models/large-checkpoint.safetensors HTTP/1.1
Range: bytes=5242880-10485759
Authorization: AWS4-HMAC-SHA256 ... SignedHeaders=host;range;x-amz-content-sha256;x-amz-date ...
With the current proxy behavior, S3 can return:
403 SignatureDoesNotMatch
because the proxy rewrites the signed Range value before forwarding the request to S3.
With this feature, dfdaemon would detect that Range is signature-bound, preserve the original header for the source request, and still allow identical ranged requests from other peers to reuse Dragonfly cache/P2P data.
UI Example:
No new user-facing command or option is required. This should work transparently when an S3 client uses dfdaemon as an HTTP/HTTPS proxy:
export HTTPS_PROXY=http://127.0.0.1:4001
export HTTP_PROXY=http://127.0.0.1:4001
s5cmd cp s3://my-bucket/models/large-checkpoint.safetensors /models/
Expected behavior: the download succeeds through the proxy without SignatureDoesNotMatch, and repeated downloads can be served from Dragonfly cache when the signed range requests match.
Feature request:
Support S3 clients that send SigV4-signed ranged
GetObjectrequests through dfdaemon's HTTP proxy.Some clients, such as
s5cmd, download large S3 objects using multipart ranged GETs. These requests include theRangeheader in AWS SigV4SignedHeaders. Today, dfdaemon's proxy may rewrite theRangeheader when fetching pieces from the source. SinceRangeis part of the canonical signed request, rewriting it causes S3 to reject the request with403 SignatureDoesNotMatch.The request is to make dfdaemon preserve the original
Rangeheader when it is explicitly covered by SigV4 authentication, while keeping the existing piece-splitting behavior for requests whereRangeis not signature-bound.Use case:
A user wants to accelerate model checkpoint downloads from S3 in a Dragonfly cluster without changing application code. The application uses
s5cmdto download large model files, for example:s5cmdperforms multipart downloads with requests like:With the current proxy behavior, S3 can return:
because the proxy rewrites the signed
Rangevalue before forwarding the request to S3.With this feature, dfdaemon would detect that
Rangeis signature-bound, preserve the original header for the source request, and still allow identical ranged requests from other peers to reuse Dragonfly cache/P2P data.UI Example:
No new user-facing command or option is required. This should work transparently when an S3 client uses dfdaemon as an HTTP/HTTPS proxy:
Expected behavior: the download succeeds through the proxy without
SignatureDoesNotMatch, and repeated downloads can be served from Dragonfly cache when the signed range requests match.