feat(s3): make S3 credentials optional to support default AWS credential chain by YQ-Wang · Pull Request #1749 · dragonflyoss/client

YQ-Wang · 2026-03-27T08:21:25Z

Summary

Makes S3 access_key_id, access_key_secret, and region fields optional in the object
storage backend. When omitted, OpenDAL falls back to the default AWS credential chain
(environment variables, IMDS, IRSA, etc.), enabling nodes to use their own IAM role without
requiring explicit static credentials.
Adds validation to reject partial credential configurations: if one of access_key_id or
access_key_secret is provided, the other must also be present; session_token requires
both access key fields.
Updates CLI help text across dfget, dfstore export, and dfstore import to document
the new optional behavior and default credential chain fallback.

Test plan

Unit tests in object_storage.rs updated to cover new valid configurations (empty
config, endpoint-only, region-only, explicit key pair) and updated error cases (partial
credentials, session token without key pair)

This PR resolves #1750

…d loaders When access_key_id and access_key_secret are omitted, OpenDAL falls back to its configured credential loaders for the selected endpoint. This preserves task identity while letting nodes use their own IAM role without requiring explicit static credentials. Region defaults to us-east-1 when omitted; OpenDAL follows S3 301 redirects for buckets in other regions. Partial credentials are still rejected: if one of access_key_id or access_key_secret is provided, the other must also be present. Session tokens require both access key fields. CLI help text across dfget, dfstore export, and dfstore import is now consistent and accurately describes the optional behavior. Tests now verify both the default-region path and request-level S3 stat behavior without depending on ambient AWS credentials in CI. Made-with: Cursor

codecov · 2026-03-27T08:40:27Z

Codecov Report

❌ Patch coverage is 60.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.63%. Comparing base (e6918d5) to head (6fd1f1c).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
dragonfly-client/src/bin/dfcache/export.rs	0.00%	1 Missing ⚠️
dragonfly-client/src/bin/dfcache/import.rs	0.00%	1 Missing ⚠️
dragonfly-client/src/bin/dfcache/stat.rs	0.00%	1 Missing ⚠️
dragonfly-client/src/bin/dfdaemon/main.rs	0.00%	1 Missing ⚠️
dragonfly-client/src/bin/dfstore/export.rs	0.00%	1 Missing ⚠️
dragonfly-client/src/bin/dfstore/import.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1749      +/-   ##
==========================================
- Coverage   46.82%   46.63%   -0.19%     
==========================================
  Files          87       87              
  Lines       24881    24807      -74     
==========================================
- Hits        11650    11569      -81     
- Misses      13231    13238       +7

Files with missing lines	Coverage Δ
dragonfly-client-backend/src/object_storage.rs	`87.90% <100.00%> (-0.87%)`	⬇️
dragonfly-client/src/bin/dfget/main.rs	`49.33% <100.00%> (ø)`
dragonfly-client/src/bin/dfcache/export.rs	`0.00% <0.00%> (ø)`
dragonfly-client/src/bin/dfcache/import.rs	`0.00% <0.00%> (ø)`
dragonfly-client/src/bin/dfcache/stat.rs	`0.00% <0.00%> (ø)`
dragonfly-client/src/bin/dfdaemon/main.rs	`0.00% <0.00%> (ø)`
dragonfly-client/src/bin/dfstore/export.rs	`0.00% <0.00%> (ø)`
dragonfly-client/src/bin/dfstore/import.rs	`0.00% <0.00%> (ø)`

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR updates the S3 object storage backend configuration to allow omitting explicit credentials so OpenDAL can use the default AWS credential chain (env/IMDS/IRSA/etc.), aligning dfget S3 downloads with AWS-hosted runtime expectations.

Changes:

Made S3 access_key_id, access_key_secret, and region optional in the backend, defaulting region to us-east-1 when omitted.
Added validation to reject partial explicit credential configurations and to require an explicit key pair when session_token is provided.
Updated CLI help text in dfget and dfstore import/export to document the new optional behavior and fallback semantics; expanded backend unit tests accordingly.

Reviewed changes

Copilot reviewed 1 out of 4 changed files in this pull request and generated no comments.

File	Description
`dragonfly-client-backend/src/object_storage.rs`	Implements optional S3 credentials + validation rules; defaults region; updates/extends tests (including a wiremock stat test).
`dragonfly-client/src/bin/dfget/main.rs`	Updates CLI examples and help text to describe explicit vs default-chain behavior.
`dragonfly-client/src/bin/dfstore/export.rs`	Updates CLI help text for storage region/endpoint/credentials/session token.
`dragonfly-client/src/bin/dfstore/import.rs`	Updates CLI help text for storage region/endpoint/credentials/session token.

Signed-off-by: Gaius <gaius.qi@gmail.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 9 changed files in this pull request and generated 4 comments.

dragonfly-client-backend/src/object_storage.rs

EvanCley

s3 will not verify param integrity when using its credential chain, maybe orther object storage should follow next step?

EvanCley

lgtm

…ial chain (dragonflyoss#1749) * feat(s3): make S3 credentials optional to support OpenDAL's configured loaders When access_key_id and access_key_secret are omitted, OpenDAL falls back to its configured credential loaders for the selected endpoint. This preserves task identity while letting nodes use their own IAM role without requiring explicit static credentials. Region defaults to us-east-1 when omitted; OpenDAL follows S3 301 redirects for buckets in other regions. Partial credentials are still rejected: if one of access_key_id or access_key_secret is provided, the other must also be present. Session tokens require both access key fields. CLI help text across dfget, dfstore export, and dfstore import is now consistent and accurately describes the optional behavior. Tests now verify both the default-region path and request-level S3 stat behavior without depending on ambient AWS credentials in CI. Made-with: Cursor * feat: add env var support for CLI args, remove S3 cred validation Signed-off-by: Gaius <gaius.qi@gmail.com> --------- Signed-off-by: Gaius <gaius.qi@gmail.com> Co-authored-by: Gaius <gaius.qi@gmail.com> Signed-off-by: Jonathan van der Steege <jonathan@jonakeys.nl>

YQ-Wang requested review from a team as code owners March 27, 2026 08:21

YQ-Wang requested review from BraveY, CooooolFrog, fcgxz2003, gaius-qi, xujihui1985 and yyzai384 March 27, 2026 08:21

github-actions bot requested a review from jim3ma March 27, 2026 08:21

github-actions bot assigned YQ-Wang Mar 27, 2026

YQ-Wang force-pushed the feat/s3-optional-credentials branch from ec7593b to 6906b45 Compare March 27, 2026 08:29

YQ-Wang force-pushed the feat/s3-optional-credentials branch from 6906b45 to 48c2a71 Compare March 27, 2026 08:36

YQ-Wang mentioned this pull request Mar 27, 2026

Support AWS default credential chain for standard S3 downloads in dfget #1750

Closed

gaius-qi added the enhancement New feature or request label Mar 30, 2026

gaius-qi requested a review from Copilot March 30, 2026 03:21

Copilot started reviewing on behalf of gaius-qi March 30, 2026 03:21 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

feat: add env var support for CLI args, remove S3 cred validation

6fd1f1c

Signed-off-by: Gaius <gaius.qi@gmail.com>

gaius-qi requested a review from Copilot March 30, 2026 09:22

Copilot started reviewing on behalf of gaius-qi March 30, 2026 09:22 View session

hhhhsdxxxx approved these changes Mar 30, 2026

View reviewed changes

gaius-qi enabled auto-merge (squash) March 30, 2026 09:24

Copilot AI reviewed Mar 30, 2026

View reviewed changes

EvanCley reviewed Mar 30, 2026

View reviewed changes

EvanCley approved these changes Mar 30, 2026

View reviewed changes

yxxhero approved these changes Mar 30, 2026

View reviewed changes

gaius-qi disabled auto-merge March 30, 2026 10:06

gaius-qi merged commit ce8ac11 into dragonflyoss:main Mar 30, 2026
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(s3): make S3 credentials optional to support default AWS credential chain#1749

feat(s3): make S3 credentials optional to support default AWS credential chain#1749
gaius-qi merged 2 commits intodragonflyoss:mainfrom
YQ-Wang:feat/s3-optional-credentials

YQ-Wang commented Mar 27, 2026 •

edited

Loading

Uh oh!

codecov bot commented Mar 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EvanCley left a comment

Uh oh!

EvanCley left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

YQ-Wang commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

codecov bot commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EvanCley left a comment

Choose a reason for hiding this comment

Uh oh!

EvanCley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

YQ-Wang commented Mar 27, 2026 •

edited

Loading

codecov bot commented Mar 27, 2026 •

edited

Loading