Skip to content

Fix: unsigned s5cmd requests and also add option to disable s5cmd #513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Mar 15, 2025

Conversation

bhimrazy
Copy link
Collaborator

@bhimrazy bhimrazy commented Mar 13, 2025

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

  • Enables support for unsigned requests with s5cmd to pass as storage options
     storage_options = {
      "AWS_NO_SIGN_REQUEST": "Yes",
      "S3_ENDPOINT_URL": "endpoint-url",
      }
    
  • Adds an option to disable s5cmd
  • Add test cases
  • Update README

Fixes #511.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@bhimrazy bhimrazy added enhancement New feature or request bugfix labels Mar 13, 2025
@bhimrazy bhimrazy self-assigned this Mar 13, 2025
@bhimrazy bhimrazy changed the title Fix: Handle Unsigned s5cmd Requests and Add Option to Disable s5cmd Fix: unsigned s5cmd requests and also add option to disable s5cmd Mar 13, 2025
Copy link

codecov bot commented Mar 13, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79%. Comparing base (32659a6) to head (2674f80).
Report is 1 commits behind head on main.

Additional details and impacted files
@@         Coverage Diff         @@
##           main   #513   +/-   ##
===================================
  Coverage    79%    79%           
===================================
  Files        39     39           
  Lines      5848   5859   +11     
===================================
+ Hits       4605   4621   +16     
+ Misses     1243   1238    -5     
🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bhimrazy bhimrazy marked this pull request as ready for review March 14, 2025 10:21
@bhimrazy bhimrazy requested a review from tchaton March 15, 2025 08:28
Copy link
Collaborator

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beautiful

@tchaton tchaton merged commit 7f965ce into Lightning-AI:main Mar 15, 2025
29 checks passed
@bhimrazy bhimrazy deleted the fix/s5cmd-no-signed branch March 15, 2025 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

storage_options endpoint_url works for S3Client but not s5cmd in S3Downloader
2 participants