Skip to content

Conversation

@lebovic
Copy link
Member

@lebovic lebovic commented Dec 15, 2021

Adds:

  • Support for multipart uploads via boto3
  • Shifts download to boto3 as well

Removes:

  • Usage of pre-signed URLs and raw file posting via requests

@lebovic lebovic requested a review from bcai2 December 15, 2021 07:18
Copy link
Contributor

@bcai2 bcai2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending review of the API PR.

@lebovic lebovic merged commit 18a6c70 into staging Dec 15, 2021
@lebovic lebovic deleted the feature/multipart-uploads branch December 15, 2021 19:16
@lebovic lebovic mentioned this pull request Dec 16, 2021
lebovic added a commit that referenced this pull request Dec 20, 2021
Adds:
- Ability to pass S3 URIs as inputs (#49)
- Modified handling of arguments + raw execution mode + more STAR arguments (#57)
- Multipart uploads / downloads and the ability to increase non-parallel input file sizes (#60, #61, #63, #65))
- Enable output file `.tar.gz`s across the board (#68)
- Add an explicit `parallelize=True` flag (#68)

Modifies:
- Various fixes (#49, #56, #58, #64, #67)

Integration tests: passing
Unit tests: passing
Manual tests: passing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants