Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-17851. Support user specified content encoding for S3A #3498

Merged
merged 12 commits into from
Sep 29, 2021
Merged

HADOOP-17851. Support user specified content encoding for S3A #3498

merged 12 commits into from
Sep 29, 2021

Commits on Sep 28, 2021

  1. Add the ability for users to specify the content encoding of the obje… 

    …cts they are putting. This is useful for people loading the data into other tools in the AWS ecosystem which don't use file extensions to infer compression type (e.g. serving compressed files from S3 or importing into RDS)
    @holdenk @steveloughran
    holdenk authored and steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    0ae488b View commit details
    Browse the repository at this point in the history

  2. Add an integration test

    @holdenk @steveloughran
    holdenk authored and steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    0f82f22 View commit details
    Browse the repository at this point in the history

  3. Update the ITestS3a*Enc* skip logic to handle nulls, make my ITestS3A… 

    … test fail.
    @holdenk @steveloughran
    holdenk authored and steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    d519c9d View commit details
    Browse the repository at this point in the history

  4. Fix setting the content encoding and change the test to a real one

    @holdenk @steveloughran
    holdenk authored and steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    8292662 View commit details
    Browse the repository at this point in the history

  5. Fix content encoding config var name to match

    @holdenk @steveloughran
    holdenk authored and steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    dcab243 View commit details
    Browse the repository at this point in the history

  6. Fix some linter errors and add a rename op

    @holdenk @steveloughran
    holdenk authored and steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    442cc2f View commit details
    Browse the repository at this point in the history

  7. Switch from using the full s3 client to just using getXattrs 

    oops
    @holdenk @steveloughran
    holdenk authored and steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    7cb781a View commit details
    Browse the repository at this point in the history

  8. getXattr fix

    @holdenk @steveloughran
    holdenk authored and steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    5a7a69f View commit details
    Browse the repository at this point in the history

  9. Code review feedback

    @holdenk @steveloughran
    holdenk authored and steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    0d736ff View commit details
    Browse the repository at this point in the history

  10. Unused import cleanup in test

    @holdenk @steveloughran
    holdenk authored and steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    b3d1b18 View commit details
    Browse the repository at this point in the history

  11. HADOOP-17851 roll back the changes to the encryption tests, as test s… 

    …etup/skip is now different

Change-Id: I0bec0926625667ca1c8be8fa11e8ff1a11e0dbe8
    @steveloughran
    steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    f4fddcb View commit details
    Browse the repository at this point in the history

  12. HADOOP-17851. tuning content encoding 

    * change option to fs.s3a.object.content.encoding to line up for
  support of the other values
* docs
* dir markers do not have an encoding set
* tests to verify that

Change-Id: I4b042d38c9e94e7d4ffa8ee6afdc7a6c26a4c489
    @steveloughran
    steveloughran committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    f457367 View commit details
    Browse the repository at this point in the history