HADOOP-18168. Fix S3A ITestMarkerTool dep. on purged public bucket, introduce PublicDatasetTestUtils #4140
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Integration tests passing against |
monthonk
suggested changes
Apr 5, 2022
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/tools/ITestMarkerTool.java
Outdated
Show resolved
Hide resolved
dannycjones
changed the title
dannycjones
changed the title
dannycjones
commented
Apr 6, 2022
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java
Outdated
Show resolved
Hide resolved
|
Integration tests not currently passing with this commit, diving deeper here. |
|
Tests passing against |
steveloughran
requested changes
Apr 28, 2022
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/PublicDatasetTestUtils.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARequesterPays.java
Outdated
Show resolved
Hide resolved
|
Tested latest patch against |
|
Thanks @steveloughran, I've added a note to I am hoping that if we get all public datasets into For now, I have not tried to move the CSV Scale tests but I'd like to see them moved in a future patch. |
dannycjones
changed the title
steveloughran
approved these changes
May 3, 2022
steveloughran
added a commit
that referenced
this pull request
May 3, 2022
asfgit
pushed a commit
that referenced
this pull request
May 3, 2022
asfgit
pushed a commit
that referenced
this pull request
May 3, 2022
…4140) This moves off use of the purged s3a://landsat-pds bucket, so fixing tests which had started failing. * Adds a new class, PublicDatasetTestUtils to manage the use of public datasets. * The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon HADOOP-14661. Consult the updated test documentation when running against other S3 stores. Contributed by Daniel Carl Jones Change-Id: Ie8585e4d9b67667f8cb80b2970225d79a4f8d257
|
merged to trunk, albeit with the commit message partially cut. reverted that and recommitted with the full text. test with branch-3.3 in progress |
asfgit
pushed a commit
that referenced
this pull request
May 3, 2022
…4140) This moves off use of the purged s3a://landsat-pds bucket, so fixing tests which had started failing. * Adds a new class, PublicDatasetTestUtils to manage the use of public datasets. * The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon HADOOP-14661. Consult the updated test documentation when running against other S3 stores. Contributed by Daniel Carl Jones Change-Id: Ie8585e4d9b67667f8cb80b2970225d79a4f8d257
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
This moves off use of the purged s3a://landsat-pds bucket, so fixing tests which had started failing. * Adds a new class, PublicDatasetTestUtils to manage the use of public datasets. * The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon HADOOP-14661. Consult the updated test documentation when running against other S3 stores. Contributed by Daniel Carl Jones
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
This reverts commit 6ab7b72.
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
…pache#4140) This moves off use of the purged s3a://landsat-pds bucket, so fixing tests which had started failing. * Adds a new class, PublicDatasetTestUtils to manage the use of public datasets. * The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon HADOOP-14661. Consult the updated test documentation when running against other S3 stores. Contributed by Daniel Carl Jones Change-Id: Ie8585e4d9b67667f8cb80b2970225d79a4f8d257
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Addressing HADOOP-18168, this change replaces the use of
landsat-pdswith its successor usgs-landsat, another public dataset.landsat-pdsrecently was cleaned up excluding some specific files and is no longer suitable for this test case.usgs-landsatis a requester pays bucket, so costs will not be passed on to the owner of the bucket - instead, they will be paid by the test runner.We introduce a new class
PublicDatasetTestUtilswhich provides a simple interface for getting file systems for a specific purpose. This change allows the bucket used to be replaced or for the tests to be skipped - for example, for S3-compatible stores or non-awspartition S3 endpoints.How was this patch tested?
It will be tested from a
eu-west-1EC2 instance against aeu-west-1bucket.