Skip to content

Conversation

ashutoshpant
Copy link
Contributor

Description of PR

added new parameter object in s3ClientCreationParameters that returns full s3a path

How was this patch tested?

Tested against the following endpoint -> s3.us-east-1.amazonaws.com

For code changes:

  • [ X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@steveloughran
Copy link
Contributor

get yetus to stop failing the build, then run the integration tests against an s3 bucket, then i will review

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 42s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 38m 31s trunk passed
+1 💚 compile 1m 2s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 compile 0m 54s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 0m 52s trunk passed
+1 💚 mvnsite 1m 2s trunk passed
+1 💚 javadoc 0m 49s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 0m 51s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 1m 33s trunk passed
+1 💚 shadedclient 21m 2s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 39s the patch passed
+1 💚 compile 0m 42s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javac 0m 42s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 0m 37s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 27s the patch passed
+1 💚 mvnsite 0m 41s the patch passed
+1 💚 javadoc 0m 25s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 0m 32s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 1m 13s the patch passed
+1 💚 shadedclient 20m 27s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 40s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 51s The patch does not generate ASF License warnings.
98m 0s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/2/artifact/out/Dockerfile
GITHUB PR #4572
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux df2b3c78addb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 19ee4f5
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/2/testReport/
Max. process+thread count 732 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@apache apache deleted a comment from hadoop-yetus Jul 19, 2022
@steveloughran
Copy link
Contributor

ok, which s3 endpoint did you run the tests against?

@ashutoshpant
Copy link
Contributor Author

ok, which s3 endpoint did you run the tests against?

Just ran the tests against us-east-1

Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good, one comment about getter/setter names

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 42m 5s trunk passed
+1 💚 compile 1m 3s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 compile 0m 50s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 0m 45s trunk passed
+1 💚 mvnsite 0m 57s trunk passed
+1 💚 javadoc 0m 41s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 0m 44s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 1m 31s trunk passed
+1 💚 shadedclient 22m 22s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 39s the patch passed
+1 💚 compile 0m 43s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javac 0m 43s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 0m 37s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 27s the patch passed
+1 💚 mvnsite 0m 41s the patch passed
+1 💚 javadoc 0m 23s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 0m 32s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 1m 13s the patch passed
+1 💚 shadedclient 20m 26s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 44s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 52s The patch does not generate ASF License warnings.
102m 46s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/3/artifact/out/Dockerfile
GITHUB PR #4572
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 32b996b7804d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1de79b6
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/3/testReport/
Max. process+thread count 719 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, merging

@steveloughran steveloughran merged commit bac2219 into apache:trunk Jul 21, 2022
asfgit pushed a commit that referenced this pull request Jul 25, 2022
…4572)

Adds a new parameter object in s3ClientCreationParameters that holds
the full s3a path URI

Contributed by Ashutosh Pant
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
…pache#4572)


Adds a new parameter object in s3ClientCreationParameters that holds 
the full s3a path URI

Contributed by Ashutosh Pant
@khancon
Copy link

khancon commented Dec 12, 2022

Greetings, I was wondering what version of hadoop was this change merged into? Is it in 3.3.4 or will this come out with 3.3.5 release?

@steveloughran
Copy link
Contributor

look at the jira; comming in 3.3.5 which will be at release candiate 0 this week https://issues.apache.org/jira/browse/HADOOP-18330

@khancon
Copy link

khancon commented Jan 10, 2023

Greetings. It looks 3.3.5 still hasnt released even though it has been a month. Looks like the current version is still 3.3.4. Can you give us an exact date of when this will release and do so ASAP?

@steveloughran
Copy link
Contributor

The first release candidate is up for testing. Download and test it. This is the only way you can be sure that the actual shipped version does what you need.

@khancon
Copy link

khancon commented Jan 17, 2023

where can I download this "candidate" 3.3.5 version you are talking about? It's nowhere on your website.

@khancon
Copy link

khancon commented Jan 24, 2023

Hi, I would appreciate a response to this question above. Thanks

@steveloughran
Copy link
Contributor

we put up RCs for review through our dev lists; if you really want to get involved in pre-release code you should subscribe. here is the announcement. as it warns, I've been on vacation for most of the month, hence my lack of replies.


---------- Forwarded message ---------
From: Steve Loughran
Date: Wed, 21 Dec 2022 at 19:28
Subject: [VOTE] Release Apache Hadoop 3.3.5
To: Hadoop Common common-dev@hadoop.apache.org, Hdfs-dev hdfs-dev@hadoop.apache.org, yarn-dev yarn-dev@hadoop.apache.org, mapreduce-dev mapreduce-dev@hadoop.apache.org

Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.

Given the time of year it's a bit unrealistic to run a 5 day vote and expect people to be able to test it thoroughly enough to make this the one we can ship.

What we would like is for anyone who can to verify the tarballs, and test the binaries, especially anyone who can try the arm64 binaries. We've got the building of those done and now the build file will incorporate them into the release -but neither of us have actually tested it yet. Maybe I should try it on my pi400 over xmas.

The maven artifacts are up on the apache staging repo -they are the ones from x86 build. Building and testing downstream apps will be incredibly helpful.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/

The git tag is release-3.3.5-RC0, commit 3262495

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1365/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

  • Big update of dependencies to try and keep those reports of
    transitive CVEs under control -both genuine and false positive.
  • HDFS RBF enhancements
  • Critical fix to ABFS input stream prefetching for correct reading.
  • Vectored IO API for all FSDataInputStream implementations, with
    high-performance versions for file:// and s3a:// filesystems.
    file:// through java native io
    s3a:// parallel GET requests.
  • This release includes Arm64 binaries. Please can anyone with
    compatible systems validate these.

Please try the release and vote on it, even though i don't know what is a good timeline here...i'm actually going on holiday in early jan. Mukund is around and so can drive the process while I'm offline.

Assuming we do have another iteration, the RC1 will not be before mid jan for that reason

Steve (and mukund)

@khancon
Copy link

khancon commented Mar 16, 2023

Hi there! When can we expect the official release for version 3.3.5. I’m curious what the hold up is.

@steveloughran
Copy link
Contributor

When can we expect the official release for version 3.3.5.
its out

I’m curious what the hold up is.

one of the big problems is lack of testing of release candidates. it is very frustating to have a release ready and then, during the week of voting on it, not enough people review it. One contribution everyone can make to any of the ASF project is to help validate the release.

If you depend on something working, this is an important thing to do. If any bug surfaces now, it wont get fixed for three months at least.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants