Skip to content

HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 #1738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Feb 20, 2021

Conversation

aajisaka
Copy link
Member

@aajisaka aajisaka commented Dec 5, 2019

JIRA: https://issues.apache.org/jira/browse/HADOOP-16748

  • Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
  • Fix formatting by IntelliJ IDEA Python plugin
  • Updated shebang to use Python 3
  • Removed code for Python 2.x
  • Removed determine-flaky-tests-hadoop.py
  • Ran the scripts with Python 3.7.5
  • Temporarily disabled shelldocs check due to YETUS-1099

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 24m 15s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ trunk Compile Tests _
+0 🆗 mvndep 1m 12s Maven dependency ordering for branch
+1 💚 shadedclient 13m 48s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for patch
-1 ❌ pylint 0m 7s The patch generated 2 new + 60 unchanged - 321 fixed = 62 total (was 381)
-1 ❌ whitespace 0m 0s The patch 2 line(s) with tabs.
+1 💚 shadedclient 13m 14s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 0m 30s The patch does not generate ASF License warnings.
55m 15s
Subsystem Report/Notes
Docker Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/1/artifact/out/Dockerfile
GITHUB PR #1738
Optional Tests dupname asflicense pylint
uname Linux 91221651a7ef 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 682e6fd
pylint https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/1/artifact/out/diff-patch-pylint.txt
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/1/artifact/out/whitespace-tabs.txt
Max. process+thread count 412 (vs. ulimit of 5500)
modules C: U:
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/1/console
versions git=2.7.4 maven=3.3.9 pylint=1.9.2
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@aajisaka aajisaka changed the title HADOOP-16747. Support Python 3 in dev-support scripts. HADOOP-16748. Support Python 3 in dev-support scripts. Dec 6, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 36s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ trunk Compile Tests _
+0 🆗 mvndep 1m 9s Maven dependency ordering for branch
+1 💚 shadedclient 13m 21s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 24s Maven dependency ordering for patch
-1 ❌ pylint 0m 6s The patch generated 1 new + 43 unchanged - 338 fixed = 44 total (was 381)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 9s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 0m 31s The patch does not generate ASF License warnings.
31m 0s
Subsystem Report/Notes
Docker Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/2/artifact/out/Dockerfile
GITHUB PR #1738
Optional Tests dupname asflicense pylint
uname Linux cc891ba47269 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 18059ac
pylint https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/2/artifact/out/diff-patch-pylint.txt
Max. process+thread count 412 (vs. ulimit of 5500)
modules C: U:
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/2/console
versions git=2.7.4 maven=3.3.9 pylint=1.9.2
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@aajisaka
Copy link
Member Author

aajisaka commented Dec 6, 2019

Python2 is still required in Dockerfile because releasedocmaker.py in Apache Yetus only supports Python2.

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/3/console in case of problems.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ docker 22m 56s Docker failed to build yetus/hadoop:104ccca9169.
Subsystem Report/Notes
GITHUB PR #1738
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/3/console
versions git=2.17.1
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/4/console in case of problems.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 33s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 shelldocs 0m 1s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ trunk Compile Tests _
+0 🆗 mvndep 1m 23s Maven dependency ordering for branch
+1 💚 shadedclient 16m 9s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 hadolint 0m 3s There were no new hadolint issues.
-1 ❌ pylint 0m 7s The patch generated 1 new + 43 unchanged - 338 fixed = 44 total (was 381)
+1 💚 shellcheck 0m 0s There were no new shellcheck issues.
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 46s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 0m 30s The patch does not generate ASF License warnings.
38m 2s
Subsystem Report/Notes
Docker Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/4/artifact/out/Dockerfile
GITHUB PR #1738
Optional Tests dupname asflicense pylint hadolint shellcheck shelldocs
uname Linux 6e7126c55028 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 8dffd8d
pylint https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/4/artifact/out/diff-patch-pylint.txt
Max. process+thread count 305 (vs. ulimit of 5500)
modules C: U:
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1738/4/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6 hadolint=1.11.1-0-g0e692dd pylint=1.9.2
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@liuml07
Copy link
Member

liuml07 commented Dec 13, 2019

Looks good to me overall, but I'm not Python expert to be +1

@aajisaka
Copy link
Member Author

Thanks @liuml07 for review.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 30s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ trunk Compile Tests _
+0 🆗 mvndep 0m 27s Maven dependency ordering for branch
+1 💚 shadedclient 14m 12s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 14m 30s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 24s Maven dependency ordering for patch
-1 ❌ pylint 0m 6s The patch generated 1 new + 43 unchanged - 338 fixed = 44 total (was 381)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 35s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 0m 32s The patch does not generate ASF License warnings.
31m 20s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/3/artifact/out/Dockerfile
GITHUB PR #1738
Optional Tests dupname asflicense pylint
uname Linux a5eb165fd2e1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / e756fe3
pylint https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/3/artifact/out/diff-patch-pylint.txt
Max. process+thread count 420 (vs. ulimit of 5500)
modules C: U:
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/3/console
versions git=2.17.1 maven=3.6.0 pylint=1.9.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@liuml07
Copy link
Member

liuml07 commented Aug 25, 2020

@aajisaka If you would like to put this forward, I can help review and vote for a binding +1 Thanks!

@asf-cloudbees-jenkins-ci-hadoop

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/7/console in case of problems.

@aajisaka
Copy link
Member Author

TODOs

  • Wait Apache Yetus 0.13.0 including YETUS-452 is released
  • Point the Yetus version from YETUS-452 to 0.13.0
  • Upgrade the Dockerfile for aarch64

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 27m 14s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 💚 shadedclient 23m 40s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 ❌ hadolint 0m 3s /diff-patch-hadolint.txt The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ pylint 0m 8s /diff-patch-pylint.txt The patch generated 1 new + 43 unchanged - 338 fixed = 44 total (was 381)
+1 💚 shellcheck 0m 0s There were no new shellcheck issues.
+1 💚 shelldocs 0m 14s There were no new shelldocs issues.
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 0m 35s The patch does not generate ASF License warnings.
68m 34s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/6/artifact/out/Dockerfile
GITHUB PR #1738
Optional Tests dupname asflicense shellcheck shelldocs pylint hadolint
uname Linux d39c5ea96eb7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / ebe1d1f
Max. process+thread count 537 (vs. ulimit of 5500)
modules C: . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/6/console
versions git=2.17.1 maven=3.6.0 shellcheck=0.4.6 hadolint=1.11.1-0-g0e692dd pylint=1.9.2
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@aajisaka
Copy link
Member Author

Umm. Pylint is still 1.9.2 in the precommit job. I updated the Dockerfile but it is not reflected.

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/8/console in case of problems.

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/9/console in case of problems.

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/11/console in case of problems.

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/12/console in case of problems.

@aajisaka
Copy link
Member Author

It seems that Python 2 is still used.

18:36:56  Running shelldocs against all identifiable shell scripts
18:36:58  Traceback (most recent call last):
18:36:58    File "/home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-1738/src/patchprocess/apache-yetus-0.13.0/bin/../lib/shelldocs/shelldocs.py", line 477, in <module>
18:36:58      main()
18:36:58    File "/home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-1738/src/patchprocess/apache-yetus-0.13.0/bin/../lib/shelldocs/shelldocs.py", line 468, in main
18:36:58      message = funcs.lint()
18:36:58    File "/home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-1738/src/patchprocess/apache-yetus-0.13.0/bin/../lib/shelldocs/shelldocs.py", line 273, in lint
18:36:58      value = getfuncs[attr]()
18:36:58  TypeError: 'str' object is not callable

@aajisaka
Copy link
Member Author

aajisaka commented Jan 26, 2021

I found the root cause and opened a PR in Yetus: apache/yetus#212

@aajisaka
Copy link
Member Author

Thank you @goiri for your review.

I'll rebase this and wait for YETUS-1099 before committing.

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/1/console in case of problems.

@aajisaka
Copy link
Member Author

Umm. I think we need to release Yetus 0.13.1

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/2/console in case of problems.

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/3/console in case of problems.

@asf-cloudbees-jenkins-ci-hadoop

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/4/console in case of problems.

@aajisaka
Copy link
Member Author

Now the patch is ready to go.

Copy link
Contributor

@amahussein amahussein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @aajisaka for working on that!

@asf-cloudbees-jenkins-ci-hadoop

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1738/5/console in case of problems.

@aajisaka
Copy link
Member Author

@goiri @amahussein Would you check the latest patch?

@amahussein
Copy link
Contributor

+1 (non-binding).
Thanks for working on that important upgrade @aajisaka !

@goiri
Copy link
Member

goiri commented Feb 19, 2021

LGTM.

Copy link
Member

@liuml07 liuml07 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@aajisaka aajisaka changed the title HADOOP-16748. Support Python 3 in dev-support scripts. HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 Feb 20, 2021
@aajisaka aajisaka merged commit b9b49ed into apache:trunk Feb 20, 2021
@aajisaka aajisaka deleted the HADOOP-16747 branch February 20, 2021 08:18
@aajisaka
Copy link
Member Author

Merged. Thank you @goiri @liuml07, and @amahussein.

aajisaka added a commit to aajisaka/hadoop that referenced this pull request Feb 22, 2021
…#1738)

- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
####
RUN pip2 install python-dateutil==2.7.3
RUN pip3 install pylint==2.6.0 python-dateutil==2.8.1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this line fails to run on arm64. it require python3-dev. I'll create a PR.

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021
…#1738)

- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
aajisaka added a commit to aajisaka/hadoop that referenced this pull request Dec 28, 2021
…#1738)

- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/bin/yetus-wrapper
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants