HADOOP-19017. Setup pre-commit CI for Windows 10 #5820

GauthamBanasandra · 2023-07-08T12:49:44Z

Description of PR

This PR sets up the pre-commit CI for validating Hadoop PRs on Windows 10.
This is where the pre-commit CI for Windows 10 runs - Apache Hadoop Multibranch pipeline for Windows 10.

How was this patch tested?

I've configured Apache Hadoop Multibranch pipeline for Windows 10 in Jenkins to point to my fork's trunk branch -

I then created a PR against my fork's trunk - HADOOP-19017. Setup pre-commit CI for Windows 10 GauthamBanasandra/hadoop#2
The Jenkins pipeline is detected the PR against my fork.
It ran the build and performed the verification on Windows 10 - HADOOP-19017. Setup pre-commit CI for Windows 10 GauthamBanasandra/hadoop#2 (comment).

I'll configure the Branch source to point to https://github.com/apache/hadoop after merging this PR.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

This reverts commit f75dce5.

This reverts commit e433bb2.

This reverts commit 9fb9ba2.

* Also do Github status recovery.

hadoop-yetus · 2023-12-24T18:40:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 17s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 37s		trunk passed
+1 💚	compile	3m 41s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	3m 45s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	mvnsite	0m 30s		trunk passed
+1 💚	shadedclient	31m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	3m 34s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	cc	3m 34s		the patch passed
+1 💚	golang	3m 34s		the patch passed
+1 💚	javac	3m 34s		the patch passed
+1 💚	compile	3m 40s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	cc	3m 40s		the patch passed
+1 💚	golang	3m 40s		the patch passed
+1 💚	javac	3m 40s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	31m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	9m 4s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		147m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5820/14/artifact/out/Dockerfile
GITHUB PR	#5820
Optional Tests	dupname asflicense codespell detsecrets shellcheck shelldocs compile cc mvnsite javac unit golang
uname	Linux d5a19f8ee2be 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7fb2222`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5820/14/testReport/
Max. process+thread count	600 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5820/14/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-12-25T11:19:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 2s		trunk passed
+1 💚	compile	3m 41s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	3m 44s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	shadedclient	31m 39s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	3m 35s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	cc	3m 35s		the patch passed
+1 💚	golang	3m 35s		the patch passed
+1 💚	javac	3m 35s		the patch passed
+1 💚	compile	3m 38s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	cc	3m 38s		the patch passed
+1 💚	golang	3m 38s		the patch passed
+1 💚	javac	3m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	31m 19s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	9m 1s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		146m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5820/15/artifact/out/Dockerfile
GITHUB PR	#5820
Optional Tests	dupname asflicense codespell detsecrets shellcheck shelldocs compile cc mvnsite javac unit golang
uname	Linux 43eec0dd8a9a 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `939b07a`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5820/15/testReport/
Max. process+thread count	555 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5820/15/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

GauthamBanasandra · 2023-12-27T18:48:28Z

@goiri , @ayushtkn could you please review my PR?

ayushtkn

Looks good, but the build result in your PR with the result has some failures

Atleast asf licence thing looks like a problem to me. That might create issues for other developers, There is some native test failing as well, is that flaky?

GauthamBanasandra · 2023-12-28T12:13:44Z

Hi @ayushtkn,

Regarding the HDFS Native Client test failures - there are a lot of test failures on Windows. Here's a run from our Nightly pipeline on Windows. We will eventually fix these tests to run on Windows. With the pre-commit CI that this PR proposes, I'm hoping that we catch any regression with the things that are currently passing -
Regarding the ASF license failure - this file isn't introduced by this PR. Upon checking further, it turns out to be a binary file, to which we can't add a license header. Our Nightly CI reports a full list of such files. I think we should just ignore this file from the license check. Pinging the author of this file, @viirya for further thoughts on this.

ayushtkn · 2023-12-28T14:38:00Z

Strange the linux build doesn't report any errors for license
https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1453/console

I'm hoping that we catch any regression with the things that are currently passing

With these many failing tests, how will other developers figure out which one is a regression & which was already broken? You should put the proposal of enabling the Pre-Commit for Windows for each PR on the dev list, I believe & let folks know the current state of the build before enabling it

For the code changes, it LGTM

viirya · 2023-12-28T18:43:31Z

2. Regarding the ASF license failure - this file isn't introduced by this PR. Upon checking further, it turns out to be a binary file, to which we can't add a license header. Our Nightly CI reports a full list of such files. I think we should just ignore this file from the license check. Pinging the author of this file, @viirya for further thoughts on this.

The file (hadoop-common-project/hadoop-common/src/test/resources/lz4/sequencefile) is created to test lz4 compatibility for a lz4 codec change. Its content is simply key/value counting from 0 until 2000. As mentioned this is a binary file so it cannot be added with license header, we can just ignore it from the license check.

GauthamBanasandra · 2023-12-29T11:09:20Z

With these many failing tests, how will other developers figure out which one is a regression & which was already broken?

Thanks for your inputs @ayushtkn. I agree with you regarding this. I'll disable running the unit tests for pre-commit CI for Windows. So that we only catch regressions in the following for now -

* This PR adds a Jenkinsfile for pre-commit CI to validate the Hadoop PRs on Windows 10.

GauthamBanasandra self-assigned this Jul 8, 2023

GauthamBanasandra added build Windows Bugs related to Hadoop on Windows. labels Jul 8, 2023

This comment was marked as outdated.

Sign in to view

GauthamBanasandra added 13 commits December 23, 2023 13:43

Add pipeline steps to build on Windows

582fe89

Run CI

7e61a97

Revert "Run CI"

84e92f9

This reverts commit f75dce5.

Remove stages

2488302

Revert "Remove stages"

a3201a4

This reverts commit e433bb2.

Add a stage

ec5db35

Fix incorrect nesting of Linux and Windows stages

263dbc6

Add stages nesting

ef9d450

Reorder agent

dbc1f45

Fix case

53f314b

Do not clone Hadoop

fd00bf0

Fix Yetus repo URL

976efa2

Add Jenkinsfile for Windows 10

2a801ee

GauthamBanasandra force-pushed the win-precommit branch from 42c3c46 to 2a801ee Compare December 23, 2023 08:27

github-actions bot added trunk Infra labels Dec 23, 2023

GauthamBanasandra added 2 commits December 23, 2023 13:59

Set IS_NIGHTLY_BUILD = 0

d026304

Update Yetus repo

be84959

This comment was marked as outdated.

Sign in to view

Set BUILD_USER

5107bb0

This comment was marked as outdated.

Sign in to view

GauthamBanasandra added 2 commits December 24, 2023 10:11

Run CI

9fb9ba2

Revert "Run CI"

0a78b30

This reverts commit 9fb9ba2.

This comment was marked as outdated.

Sign in to view

GauthamBanasandra added 2 commits December 24, 2023 21:27

Don't use Jenkinsfile

b960533

Publish JUnit results

7fb2222

* Also do Github status recovery.

GauthamBanasandra changed the title ~~[Testing] Setup pre-commit CI for Windows 10~~ HADOOP-19017. Setup pre-commit CI for Windows 10 Dec 24, 2023

GauthamBanasandra added 3 commits December 25, 2023 13:32

Publish JUnit results first

443687e

Use ContainerAdministrator user for permissions

1782407

Pass built-in Jenkins env vars

939b07a

GauthamBanasandra marked this pull request as ready for review December 27, 2023 18:47

goiri approved these changes Dec 27, 2023

View reviewed changes

ayushtkn reviewed Dec 28, 2023

View reviewed changes

GauthamBanasandra merged commit 98656db into apache:trunk Dec 29, 2023

jiajunmao pushed a commit to jiajunmao/hadoop-MLEC that referenced this pull request Feb 6, 2024

HADOOP-19017. Setup pre-commit CI for Windows 10 (apache#5820)

7076743

* This PR adds a Jenkinsfile for pre-commit CI to validate the Hadoop PRs on Windows 10.

GauthamBanasandra mentioned this pull request Mar 25, 2024

HADOOP-19127. Do not run unit tests on Windows pre-commit CI #6672

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19017. Setup pre-commit CI for Windows 10 #5820

HADOOP-19017. Setup pre-commit CI for Windows 10 #5820

Uh oh!

GauthamBanasandra commented Jul 8, 2023 •

edited

Loading

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

hadoop-yetus commented Dec 24, 2023

Uh oh!

hadoop-yetus commented Dec 25, 2023

Uh oh!

GauthamBanasandra commented Dec 27, 2023

Uh oh!

ayushtkn left a comment

Uh oh!

GauthamBanasandra commented Dec 28, 2023 •

edited

Loading

Uh oh!

ayushtkn commented Dec 28, 2023

Uh oh!

viirya commented Dec 28, 2023

Uh oh!

GauthamBanasandra commented Dec 29, 2023

Uh oh!

Uh oh!

HADOOP-19017. Setup pre-commit CI for Windows 10 #5820

HADOOP-19017. Setup pre-commit CI for Windows 10 #5820

Uh oh!

Conversation

GauthamBanasandra commented Jul 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

hadoop-yetus commented Dec 24, 2023

Uh oh!

hadoop-yetus commented Dec 25, 2023

Uh oh!

GauthamBanasandra commented Dec 27, 2023

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

GauthamBanasandra commented Dec 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayushtkn commented Dec 28, 2023

Uh oh!

viirya commented Dec 28, 2023

Uh oh!

GauthamBanasandra commented Dec 29, 2023

Uh oh!

Uh oh!

GauthamBanasandra commented Jul 8, 2023 •

edited

Loading

GauthamBanasandra commented Dec 28, 2023 •

edited

Loading