HDFS-14869 Copy renamed files which are not excluded anymore by filter #1530

aasha · 2019-09-26T10:17:21Z

No description provided.

hadoop-yetus · 2019-09-26T11:45:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	124	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1354	trunk passed
+1	compile	25	trunk passed
+1	checkstyle	20	trunk passed
+1	mvnsite	28	trunk passed
+1	shadedclient	837	branch has no errors when building and testing our client artifacts.
+1	javadoc	21	trunk passed
0	spotbugs	41	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	40	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	25	the patch passed
+1	compile	19	the patch passed
+1	javac	19	the patch passed
-0	checkstyle	15	hadoop-tools/hadoop-distcp: The patch generated 4 new + 47 unchanged - 0 fixed = 51 total (was 47)
+1	mvnsite	22	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	855	patch has no errors when building and testing our client artifacts.
+1	javadoc	19	the patch passed
+1	findbugs	44	the patch passed
		_ Other Tests _
+1	unit	871	hadoop-distcp in the patch passed.
+1	asflicense	44	The patch does not generate ASF License warnings.
		4475

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/1/artifact/out/Dockerfile
GITHUB PR	#1530
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux cb0035532a13 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `7b6219a`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/1/testReport/
Max. process+thread count	347 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran

I do not know enough about distCp to be safe to review this.

I've had a quick glance at the diff and suggested changes to the test based on what I like to see in tests, but someone else is going to need to have to look at the core algorithm.

Sorry

hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java

hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java

bshashikant · 2019-11-11T10:23:29Z

Thanks @aasha for working on this. The changes overall look good to me. I have one comment.
Can we also modify the test to include a case of rename, where the filter excludes the rename target but the source file is included which after the distcp completes should be deleted on the target?

bshashikant · 2019-11-13T12:39:31Z

/retest

steveloughran · 2019-11-17T19:02:23Z

LGTM, we just need yetus to see it. can you do a rebase + force push to see if that works?

bshashikant · 2019-11-21T11:05:12Z

@steveloughran , even an empty commit seems to not trigger yetus. Any other way to trigger this?
Can the patch be just uploaded into the jira to see if yetus picks up?

hadoop-yetus · 2019-12-04T15:47:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 12s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 9s	trunk passed
+1 💚	compile	0m 24s	trunk passed
+1 💚	checkstyle	0m 20s	trunk passed
+1 💚	mvnsite	0m 27s	trunk passed
+1 💚	shadedclient	14m 55s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 21s	trunk passed
+0 🆗	spotbugs	0m 41s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 39s	trunk passed
-0 ⚠️	patch	0m 54s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s	the patch passed
+1 💚	compile	0m 19s	the patch passed
+1 💚	javac	0m 19s	the patch passed
-0 ⚠️	checkstyle	0m 14s	hadoop-tools/hadoop-distcp: The patch generated 10 new + 47 unchanged - 0 fixed = 57 total (was 47)
+1 💚	mvnsite	0m 22s	the patch passed
-1 ❌	whitespace	0m 0s	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	shadedclient	14m 52s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 18s	the patch passed
+1 💚	findbugs	0m 43s	the patch passed
		_ Other Tests _
+1 💚	unit	12m 42s	hadoop-distcp in the patch passed.
+1 💚	asflicense	0m 28s	The patch does not generate ASF License warnings.
		71m 16s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/2/artifact/out/Dockerfile
GITHUB PR	#1530
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 8f01e99a911b 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `47fdae7`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/2/artifact/out/whitespace-eol.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/2/testReport/
Max. process+thread count	314 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-12-05T09:28:35Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 26s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	22m 22s	trunk passed
+1 💚	compile	0m 30s	trunk passed
+1 💚	checkstyle	0m 27s	trunk passed
+1 💚	mvnsite	0m 37s	trunk passed
+1 💚	shadedclient	17m 30s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 26s	trunk passed
+0 🆗	spotbugs	0m 57s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 55s	trunk passed
-0 ⚠️	patch	1m 16s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s	the patch passed
+1 💚	compile	0m 23s	the patch passed
+1 💚	javac	0m 23s	the patch passed
-0 ⚠️	checkstyle	0m 19s	hadoop-tools/hadoop-distcp: The patch generated 10 new + 47 unchanged - 0 fixed = 57 total (was 47)
+1 💚	mvnsite	0m 28s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	17m 55s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 22s	the patch passed
+1 💚	findbugs	0m 55s	the patch passed
		_ Other Tests _
+1 💚	unit	16m 31s	hadoop-distcp in the patch passed.
+1 💚	asflicense	0m 36s	The patch does not generate ASF License warnings.
		86m 7s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/3/artifact/out/Dockerfile
GITHUB PR	#1530
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux fe4151e5f851 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `da9aa9c`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/3/testReport/
Max. process+thread count	303 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

bshashikant · 2019-12-05T12:33:29Z

@aasha , can you please address the checkstyle issues?

hadoop-yetus · 2019-12-06T11:43:18Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 7s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 6s	trunk passed
+1 💚	compile	0m 24s	trunk passed
+1 💚	checkstyle	0m 23s	trunk passed
+1 💚	mvnsite	0m 27s	trunk passed
+1 💚	shadedclient	14m 32s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 20s	trunk passed
+0 🆗	spotbugs	0m 45s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 43s	trunk passed
-0 ⚠️	patch	1m 0s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s	the patch passed
+1 💚	compile	0m 19s	the patch passed
+1 💚	javac	0m 19s	the patch passed
+1 💚	checkstyle	0m 14s	the patch passed
+1 💚	mvnsite	0m 22s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	14m 43s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 18s	the patch passed
+1 💚	findbugs	0m 43s	the patch passed
		_ Other Tests _
+1 💚	unit	12m 44s	hadoop-distcp in the patch passed.
+1 💚	asflicense	0m 28s	The patch does not generate ASF License warnings.
		70m 40s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/4/artifact/out/Dockerfile
GITHUB PR	#1530
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 8127bfe342ab 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `e575df5`
Default Java	1.8.0_222
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/4/testReport/
Max. process+thread count	309 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1530/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

apache#1530)

…ore by filter (apache#1530) (cherry picked from commit fccccc9) Change-Id: I6acd2358f438849311c2e31e489bfdb5447fbefc

HDFS-14869 Copy renamed files which are not excluded by filter

3f9d8f4

steveloughran requested changes Oct 10, 2019

View reviewed changes

Resolve review comments

a601ad8

jojochuang added the HDFS label Oct 29, 2019

Resolve review comments

713b08c

aasha requested a review from steveloughran November 12, 2019 06:07

bshashikant approved these changes Nov 13, 2019

View reviewed changes

bshashikant assigned aasha Nov 13, 2019

Resolve Review Comments

fba30be

aasha force-pushed the HDFS-14869 branch from 94fd86f to fba30be Compare November 19, 2019 10:05

aasha requested a review from bshashikant November 19, 2019 10:09

Empty commit

1e6a27a

bshashikant approved these changes Nov 21, 2019

View reviewed changes

Aasha Medhi added 2 commits December 5, 2019 11:22

Fix whitespace

5e9ddc8

Fix whitespace

445243e

Fix Checkstyle

4343c38

bshashikant merged commit fccccc9 into apache:trunk Dec 6, 2019

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HDFS-14869 Copy renamed files which are not excluded anymore by filter (

1e5e279

apache#1530)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-14869 Copy renamed files which are not excluded anymore by filter #1530

HDFS-14869 Copy renamed files which are not excluded anymore by filter #1530

Uh oh!

aasha commented Sep 26, 2019 •

edited

Loading

Uh oh!

hadoop-yetus commented Sep 26, 2019

Uh oh!

steveloughran left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bshashikant commented Nov 11, 2019

Uh oh!

bshashikant commented Nov 13, 2019

Uh oh!

steveloughran commented Nov 17, 2019

Uh oh!

bshashikant commented Nov 21, 2019

Uh oh!

hadoop-yetus commented Dec 4, 2019

Uh oh!

hadoop-yetus commented Dec 5, 2019

Uh oh!

bshashikant commented Dec 5, 2019

Uh oh!

hadoop-yetus commented Dec 6, 2019

Uh oh!

Uh oh!

HDFS-14869 Copy renamed files which are not excluded anymore by filter #1530

HDFS-14869 Copy renamed files which are not excluded anymore by filter #1530

Uh oh!

Conversation

aasha commented Sep 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Sep 26, 2019

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bshashikant commented Nov 11, 2019

Uh oh!

bshashikant commented Nov 13, 2019

Uh oh!

steveloughran commented Nov 17, 2019

Uh oh!

bshashikant commented Nov 21, 2019

Uh oh!

hadoop-yetus commented Dec 4, 2019

Uh oh!

hadoop-yetus commented Dec 5, 2019

Uh oh!

bshashikant commented Dec 5, 2019

Uh oh!

hadoop-yetus commented Dec 6, 2019

Uh oh!

Uh oh!

aasha commented Sep 26, 2019 •

edited

Loading