HADOOP-16848. Experimental directory marker optimization. #1834

steveloughran · 2020-02-06T15:12:20Z

Optimizes directory marker operations through:-

Async create/clear
An option "fs.s3a.experimental.optimized.directory.operations"

That performs whatever optimisations to directory marker IO are considered
effective to reduce that I while still avoiding problems with missing markers
and/or markers not deleted.

Important

Work in progress
Experimental: use at own risk
Not for use against buckets whether other S3A releases are active
And not for use with data you value.

View it as a Proof of Concept to explore what we can do here.

steveloughran · 2020-02-06T15:15:19Z

Tested. Ireland

Tests failed; not looked into the details

some were clearly as the #of IOPs were less than the metrics' assertions
and rename() contract tests were failing with, as usual. nothing meaningful from rename() other than "false"

hadoop-yetus · 2020-02-06T17:31:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 33s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 36s	Maven dependency ordering for branch
+1 💚	mvninstall	26m 3s	trunk passed
+1 💚	compile	19m 33s	trunk passed
+1 💚	checkstyle	2m 52s	trunk passed
+1 💚	mvnsite	2m 6s	trunk passed
+1 💚	shadedclient	20m 51s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 1s	trunk passed
+0 🆗	spotbugs	1m 13s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 32s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 28s	the patch passed
+1 💚	compile	17m 59s	the patch passed
+1 💚	javac	17m 59s	the patch passed
-0 ⚠️	checkstyle	2m 57s	root: The patch generated 1 new + 25 unchanged - 0 fixed = 26 total (was 25)
+1 💚	mvnsite	2m 16s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	14m 26s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 4s	the patch passed
+1 💚	findbugs	3m 42s	the patch passed
		_ Other Tests _
-1 ❌	unit	9m 24s	hadoop-common in the patch failed.
+1 💚	unit	1m 34s	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 45s	The patch does not generate ASF License warnings.
		138m 7s

Reason	Tests
Failed junit tests	hadoop.security.TestFixKerberosTicketOrder
	hadoop.security.TestRaceWhenRelogin

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/1/artifact/out/Dockerfile
GITHUB PR	#1834
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux dc58b925e539 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `146ca0f`
Default Java	1.8.0_232
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/1/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/1/testReport/
Max. process+thread count	1334 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-02-06T17:40:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	25m 15s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 13s	Maven dependency ordering for branch
+1 💚	mvninstall	18m 29s	trunk passed
+1 💚	compile	16m 47s	trunk passed
+1 💚	checkstyle	2m 33s	trunk passed
+1 💚	mvnsite	2m 14s	trunk passed
+1 💚	shadedclient	18m 49s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 19s	trunk passed
+0 🆗	spotbugs	1m 10s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 8s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 19s	the patch passed
+1 💚	compile	15m 51s	the patch passed
+1 💚	javac	15m 51s	the patch passed
-0 ⚠️	checkstyle	2m 36s	root: The patch generated 1 new + 25 unchanged - 0 fixed = 26 total (was 25)
+1 💚	mvnsite	2m 14s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	12m 49s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 13s	the patch passed
+1 💚	findbugs	3m 29s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 6s	hadoop-common in the patch passed.
+1 💚	unit	1m 39s	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		143m 46s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/2/artifact/out/Dockerfile
GITHUB PR	#1834
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c84056736665 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `146ca0f`
Default Java	1.8.0_242
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/2/artifact/out/diff-checkstyle-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/2/testReport/
Max. process+thread count	1444 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Optimizes directory marker operations through:- * Async create/clear * An option "fs.s3a.experimental.optimized.directory.operations" That performs whatever optimisations to directory marker IO are considered effective to reduce that I while still avoiding problems with missing markers and/or markers not deleted. 1. Work in progress 2. Experimental: use at own risk 3. Not for use against buckets whether other S3A releases are active 4. And not for use with data you value. View it as a Proof of Concept to explore what we can do here. Change-Id: Ia0dadc5e513db00c650376769e37d2ef1fab12f5

hadoop-yetus · 2020-02-14T20:01:29Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 54s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	26m 32s	trunk passed
+1 💚	compile	0m 43s	trunk passed
+1 💚	checkstyle	0m 31s	trunk passed
+1 💚	mvnsite	0m 41s	trunk passed
+1 💚	shadedclient	18m 49s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 27s	trunk passed
+0 🆗	spotbugs	1m 8s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 5s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s	the patch passed
+1 💚	compile	0m 32s	the patch passed
+1 💚	javac	0m 32s	the patch passed
-0 ⚠️	checkstyle	0m 22s	hadoop-tools/hadoop-aws: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12)
+1 💚	mvnsite	0m 38s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	18m 8s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 28s	the patch passed
+1 💚	findbugs	1m 18s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 47s	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 33s	The patch does not generate ASF License warnings.
		76m 36s

Subsystem	Report/Notes
Docker	Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/3/artifact/out/Dockerfile
GITHUB PR	#1834
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 8ce8b8fb1c88 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / c75756f
Default Java	1.8.0_232
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/3/testReport/
Max. process+thread count	337 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2020-04-27T15:35:07Z

superceded by #1861

steveloughran added enhancement fs/s3 changes related to hadoop-aws; submitter must declare test endpoint work in progress PRs still Work in Progress; reviews not expected but still welcome labels Feb 6, 2020

steveloughran force-pushed the s3/HADOOP-16846-experimental-directory branch from 8c134f1 to 2583d35 Compare February 6, 2020 15:15

steveloughran force-pushed the s3/HADOOP-16846-experimental-directory branch from 2583d35 to 85e0f97 Compare February 14, 2020 18:43

steveloughran closed this Apr 27, 2020

steveloughran deleted the s3/HADOOP-16846-experimental-directory branch October 15, 2021 19:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-16848. Experimental directory marker optimization. #1834

HADOOP-16848. Experimental directory marker optimization. #1834

Uh oh!

steveloughran commented Feb 6, 2020

Uh oh!

steveloughran commented Feb 6, 2020

Uh oh!

hadoop-yetus commented Feb 6, 2020

Uh oh!

hadoop-yetus commented Feb 6, 2020

Uh oh!

hadoop-yetus commented Feb 14, 2020

Uh oh!

steveloughran commented Apr 27, 2020

Uh oh!

Uh oh!

HADOOP-16848. Experimental directory marker optimization. #1834

HADOOP-16848. Experimental directory marker optimization. #1834

Uh oh!

Conversation

steveloughran commented Feb 6, 2020

Optimizes directory marker operations through:-

Important

Uh oh!

steveloughran commented Feb 6, 2020

Uh oh!

hadoop-yetus commented Feb 6, 2020

Uh oh!

hadoop-yetus commented Feb 6, 2020

Uh oh!

hadoop-yetus commented Feb 14, 2020

Uh oh!

steveloughran commented Apr 27, 2020

Uh oh!

Uh oh!