HADOOP-19256. S3A: Support Conditional Writes

[steveloughran]

HADOOP-19256 for branch-3.4

Merge pain because branch-3.4 still supports deleting directory markers.
I wonder if we could backport that.

---

Amazon S3 now supports conditional overwrites, which can be be used when creating files through the createFile() API with two new builder options:

      fs.option.create.conditional.overwrite

Write if and only if there is no object at the target path. This is an atomic PUT-no-overwrite, checked in close(), not create().

      fs.option.create.conditional.overwrite.etag

Write a file if and only if it is overwriting a file with a specific etag.

If the "fs.s3a.performance.flags" enumeration includes the flag "create" then file creation will use conditional creation to detect and reject overwrites.

The configuration option "fs.s3a.create.conditional.enabled" can be set to false to disable these features on third-party stores.

Contributed by Diljot Grewal, Saikat Roy and Steve Loughran

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[steveloughran]

s3 london, all but the signing problem and those AAL ones that Ahmar is working on.
If nothing major is shown up as a regression on the PR, will merge

[ERROR]   ITestS3AContractAnalyticsStreamVectoredRead>AbstractContractVectoredReadTest.testVectoredReadAfterNormalRead:490 » IO
[ERROR]   ITestS3AContractAnalyticsStreamVectoredRead>AbstractContractVectoredReadTest.testVectoredReadAfterNormalRead:490 » IO
[ERROR]   ITestS3AConfiguration.testS3SpecificSignerOverride:600 » SdkClient Unable to l...
[INFO] 

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ branch-3.4 Compile Tests _
+0 🆗	mvndep	1m 53s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 47s		branch-3.4 passed
+1 💚	compile	16m 32s		branch-3.4 passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	15m 7s		branch-3.4 passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚	checkstyle	4m 11s		branch-3.4 passed
+1 💚	mvnsite	2m 32s		branch-3.4 passed
+1 💚	javadoc	1m 53s		branch-3.4 passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 43s		branch-3.4 passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚	spotbugs	3m 48s		branch-3.4 passed
+1 💚	shadedclient	33m 23s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 24s		the patch passed
+1 💚	compile	15m 47s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	15m 47s		the patch passed
+1 💚	compile	15m 6s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚	javac	15m 6s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 8 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	4m 5s	/results-checkstyle-root.txt	root: The patch generated 51 new + 23 unchanged - 0 fixed = 74 total (was 23)
+1 💚	mvnsite	2m 30s		the patch passed
+1 💚	javadoc	1m 49s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 42s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚	spotbugs	4m 9s		the patch passed
+1 💚	shadedclient	33m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 44s		hadoop-common in the patch passed.
+1 💚	unit	3m 1s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		238m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7594/1/artifact/out/Dockerfile
GITHUB PR	#7594
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux f08f35e39da8 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / adfaeb3
Default Java	Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7594/1/testReport/
Max. process+thread count	3150 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7594/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.