Skip to content

HADOOP-17817. Throw an exception if S3 client-side encryption is enabled on S3Guard enabled bucket #3239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 28, 2021
Merged

HADOOP-17817. Throw an exception if S3 client-side encryption is enabled on S3Guard enabled bucket #3239

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eec1937
HADOOP-17817. Throw an exception if S3 client-side encryption is enab…
Jul 27, 2021
40f1a6a
HADOOP-17817. review comments
Jul 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,6 +539,10 @@ public void initialize(URI name, Configuration originalConf)
if (hasMetadataStore()) {
LOG.debug("Using metadata store {}, authoritative store={}, authoritative path={}",
getMetadataStore(), allowAuthoritativeMetadataStore, allowAuthoritativePaths);
if (isCSEEnabled) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initialize is allowed to throw an IOE; our code tends to expect it.

Can you throw a PathIOE(uri, "S3-CSE cannot be used with S3Guard")

i.e give bucket and problem, but don't propose a solution...leave that to the docs as disabling s3guard is probably the better one, depending on what the user wants

throw new PathIOException(uri.toString(), "S3-CSE cannot be used "
+ "with S3Guard");
}
}

// LOG if S3Guard is disabled on the warn level set in config
Expand Down
1 change: 1 addition & 0 deletions hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/encryption.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -601,6 +601,7 @@ clients where S3-CSE has not been enabled.

### Limitations

- S3Guard is not supported with S3-CSE.
- Performance will be reduced. All encrypt/decrypt is now being done on the
client.
- Writing files may be slower, as only a single block can be encrypted and
Expand Down
25 changes: 25 additions & 0 deletions hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1435,6 +1435,31 @@ The user trying to use the KMS Key ID should have the right permissions to acces
If not, then add permission(or IAM role) in "Key users" section by selecting the
AWS-KMS CMK Key on AWS console.

### S3-CSE cannot be used with S3Guard

S3-CSE not supported for S3Guard enabled buckets.
```
org.apache.hadoop.fs.PathIOException: `s3a://test-bucket': S3-CSE cannot be used with S3Guard
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:543)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3460)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3565)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3512)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:539)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:366)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:342)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:252)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:235)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:105)
at org.apache.hadoop.fs.shell.Command.run(Command.java:179)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:95)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
```
If you want to use S3Guard then disable S3-CSE or disable S3Guard if you want
to use S3-CSE.

### <a name="not_all_bytes_were_read"></a> Message appears in logs "Not all bytes were read from the S3ObjectInputStream"


Expand Down