Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS: fix bugs around using S3FileIO for table operations #1749

Merged
merged 1 commit into from
Nov 11, 2020

Conversation

jackye1995
Copy link
Contributor

When used in BaseMetastoreTableOperations, in TableMetadataParser.internalWrite(), an output stream is closed twice because of the closure of the output stream and its writer:

try (OutputStream ou = isGzip ? new GZIPOutputStream(stream) : stream;
         OutputStreamWriter writer = new OutputStreamWriter(ou, StandardCharsets.UTF_8)) {
   ...
}

This causes issue in S3OutputStream because the stream uploads the file from local to S3 during the call to close.

Also, S3FileIO cannot be properly initialized when there are multiple HTTP client builders available in the Java class path at runtime, so AwsClientUtil is used to provide a stable default client for S3FileIO to use.

Copy link
Contributor

@rdblue rdblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but I'd like to make sure @danielcweeks agrees.

@danielcweeks
Copy link
Contributor

+1 LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants