Allow Server-Side Encryption KMS option when uploading objects to S3 with Tensorflow

[pranavpsv]

Note: The original issue was posted on the Tensorflow repository and I was rerouted to this repository by @mihaimaruseac since the S3 filesystem is moving to tensorflow/io.

The Problem

Productive systems frequently impose a requirement to use server-side encryption (SSE) to protect all data stored on S3, e.g. via AWS KMS.

Although Tensorflow (and tensorboard) has an integration with S3, it does not seem to support specifying SSE options for write operations. This is a problem as we cannot write logs to S3 buckets (which impose the SSE requirement) using Tensorboard currently.

Is there a way in to specify SSE options for file uploads to S3 when the S3 filesystem is moved to IO?

If not, I'd suggest to add such a feature, e.g. by exposing an environment variable for extra arguments to S3 file upload operations (S3_UPLOAD_EXTRA_ARGS) which allows to specify SSE options.

Current behaviour in this scenario is to issue an error:

tensorflow.python.framework.errors_impl.FailedPreconditionError: AWS Credentials have not been set properly. Unable to access the specified S3 location [Op:CreateSummaryFileWriter]

when following this example but using an S3 bucket with SSE enabled.

Who will benefit from this feature?

Any users of Tensorflow in production systems as they will typically use server-side encryption (e.g. kms) with S3.

[vnghia]

I think we have to put this issue on hold for some time. I think we could take a look at it after the moving of s3 filesystem is done ( probably in TF 2.6 ).

[pranavpsv]

@vnvo2409 I see, thank you for the update!