Migration of modular file system plugins in tensorflow (s3/hdfs/gcs) to tensorflow-io

[yongtang]

As was discussed in our September monthly meetings, we are looking into migrate modular file system plugins in tensorflow (s3/hdfs/gcs) to tensorflow-io package.

Once the migration is done the size and build time of tensorflow repo could be substantially improved, this can greatly help external tensorflow contributors experience as they could selectively decide to only build some components (e.g., tensorflow core, tensorflow-io, tensorflow-addons, etc).

Currently it takes at least 8 hours (or even more) for many external tensorflow contributors to build complete tensorflow package due to the lack of bazel cache that is only internally available in google.

Now as we already have a couple of modular file systems in tensorflow-io (azure blob file system and http file system. see #1111), it might be time to start looking into migration of plugins in tensorflow to tensorflow-io.

/cc @mihaimaruseac @vnvo2409

Item list:

• S3 file system (PR Initial commit of s3 modular file system plugin (credit @vnvo2409) #1191)
• GCS file system (PR Initial commit of gcs modular file system plugin (vnvo2409) #1203)
• HDFS file system (PR Initial commit of hadoop file system (credit vnvo2409) #1197)

Follow up:

• [S3 file system] Port s3_filesystem_test.cc to python tests (Initial commit of s3 modular file system plugin (credit @vnvo2409) #1191 (comment)).
• [S3 file system] Bump AWS SDK to 1.8.x (Initial commit of s3 modular file system plugin (credit @vnvo2409) #1191 (comment))
• [S3 file system] Fix multi download issue (Initial commit of s3 modular file system plugin (credit @vnvo2409) #1191 (comment))
• [S3 file system] Enable logging through TensorFlow C API (Initial commit of s3 modular file system plugin (credit @vnvo2409) #1191 (comment))
• [HDFS file system] Port hadoop_filesystem_test.cc to python tests
• [GCS file system] Port gcs_filesystem_test.cc to python tests

[vnghia]

Let me check if there is any incompatible between the core implementation and the plugins.

There are some points which I would like to make it clear.

---

Documentation: There are some issues of TF related to the behavior of the environment variables:

• S3: TF 2.3 S3 client having permission issues with tf.data.TFRecordDataset() tensorflow#44564 Undocumented S3_DISABLE_MULTI_PART_DOWNLOAD variable and its behavior tensorflow#42389
• GCS: Bug?: reading from Google Cloud Storage appears to be accessing cached version tensorflow#15530

Up till now, I don't see much documentation about those filesystems. I would like to have a working example and documentation about other envs. Where should I make a PR to ?

---

Issues: Could we transfer the issues related to those filesystems from TF repo to this repo ?

---

Testing: What we should do about testing ? I think we could set up a s3 and gcs emulator in github actions. Another possibility is tagging those tests manual and run it locally.

[yongtang]

Thanks @vnvo2409 !

For testing: I have created a PR #1184 which setup the AWS Localstack emulator in GitHub Actions. See https://github.com/tensorflow/io/blob/master/tests/test_aws/aws_test.sh for localstack emulator setup.

The test is working for Kinesis dataset in tensorflow-io. I configured the localstack emulator to have s3 and a bare-bone stub test has been created as well as a starting. Please take a look (https://github.com/tensorflow/io/blob/master/tests/test_s3_eager.py)

I will take a look at gcs emulator and see if a similar setup could be done as well.

For issue transfer: Will take a look to see if issue can be transferred here.

For documentation: At the moment the documentation source is inside the docs directory of this repo (https://github.com/tensorflow/io/tree/master/docs). The PRs are reviewed by docs team. PR #1172 is an example.

Once PR is merged the content are being pushed to tensorflow.org website (see https://www.tensorflow.org/io/overview for example).

[vnghia]

For testing: I have created a PR #1184 which setup the AWS Localstack emulator in GitHub Actions.

Thank you!

I will take a look at gcs emulator and see if a similar setup could be done as well.

I think we could use the emulator of google-cloud-cpp, which is the library underlies gcs plugins. The emulator isn't done yet ( It will be done around mid-november ). It is written entirely in python so setup should be quite easy.

I am a little busy working with the emulator of google-cloud-cpp, I will take over this migration when I finish the other work. Is there any timeline for it ?

[yongtang]

@vnvo2409 We don't have a hard timeline for it though we are looking at before TF 2.5 release. Since 2.4 is not released yet (likely in several weeks), the TF 2.5 release will be about early next year. I will try to take a look and see if I can help migrating a plugin or so, and may ask you for review 😄 .

[vnghia]

the TF 2.5 release will be about early next year

Early next year should be enough for me.

I will try to take a look and see if I can help migrating a plugin or so, and may ask you for review 😄 .

Thank you very much 😄

[mihaimaruseac]

I could help with moving issues.

Thank you for making the tracking issue to move the plugins.

[yongtang]

@mihaimaruseac One issue I noticed is that at the moment there is no way to unload the existing file systems embedded in TF. For example, if I am trying to register a shared object plugin for S3, the following error will thrown out because there is already a s3 registered:

/Library/Python/3.8/site-packages/tensorflow/python/framework/load_library.py:178: in register_filesystem_plugin
    py_tf.TF_RegisterFilesystemPlugin(plugin_location)
E   tensorflow.python.framework.errors_impl.AlreadyExistsError: File system for s3 already registered

It looks like the error arises from the code in https://github.com/tensorflow/tensorflow/blob/dc3099c444d294b39cd79fe1d1a4bff59a0c6180/tensorflow/core/platform/env.cc#L82-L90

I am wondering if it makes sense to allow plugin registration to override existing static built file systems (s3/gcs/hdf5)? I think that will allow user to have more flexibilities. (We can lock local, unix file system to never be overridden).

[mihaimaruseac]

We should comment the TF_REGISTER_FILESYSTEM lines in the old code for every filesystem that is integrally moved to plugins. In fact, we can also delete the non-modular implementation.

It will be a risky thing to allow overriding filesystem registrations.

[kvignesh1420]

@vnvo2409 @mihaimaruseac what is the current approach for testing hdfs plugins in tensorflow?

[vnghia]

AFAIK, we could run hdfs locally in pseudo-distributed mode and it should be enough for testing

[kvignesh1420]

how about hdfs on docker for automated testing in github actions?

[vnghia]

I find this tutorial. I think it should work. I will try it and will report to you later.

[kvignesh1420]

Okay. Thanks!