arm64 support for M1

[pietmlr]

Dear TensorFlow-io team and community,
are there any plans for adding support for arm64 machines? I use a M1 mac and I'm not able to install the wheel. Anyone who's reading this, do you know any alternatives that I can use? My goal is to create a mel-spectogram from a .wav file. ps: M1 cannot use Scipy at this time.

[yongtang]

@redmlr In order to get M1 support I think we needs

1. The M1 version of the tensorflow package (TensorFlow support for Apple Silicon (M1 Chips) tensorflow#44751 provides a way to build tensorflow on M1, though no release package)
2. A M1 build machine (ideally with GitHub Actions) (Add ARM + MacOS target to be able to use on self-hosted runners actions/runner#805)

I don't have access to M1 laptop yet, may take a look after I get one.

[pietmlr]

@yongtang Thank you for answering,
There’s a m1 accelerated version of tensorflow in apples tensorflow repo, which is in alpha version. I’m running the latest version of this and it runs fine but tensorflow-io is a separate library which won’t install.

[yongtang]

@redmlr The tensorflow-io package will need to be re-compiled for all functionalities to work. Though if you are only looking for functions that does not requires C++ code (only need python code) then it should be possible to just package the source tree.

Which function (or list of functions) you intend to use?

[pietmlr]

@yongtang I'm planning to use the following:

tfio.experimental.audio.trim tfio.experimental.audio.spectrogram tfio.experimental.audio.melscale tfio.experimental.audio.dbscale tfio.experimental.audio.freq_mask tfio.experimental.audio.time_mask

[pietmlr]

Thank you @yongtang !
I'm using not that long.. What does just package the source tree mean and how do I do this? I already downloaded the latest release of io and changed the desired file (init.py) as mentioned in #1302

[yongtang]

@redmlr With PR #1302 in you can build the wheel with:

python3 setup.py -q bdist_wheel

The wheel file will be located under dist directory.

You can then install the wheel with:

python3 -m pip install --no-deps <wheel-file-build-from-last-step>

That will at least allows you to use python only functions in tensorflow-io.

Note --no-deps will ignore the dependency of tensorflow package (as the official tensorflow package is different from the apple's tensorflow package).

Note normally the wheel file needs to pass --data bazel-bin

python3 setup.py --data bazel-bin -q bdist_wheel

where --data bazel-bin indicates the location of the complied C++ kernels. However, it is not yet possible to build C++ kernels on M1 so you can skip this part.

Note also the wheel package built from the above will not have file systems support (e.g, azure/etc) as well (file system support also requires compiled C++ kernels).

[TylerADavis]

Are there any updates on this front? I am interested in doing some development on TensorFlow itself, and I'm unable to install my tf-nightly wheel as there's no compatible version of tensorflow-io-gcs on pypi for arm64 macOS. I'll try setting up the io wheel from source, but being able to install straight from pypi would be great. If there's anything I can do on my end to help get this released, I'd be glad to help.

System Setup:
Python 3.9
macOS 12
Apple M1 Pro processor

Steps Taken:

The below steps are how I am building TensorFlow itself, all taken from the official documentation. This stage works without issue.

$ python3.9 -m venv ./tensorflow_39_compile
$ source compile_tensorflow_39/bin/activate
$ pip install -U pip numpy wheel
$ pip install -U keras_preprocessing --no-deps
$ git clone https://github.com/tensorflow/tensorflow.git
$ cd tensorflow
$ ./configure
$ bazel build  //tensorflow/tools/pip_package:build_pip_package
$ ./bazel-bin/tensorflow/tools/pip_package/build_pip_package --nightly_flag /tmp/tensorflow_pkg

Now I try to install the wheel I made, and it fails

$ cd /tmp/tensorflow_pkg
$ pip3 install -v tf_nightly-2.8.0-cp39-cp39-macosx_12_0_arm64.whl
Using pip 21.3.1 from /Users/tylerdavis/tensorflow_39_compile/lib/python3.9/site-packages/pip (python 3.9)
Processing ./tf_nightly-2.8.0-cp39-cp39-macosx_12_0_arm64.whl
Collecting keras-nightly~=2.8.0.dev
  Using cached keras_nightly-2.8.0.dev2021110107-py2.py3-none-any.whl (1.4 MB)
Collecting gast<0.5.0,>=0.2.1
  Using cached gast-0.4.0-py3-none-any.whl (9.8 kB)
Collecting astunparse>=1.6.0
  Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
ERROR: Could not find a version that satisfies the requirement tensorflow-io-gcs-filesystem>=0.21.0 (from tf-nightly) (from versions: none)
ERROR: No matching distribution found for tensorflow-io-gcs-filesystem>=0.21.0

[TylerADavis]

I can confirm that the below let me install my copy of tensorflow built from HEAD:

# From root of io project
$ python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem          
$ cd dist
$ pip3 install tensorflow_io_gcs_filesystem-0.21.0-cp39-cp39-macosx_12_0_arm64.whl
$ cd /tmp/tensorflow_pkg
# Install h5py following the instructions in https://github.com/h5py/h5py/issues/1982
# as pip install h5py is currently broken on arm64 macs
$ pip3 install ./tf_nightly-2.8.0-cp39-cp39-macosx_12_0_arm64.whl

[matteosantama]

I too am looking to install tensorflow-io-gcs-filesystem directly from PyPI.

MacOS Monterey 12.0 Apple M1 Pro

It looks like @TylerADavis has the right commands to produce the wheel. How can we publish it to PyPI?

[admackin]

@matteosantama I think to make the above work out-of-the-box, it would need Github Actions to support ARM runners which is not yet on the roadmap. Also see actions/runner#805

Cross-compilation could work in theory I guess, but the above commands aren't for cross-compiling, and I imagine it will be a little more fiddly to get it going.

[yingshaoxo]

No, it can't be built

(pornstar) yingshaoxo@macos io % python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem 
Project: tensorflow-io-gcs-filesystem
Exclude: ['tests', 'tests.*', 'tensorflow_io', 'tensorflow_io.*']
Install Requires: []
Project Rootpath: tensorflow_io_gcs_filesystem
/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
Traceback (most recent call last):
  File "setup.py", line 138, in <module>
    setuptools.setup(
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/__init__.py", line 155, in setup
    return distutils.core.setup(**attrs)
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 328, in run
    impl_tag, abi_tag, plat_tag = self.get_tag()
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 278, in get_tag
    assert tag in supported_tags, "would build wheel with unsupported tag {}".format(tag)
AssertionError: would build wheel with unsupported tag ('cp38', 'cp38', 'macosx_11_0_arm64')

[albirrkarim]

No, it can't be built

(pornstar) yingshaoxo@macos io % python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem 
Project: tensorflow-io-gcs-filesystem
Exclude: ['tests', 'tests.*', 'tensorflow_io', 'tensorflow_io.*']
Install Requires: []
Project Rootpath: tensorflow_io_gcs_filesystem
/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
Traceback (most recent call last):
  File "setup.py", line 138, in <module>
    setuptools.setup(
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/__init__.py", line 155, in setup
    return distutils.core.setup(**attrs)
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 328, in run
    impl_tag, abi_tag, plat_tag = self.get_tag()
  File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 278, in get_tag
    assert tag in supported_tags, "would build wheel with unsupported tag {}".format(tag)
AssertionError: would build wheel with unsupported tag ('cp38', 'cp38', 'macosx_11_0_arm64')

i got same result