Tensorflow 2.13.0 cannot be imported after install

[dre-hh]

• Poetry version: Poetry (version 1.5.1)
• Python version: Python 3.11.4
• OS version and name: macOS Ventura 13.3 M2 Chip macOS-13.3-arm64-arm-64bit
• pyproject.toml: pyproject.toml

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

After installing tensorflow 2.13.0 with above pyproject.toml on macOs Apple Sillicon , importing fails with

ModuleNotFoundError: No module named 'tensorflow

Same method works under linux.
HINT: in tensorflow 2.13 the authors introduce platform specific install for pip. Under macOs arm it is supposed to install
tensorflow-macos . under linux there is another dependency tree

[dimbleby]

tensorflow's metadata does not express that dependency:

$ curl -s https://pypi.org/pypi/tensorflow/2.13.0/json | jq '.info.requires_dist'
[
  "absl-py (>=1.0.0)",
  "astunparse (>=1.6.0)",
  "flatbuffers (>=23.1.21)",
  "gast (<=0.4.0,>=0.2.1)",
  "google-pasta (>=0.1.1)",
  "h5py (>=2.9.0)",
  "libclang (>=13.0.0)",
  "numpy (<=1.24.3,>=1.22)",
  "opt-einsum (>=2.3.2)",
  "packaging",
  "protobuf (!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3)",
  "setuptools",
  "six (>=1.12.0)",
  "termcolor (>=1.1.0)",
  "typing-extensions (<4.6.0,>=3.6.6)",
  "wrapt (>=1.11.0)",
  "grpcio (<2.0,>=1.24.3)",
  "tensorboard (<2.14,>=2.13)",
  "tensorflow-estimator (<2.14,>=2.13.0)",
  "keras (<2.14,>=2.13.1)",
  "tensorflow-io-gcs-filesystem (>=0.23.1) ; platform_machine != \"arm64\" or platform_system != \"Darwin\""
]

No mention here of tensorflow-macos.

I guess this is another variation on tensorflow/tensorflow#58674

please encourage the tensorflow folk to publish consistent metadata in all of their wheels, with platform-specific variations described by markers

[oldfielj-ansto]

@dre-hh Have using environment markers to get around the issue?

If we take a look at tensorflow/tools/pip_package/setup.py, we see that they require the tensorflow-macos package where the environment marker has been satisfied.

It's generated, but would look something like this.
tensorflow-macos == 2.15.0 ;platform_system=="Darwin" and platform_machine=="arm64"

Following the Poetry documentation here, your dependency specification should look this.

[tool.poetry.dependencies]
python = ">=3.11,<3.12"
tensorflow-macos = { version = "^2.15.0", markers = "platform_system=='Darwin' and platform_machine=='arm64'" }
tensorflow = "^2.13.0"

I'd tend to agree with @dimbleby, the devs over at @tensorflow should improve the compatibility of their project with package managers, but that's another battle entirely.

[Wirg]

For people still looking for a full combo workaround, this should work for tensorflow 2.11 in all situations (in reality I haven't tested everything).
Don't hesitate to comment with your problems so that we improve on this.
I've tested :

• new apple silicon
• docker in new apple silicon
• standard linux

This should work with windows, new mac, old mac, docker in new mac, amd and non-amd linux machines ...
EDIT :
Tested :

• windows by @Vuizur

Currently, I advise to copy the full code below in your pyproject.toml in the poetry's dependencies.
Using a poetry group, [tool.poetry.group.tensorflow.dependencies] section could be a good thing to be able to isolate it :

# Issue between poetry and tensorflow metadata since >=2.11
# This is a temporary workaround
# related to https://github.com/python-poetry/poetry/issues/8271
# Inspired from https://github.com/tensorflow/tensorflow/blob/adb39b04e9cb116df4659a7e2de9eea27e62f25c/tensorflow/tools/pip_package/setup.py#L148-L162
tensorflow = {version = "^2.13.0" }
tensorflow-macos = { version = "^2.13.0", platform = "darwin", markers = "platform_machine=='arm64'" }
tensorflow-intel = { version = "^2.13.0", platform = "win32" }
tensorflow-cpu = [
    { version = "^2.13.0", platform = "linux", markers = "platform_machine!='arm64' and platform_machine!='aarch64'" },
    { version = "^2.13.0", platform = "darwin", markers = "platform_machine!='arm64' and platform_machine!='aarch64'" },
]
tensorflow-cpu-aws = { version = "^2.13.0", platform = "linux", markers = "platform_machine=='arm64' or platform_machine=='aarch64'" }
# https://github.com/tensorflow/tensorflow/blob/adb39b04e9cb116df4659a7e2de9eea27e62f25c/tensorflow/tools/pip_package/setup.py#L107-L108
# https://github.com/python-poetry/poetry/issues/8271#issuecomment-1697740447
tensorflow-io-gcs-filesystem = [
    { version = ">= 0.23.1", markers = "platform_machine!='arm64' or platform_system!='Darwin'" },
    { version = "< 0.32.0", markers = "platform_system == 'Windows'" }
]

EDIT : I fixed above the tensorflow-io-gcs-filesystem issue from @RRiva
EDIT 2 : adding @radoering answer
EDIT 3 : fixing windows from @Vuizur

[RRiva]

Thanks a lot for the suggestion. Poetry generated the lock file, but when I test my package on Windows the installation fails with

Installing tensorflow-io-gcs-filesystem (0.33.0)
  RuntimeError
  Unable to find installation candidates for tensorflow-io-gcs-filesystem (0.33.0)

It's not entirely Poetry's fault, as this package doesn't provide wheels for Windows on PyPi, but still...

[Wirg]

Hi @RRiva ,

I went back to the tensorflow file and added what seems to be their way to specify "tensorflow-io-gcs-filesystem"
Can you try again and confirm it works ?

Still a temporary fix. Wondering about publishing something like a package with just a pyproject.toml to avoid everybody having to fix it. I am afraid it will made both side lazy. :/

@dimbleby you seem to better grasp what's up. Do you have any documentation or an issue that will give an initial roadway on how to fix this ?

[RRiva]

Thanks again 🙂 Unfortunately, the lock file is the same as before, and therefore the installation fails in the same way. Might be related to #1616.

[radoering]

#1616 is about different path dependencies with the same version (and resolved), thus, not related.

It's related to #2012 and #4046. If there is no wheel for Windows (and no sdist), you'll want to exclude that version for Windows:

tensorflow-io-gcs-filesystem = [
    { version = ">= 0.23.1", markers = "platform_machine!='arm64' or platform_system!='Darwin'" },
    { version = "< 0.32.0", markers = "platform_system == 'Windows'" }
]

[Wirg]

@radoering adding it to my previous answering and citing this to have a standalone.

[Vuizur]

Unfortunately the workaround by Wirg and radoering still does not seem to work on Windows.

The pyproject.toml:

[tool.poetry]
name = "tt2"
version = "0.1.0"
description = ""
authors = ["Hannes Krumbiegel <Vuizur@users.noreply.github.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10,<3.12"
# Issue between poetry and tensorflow metadata since >=2.11
# This is a temporary workaround
# related to https://github.com/python-poetry/poetry/issues/8271
# Inspired from https://github.com/tensorflow/tensorflow/blob/adb39b04e9cb116df4659a7e2de9eea27e62f25c/tensorflow/tools/pip_package/setup.py#L148-L162
tensorflow = {version = "^2.13.0" }
tensorflow-macos = { version = "^2.13.0", platform = "darwin", markers = "platform_machine=='arm64'" }
tensorflow-intel = { version = "^2.13.0", platform = "windows" }
tensorflow-cpu = [
    { version = "^2.13.0", platform = "linux", markers = "platform_machine!='arm64' and platform_machine!='aarch64'" },
    { version = "^2.13.0", platform = "darwin", markers = "platform_machine!='arm64' and platform_machine!='aarch64'" },
]
tensorflow-cpu-aws = { version = "^2.13.0", platform = "linux", markers = "platform_machine=='arm64' or platform_machine=='aarch64'" }
# https://github.com/tensorflow/tensorflow/blob/adb39b04e9cb116df4659a7e2de9eea27e62f25c/tensorflow/tools/pip_package/setup.py#L107-L108
# https://github.com/python-poetry/poetry/issues/8271#issuecomment-1697740447
tensorflow-io-gcs-filesystem = [
    { version = ">= 0.23.1", markers = "platform_machine!='arm64' or platform_system!='Darwin'" },
    { version = "< 0.32.0", markers = "platform_system == 'Windows'" }
]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

causes an error with:

Creating virtualenv tt2 in D:\Programs\tt2\.venv
Updating dependencies
Resolving dependencies...

Package operations: 39 installs, 1 update, 0 removals

  • Installing certifi (2023.7.22)
  • Installing charset-normalizer (3.2.0)
  • Installing idna (3.4)
  • Installing pyasn1 (0.5.0)
  • Installing urllib3 (1.26.16)
  • Installing cachetools (5.3.1)
  • Installing oauthlib (3.2.2)
  • Installing rsa (4.9)
  • Installing requests (2.31.0)
  • Installing pyasn1-modules (0.3.0)
  • Installing six (1.16.0)
  • Installing google-auth (2.22.0)
  • Installing markupsafe (2.1.3)
  • Installing requests-oauthlib (1.3.1)
  • Installing absl-py (1.4.0)
  • Installing google-auth-oauthlib (1.0.0)
  • Installing numpy (1.24.3)
  • Installing markdown (3.4.4)
  • Installing tensorboard-data-server (0.7.1)
  • Updating setuptools (68.0.0 -> 68.2.0)
  • Installing protobuf (4.24.3)
  • Installing grpcio (1.58.0)
  • Installing werkzeug (2.3.7)
  • Installing astunparse (1.6.3)
  • Installing flatbuffers (23.5.26)
  • Installing h5py (3.9.0)
  • Installing google-pasta (0.2.0)
  • Installing gast (0.4.0)
  • Installing keras (2.13.1)
  • Installing tensorflow-io-gcs-filesystem (0.34.0)
  • Installing packaging (23.1)
  • Installing tensorflow-estimator (2.13.0)
  • Installing libclang (16.0.6)
  • Installing wrapt (1.15.0)
  • Installing typing-extensions (4.5.0)
  • Installing termcolor (2.3.0)
  • Installing tensorboard (2.13.0)
  • Installing opt-einsum (3.3.0)
  • Installing tensorflow-io-gcs-filesystem (0.31.0)

  RuntimeError

  Unable to find installation candidates for tensorflow-io-gcs-filesystem (0.34.0)

  at ~\.local\pipx\venvs\poetry\Lib\site-packages\poetry\installation\chooser.py:76 in choose_for
       72│ 
       73│             links.append(link)
       74│
       75│         if not links:
    →  76│             raise RuntimeError(f"Unable to find installation candidates for {package}")
       77│
       78│         # Get the best link
       79│         chosen = max(links, key=lambda link: self._sort_key(package, link))
       80│

(For both Python 3.10 and 3.11 I get an error like this.)

I got the idea to upgrade Poetry from 3.5 to 3.6 or so, and the install seems to work:

poetry install
Updating dependencies
Resolving dependencies...

Package operations: 1 install, 0 updates, 0 removals      

  • Installing tensorflow (2.13.0)

Writing lock file

Installing the current project: tt2 (0.1.0)

But when I try to execute a module with import tensorflow, I get:

  File "d:\Programs\tt2\tt2\test.py", line 1, in <module>
    import tensorflow
ModuleNotFoundError: No module named 'tensorflow'

[Vuizur]

I found the solution:
Wirg's solution actually specified the wrong platform for pytorch-intel, one doesn't have to write windows, but win32.
Here is a corrected version:

[tool.poetry.dependencies]
python = "^3.10,<3.12"
# Issue between poetry and tensorflow metadata since >=2.11
# This is a temporary workaround
# related to https://github.com/python-poetry/poetry/issues/8271
# Inspired from https://github.com/tensorflow/tensorflow/blob/adb39b04e9cb116df4659a7e2de9eea27e62f25c/tensorflow/tools/pip_package/setup.py#L148-L162
tensorflow = {version = "^2.13.0" }
tensorflow-macos = { version = "^2.13.0", platform = "darwin", markers = "platform_machine=='arm64'" }
tensorflow-intel = { version = "^2.13.0", platform = "win32" }
tensorflow-cpu = [
    { version = "^2.13.0", platform = "linux", markers = "platform_machine!='arm64' and platform_machine!='aarch64'" },
    { version = "^2.13.0", platform = "darwin", markers = "platform_machine!='arm64' and platform_machine!='aarch64'" },]
tensorflow-cpu-aws = { version = "^2.13.0", platform = "linux", markers = "platform_machine=='arm64' or platform_machine=='aarch64'" }
# https://github.com/tensorflow/tensorflow/blob/adb39b04e9cb116df4659a7e2de9eea27e62f25c/tensorflow/tools/pip_package/setup.py#L107-L108
# https://github.com/python-poetry/poetry/issues/8271#issuecomment-1697740447
tensorflow-io-gcs-filesystem = [
    { version = ">= 0.23.1", markers = "platform_machine!='arm64' or platform_system!='Darwin'" },
    { version = "< 0.32.0", markers = "platform_system == 'Windows'" }
]