stardist-napari requires tensorflow but pip doesn't reacognize tensorflow-macos

[psobolewskiPhD]

I was reminded about this when I went to update my stardist-napari.
The setup.py includes tensorflow but on macOS the package is tensorflow-macos.
pip doesn't recognize the later as the former and the install/update fails—even though it's imported as tensorflow.
Is there a way to flag that? by platform or something? Or maybe remove it?
I do notice that stardist doesn't list tensorflow as a requirement, but of course it's in the README. Here the situation is different, due to the Plugin menu install...

[maweigert]

Indeed, it seems that platform dependent requirements could be include like so:

    install_requires=[
        'tensorflow;  platform_system!="Darwin" or platform_machine!="arm64"',
        'tensorflow-macos;  platform_system=="Darwin" and platform_machine=="arm64"',
    ],

But then again, maybe its best to remove tf as explicit dependency altogether...

[uschmidt83]

Here the situation is different, due to the Plugin menu install...

That's exactly the reason why I included tensorflow in setup.py, because there's no README that people would look at before the installation. Of course, then the whole issue with TensorFlow on Apple Silicon happened and they decided to call the package tensorflow-macos.

it seems that platform dependent requirements could be include like so

Is it already known whether they'll keep the package name tensorflow-macos? Or will "regular" tensorflow just work on Apple Silicon in the future?

[psobolewskiPhD]

No idea, but they did update the package (and tensorflow-metal) on pypi, so they are developing/supporting it.
Hopefully macOS 12 brings some more clarity?
Also, hopefully all the people hyped on the M1Max will lead to more development in these ecosystems.

[uschmidt83]

I'm not sure what to do.

[psobolewskiPhD]

One option is to leave things as are, and rely on the README and napari-hub to provide info regarding installing tensorflow, etc. at first installation.
Then, perhaps the napari devs could provide an option to use --no-deps when updating a plugin, since presumably the user environment contains the deps?
This would address: #3 as well I think?

Now obviously a major update that requires an update to the deps would have to provide an error/warning...

[psobolewskiPhD]

BTW, why doesn't this issue happen with stardist?
Also, does regular tensorflow install on x86 macOS 11 or 12? Because the pypi for tf-macos has x86 as well: https://pypi.org/project/tensorflow-macos/#files
And so far, they have been updating it, it's got TF 2.7 now.

[maweigert]

BTW, why doesn't this issue happen with stardist?

Yes, for this reason regular stardist (and its neural network dependency csbdeep) doesn't depend on tensorflow and requires the user to install an appropriate version of it before. Not really ideal, I know! 😅

[psobolewskiPhD]

Indeed, it seems that platform dependent requirements could be include like so:

    install_requires=[
        'tensorflow;  platform_system!="Darwin" or platform_machine!="arm64"',
        'tensorflow-macos;  platform_system=="Darwin" and platform_machine=="arm64"',
    ],

But then again, maybe its best to remove tf as explicit dependency altogether...

So I tried this in a local clone (macOS 12, arm64) and it still doesn't install via pip install -e . in an env that has csbdeep tensorflow-macos and stardist already, everything functioning. (Nota bene: pip install stardist -U JustWorks)
pip finds everything and then decides it needs to downgrade numpy which fails.
☹️
CC=gcc-11 CXX=g++-11 python setup.py install does work tho, so maybe there is a clue there?
🤷‍♂️

[maweigert]

pip finds everything and then decides it needs to downgrade numpy which fails.
☹️

Yes, that is what I recurrenty observe too, most likely due to tensorflow again pinning some old numpy version. Hard to go around that :(

[psobolewskiPhD]

Looks like Apple will continue to maintain tensorflow-macos and it has something to do with the GPU aspects. see:
https://developer.apple.com/forums/thread/700083?answerId=704971022#704971022

At the same time, there is now a TF for arm64 on conda-forge (2.7.0 and 2.6.2, 2.6.0, etc) that reports as tensorflow with almost equivalent performance—but slightly slower—in my simple StarDist tests:
https://forum.image.sc/t/napari-tensorflow-aicsimageio-stardist-care-n2v-pyclesperanto-running-native-on-apple-silicon-m1/55051/34?u=psobolewskiphd

[uschmidt83]

Looks like Apple will continue to maintain tensorflow-macos and it has something to do with the GPU aspects. see: https://developer.apple.com/forums/thread/700083?answerId=704971022#704971022

I think we will go for tensorflow-macos if Apple is committed to maintain that. The necessary change to setup.py is already part of a pending PR.

[psobolewskiPhD]

I wonder if there is any logic that would permit tensorflow (e.g. the conda-forge one) on macOS arm64 while at the same time permitting tensorflow-macos.
Also, I have no idea how these names work, after all it's always import tensorflow ......

[maweigert]

Also, I have no idea how these names work, after all it's always import tensorflow ......

Yes, its complicated :))

Btw, thanks a lot for testing this all out - your image.sc post is my goto reference now :)

[psobolewskiPhD]

Btw, thanks a lot for testing this all out - your image.sc post is my goto reference now :)

Thanks 😊 I wish i could edit that OP to keep it current.
Maybe I should start a new thread with the latest info condensed together?

[psobolewskiPhD]

Looks like Apple will continue to maintain tensorflow-macos and it has something to do with the GPU aspects. see: https://developer.apple.com/forums/thread/700083?answerId=704971022#704971022

I think we will go for tensorflow-macos if Apple is committed to maintain that. The necessary change to setup.py is already part of a pending PR.

I posted this in the other issue (#3 (comment)) but Apple has updated the tensorflow-deps conda package, so now it has numpy 1.21. So they are clearing continuing support. Tensorflow-macos just got bumped to 2.8 and tensorflow-metal just got updated too.

[psobolewskiPhD]

Another update on the tensorflow/tensorflow-macos issue from an Apple engineer. Good news, they're aware of the issue.

So here I think the issue is that a lot of the packages that check for Tensorflow do it against the package name reserved for the baseline TensorFlow called tensorflow, whereas the TensorFlow package on MacOS is tensorflow-macos. We are working on a generic solution that will fix this issue but it will take until TF2.9 timeline for that to take effect.

See: https://developer.apple.com/forums/thread/701656?answerId=706752022#706752022

[psobolewskiPhD]

Update: tensorflow-macos 2.9 is live, but I don't see any difference in pip behavior.
Anyhow, this issue was resolved by:
e91f806

[psobolewskiPhD]

Update2: in a conda env (macOS, M1 arm64) with napari, but not having TF:
mamba create -y -n test-env python=3.9 napari
You can launch napari from the terminal and use the GUI to install stardist-napari and IT JUST WORKS!
🎉
Y'all are awesome 🤩
(no tensorflow-metal so it's slow, but it works. Maybe add that behind a gate like tensorflow-macos? Although tensorflow-metal also works on x86 macs with AMD GPU...)