Package Addons on conda-forge

[seanpmorgan]

This will likely come after we finalize our build automation and toolchain, but wanted to start an issue for visibility. When the time comes we will enable anaconda installs using conda-forge.

cc @kyleabeauchamp

[kyleabeauchamp]

As a starting point, I made a WIP PR to get a conda package for TF2 nightly: conda-forge/tensorflow-feedstock#68

[Smokrow]

Can I help with anything?

[seanpmorgan]

Hi @jjhelmus -- we're starting the process of packaging tensorflow/addons in conda and wanted to ask a few questions to see if you could guide us.

1. Is conda-forge the best channel for us?
2. It looks like there is a conda-forge/tensorflow and anaconda/tensorflow. Which dependency should we build against? (I think the conda-forge docs say to avoid external channel dependencies?)
3. Do you have an idea what the lag time is for tensorflow packages to show up on conda? (i.e. how long after tf2 release will we be able to build our conda package which depends on tf2?) It looks like conda-forge/tensorflow is still 1.13

[kyleabeauchamp]

1. I'm 90% sure the answer is yes, but there will be issues with GPU support given the state of nvidia licensing (doesn't support OSS projects like conda-forge).
2. We're definitely in a stange place with wanting to support both CPU and GPU work. I think this is a very good question that we need to resolve.
3. The lag time for conda-forge is about an hour after the pull request is merged. However, the review cycle can take up to a week.

PS I still think it would be good to see if @jjhelmus agrees.

[xhochy]

As a conda-forge maintainer (not part of core though), I would answer:

1. Yes, it is. This where nearly all community-led conda package development happens except for specialized things like bioconda.
2. For CUDA in conda-forge and the legal problems see GPU Support? conda-forge/conda-forge.github.io#63 and How to specify CUDA version in a conda package? conda-forge/conda-forge.github.io#687
3. Review cycles can take a week on complicated PRs but normally once CI has passed it's more around 1-2h during european and US daytime.

[jjhelmus]

1. conda-forge is the largest community-led packaging community and would make an ideal channel to use and contribute too (note that I am on the conda-forge core team so I might be bias). Another option is to create your own channel and publish packages there.
   Some organizations like pytorch and rapids find this flexibility useful. Using your own channel will require you to setup the necessary infrastructure for producing packages and users may find it harder to find your packages than if they were on conda-forge.

2. This depends on how addons interacts with tensorflow. If the interaction is via Python only then either channel should be fine. If you make use of the C/C++ components then you will need to take care to match the ABI used by the channel.
   With the newer 1.13 builds conda-forge's linux-64 package is build from source using the conda compilers. The osx-64 and win-64 packages are repackaged wheels. The packages in defaults have been built from source using the conda provided compilers for a number of versions.
   Only defaults provides a GPU package at this time. The package name is the same, tensorflow, but the tensorflow-gpu metapackage can be used to select the GPU variant.
   Please avoid using the anaconda channel. It is a mixture of the two subchannels which make up defaults but without any subchannel priority which is needed for reliable solutions.

3. For conda-forge packages the time between a release and conda packages depends on entirely on volunteers.
   For defaults the time depends on where the package is prioritized by Anaconda, Inc. Typically packages are build within 2 weeks but this depends greatly on the difficulty of the build and other team priorities. Given the large number of changes in TF 2.0 I would not be surprised if the lag for that release is longer than normal.

[seanpmorgan]

Thanks for all the feedback everyone. So to answer one question: Addons does indeed utilize the pre-compiled libtensorflow_framework.so for our C++ op linking.

Just want verify my understanding from the above information is correct:

1. It seems like packaging in conda-forge will be ideal
2. We should link to defaults channel's tensorflow making sure we match the ABI that TF was compiled against
   • Should we also encourage users to install cudatoolkit if they want GPU support? Currently the plan is to link to tensorflow-gpu and use CPU fallback in case it is not available. In 2.1 these packages will be consolidated.
3. We are blocked on this issue until a TF2 full release is made and subsequent package is distributed on conda

[Smokrow]

Should we add like a "WiP" or "blocked" Label or something like that?

[seanpmorgan]

This is now unblocked as TF2 is on conda.

[jonas-eschle]

Hi all, what's the status of this?