Upgrade pandas to 1.4 to fix pandas binary incompatibility with numpy #295
Description
When I install tensorflow-federated I end up with incompatible builds of numpy and pandas. Here's the main error from the stacktrace (full stack trace below).
from pandas._libs.interval import Interval
File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
Based on this Stack Overflow post it looks like there is a problem in the way pandas is getting built and pandas is being built against a version of numpy which is different from the installed version of numpy.
I was able to resolve the issue just by installing a newer version of pandas.
pip install --upgrade pandas~=1.4.3
I think this may solve the problem because newer versions of pandas have wheel's available and don't require compiling from source.
Given pandas 1.1.4 is almost 2 years old. Would it be reasonable to upgrade the version of pandas in tensorflow/privacy?
It looks like pandas is only used in a couple places.
https://github.com/tensorflow/privacy/search?q=pandas
The usage of pandas seems pretty straightforward and seems like low risk to an upgrade.
Here's the full stack trace for import tensorflow_federated
>>> import tensorflow_federated as tff
2022-08-01 21:46:22.522372: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/lib:/usr/local/lib/x86_64-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:
2022-08-01 21:46:22.522419: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/__init__.py", line 20, in <module>
from tensorflow_federated.python import aggregators
File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/python/aggregators/__init__.py", line 24, in <module>
from tensorflow_federated.python.aggregators.concat import concat_factory
File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/python/aggregators/concat.py", line 24, in <module>
import tensorflow as tf
File "/opt/conda/lib/python3.10/site-packages/tensorflow/__init__.py", line 473, in <module>
keras._load()
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/util/lazy_loader.py", line 41, in _load
module = importlib.import_module(self.__name__)
File "/opt/conda/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/opt/conda/lib/python3.10/site-packages/keras/__init__.py", line 24, in <module>
from keras import models
File "/opt/conda/lib/python3.10/site-packages/keras/models/__init__.py", line 18, in <module>
from keras.engine.functional import Functional
File "/opt/conda/lib/python3.10/site-packages/keras/engine/functional.py", line 31, in <module>
from keras.engine import training as training_lib
File "/opt/conda/lib/python3.10/site-packages/keras/engine/training.py", line 30, in <module>
from keras.engine import compile_utils
File "/opt/conda/lib/python3.10/site-packages/keras/engine/compile_utils.py", line 20, in <module>
from keras import metrics as metrics_mod
File "/opt/conda/lib/python3.10/site-packages/keras/metrics/__init__.py", line 33, in <module>
from keras.metrics.metrics import MeanRelativeError
File "/opt/conda/lib/python3.10/site-packages/keras/metrics/metrics.py", line 22, in <module>
from keras import activations
File "/opt/conda/lib/python3.10/site-packages/keras/activations.py", line 20, in <module>
import keras.layers.activation as activation_layers
File "/opt/conda/lib/python3.10/site-packages/keras/layers/__init__.py", line 27, in <module>
from keras.engine.base_preprocessing_layer import PreprocessingLayer
File "/opt/conda/lib/python3.10/site-packages/keras/engine/base_preprocessing_layer.py", line 19, in <module>
from keras.engine import data_adapter
File "/opt/conda/lib/python3.10/site-packages/keras/engine/data_adapter.py", line 38, in <module>
import pandas as pd # pylint: disable=g-import-not-at-top
File "/opt/conda/lib/python3.10/site-packages/pandas/__init__.py", line 30, in <module>
from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
File "/opt/conda/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in <module>
from pandas._libs.interval import Interval
File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject