Skip to content

Upgrade pandas to 1.4 to fix pandas binary incompatibility with numpy #295

Open
@jlewi

Description

When I install tensorflow-federated I end up with incompatible builds of numpy and pandas. Here's the main error from the stacktrace (full stack trace below).

    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Based on this Stack Overflow post it looks like there is a problem in the way pandas is getting built and pandas is being built against a version of numpy which is different from the installed version of numpy.

I was able to resolve the issue just by installing a newer version of pandas.

pip install --upgrade pandas~=1.4.3

I think this may solve the problem because newer versions of pandas have wheel's available and don't require compiling from source.

Given pandas 1.1.4 is almost 2 years old. Would it be reasonable to upgrade the version of pandas in tensorflow/privacy?

It looks like pandas is only used in a couple places.
https://github.com/tensorflow/privacy/search?q=pandas

The usage of pandas seems pretty straightforward and seems like low risk to an upgrade.

Here's the full stack trace for import tensorflow_federated

>>> import tensorflow_federated as tff
2022-08-01 21:46:22.522372: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/lib:/usr/local/lib/x86_64-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:
2022-08-01 21:46:22.522419: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/__init__.py", line 20, in <module>
    from tensorflow_federated.python import aggregators
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/python/aggregators/__init__.py", line 24, in <module>
    from tensorflow_federated.python.aggregators.concat import concat_factory
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/python/aggregators/concat.py", line 24, in <module>
    import tensorflow as tf
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/__init__.py", line 473, in <module>
    keras._load()
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/util/lazy_loader.py", line 41, in _load
    module = importlib.import_module(self.__name__)
  File "/opt/conda/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/opt/conda/lib/python3.10/site-packages/keras/__init__.py", line 24, in <module>
    from keras import models
  File "/opt/conda/lib/python3.10/site-packages/keras/models/__init__.py", line 18, in <module>
    from keras.engine.functional import Functional
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/functional.py", line 31, in <module>
    from keras.engine import training as training_lib
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/training.py", line 30, in <module>
    from keras.engine import compile_utils
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/compile_utils.py", line 20, in <module>
    from keras import metrics as metrics_mod
  File "/opt/conda/lib/python3.10/site-packages/keras/metrics/__init__.py", line 33, in <module>
    from keras.metrics.metrics import MeanRelativeError
  File "/opt/conda/lib/python3.10/site-packages/keras/metrics/metrics.py", line 22, in <module>
    from keras import activations
  File "/opt/conda/lib/python3.10/site-packages/keras/activations.py", line 20, in <module>
    import keras.layers.activation as activation_layers
  File "/opt/conda/lib/python3.10/site-packages/keras/layers/__init__.py", line 27, in <module>
    from keras.engine.base_preprocessing_layer import PreprocessingLayer
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/base_preprocessing_layer.py", line 19, in <module>
    from keras.engine import data_adapter
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/data_adapter.py", line 38, in <module>
    import pandas as pd  # pylint: disable=g-import-not-at-top
  File "/opt/conda/lib/python3.10/site-packages/pandas/__init__.py", line 30, in <module>
    from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
  File "/opt/conda/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions