Unable to install `plotly-resampler` on some linux distributions

[Alexander-Serov]

I am using plotly-resampler, which installs correctly on my local Windows machine and on another Linux machine I have access to (by just using pip install plotly-resampler). However, we also run Gitlab-CI tests in a controlled environment and installing with pip kept failing in that environment. The exact error was

Building wheel for lttbc (setup.py): started
  Building wheel for lttbc (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jgc_t9dg/lttbc_[494](https://gitlab.edf-sf.com/optim/statistical_analysis/-/jobs/131076#L494)9a59daf574371b0f97218e19bdac5/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jgc_t9dg/lttbc_4949a59daf574371b0f97218e19bdac5/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-6yato32h
       cwd: /tmp/pip-install-jgc_t9dg/lttbc_4949a59daf574371b0f97218e19bdac5/
  Complete output (16 lines):
  running bdist_wheel
  running build
  running build_ext
  building 'lttbc' extension
  creating build
  creating build/temp.linux-x86_64-3.9
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/lib/python3.9/site-packages/numpy/core/include -I/tmp/pip-install-jgc_t9dg/lttbc_4949a59daf574371b0f97218e19bdac5 -I/usr/local/lib/python3.9/site-packages/numpy/core/include -I/tmp/pip-install-jgc_t9dg/lttbc_4949a59daf574371b0f97218e19bdac5 -I/usr/local/include/python3.9 -c lttbc.c -o build/temp.linux-x86_64-3.9/lttbc.o
  In file included from /usr/lib/gcc/x86_64-linux-gnu/10/include/syslimits.h:7,
                   from /usr/lib/gcc/x86_64-linux-gnu/10/include/limits.h:34,
                   from /usr/local/include/python3.9/Python.h:11,
                   from lttbc.c:2:
  /usr/lib/gcc/x86_64-linux-gnu/10/include/limits.h:195:15: fatal error: limits.h: No such file or directory
    195 | #include_next <limits.h>  /* recurse down to the real one */
        |               ^~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for lttbc
  Running setup.py clean for lttbc
 Running setup.py install for lttbc: started
    Running setup.py install for lttbc: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-lgmdp697/lttbc_3817a2fad4f7468ea159ce68739ae851/setup.py'"'"'; __file__='"'"'/tmp/pip-install-lgmdp697/lttbc_3817a2fad4f7468ea159ce68739ae851/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-uwdc83fp/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.9/lttbc
         cwd: /tmp/pip-install-lgmdp697/lttbc_3817a2fad4f7468ea159ce68739ae851/
    Complete output (16 lines):
    running install
    running build
    running build_ext
    building 'lttbc' extension
    creating build
    creating build/temp.linux-x86_64-3.9
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/lib/python3.9/site-packages/numpy/core/include -I/tmp/pip-install-lgmdp697/lttbc_3817a2fad4f7468ea159ce68739ae851 -I/usr/local/lib/python3.9/site-packages/numpy/core/include -I/tmp/pip-install-lgmdp697/lttbc_3817a2fad4f7468ea159ce68739ae851 -I/usr/local/include/python3.9 -c lttbc.c -o build/temp.linux-x86_64-3.9/lttbc.o
    In file included from /usr/lib/gcc/x86_64-linux-gnu/10/include/syslimits.h:7,
                     from /usr/lib/gcc/x86_64-linux-gnu/10/include/limits.h:34,
                     from /usr/local/include/python3.9/Python.h:11,
                     from lttbc.c:2:
    /usr/lib/gcc/x86_64-linux-gnu/10/include/limits.h:195:15: fatal error: limits.h: No such file or directory
      195 | #include_next <limits.h>  /* recurse down to the real one */
          |               ^~~~~~~~~~
    compilation terminated.
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-lgmdp697/lttbc_3817a2fad4f7468ea159ce68739ae851/setup.py'"'"'; __file__='"'"'/tmp/pip-install-lgmdp697/lttbc_3817a2fad4f7468ea159ce68739ae851/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-uwdc83fp/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.9/lttbc Check the logs for full command output.

I am posting it here in case it helps other folks who might encounter the same problem.

I have played around with the test environment and was able to install all packages by executing

apt update && apt install -yqq --no-install-recommends gcc musl-dev linux-headers-amd64 libc-dev

before the pip command. This allowed me to install the apparently missing linux header, lttbc and ploty-resampler. However, for some reason resulted in an incompatibility with numpy:

------------------------------- Captured stderr --------------------------------
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
___________________ ERROR collecting pv/test_pv_generator.py ___________________
ImportError while importing test module '/builds/--YGsyLe/2/optim/statistical_analysis/tests/pv/test_pv_generator.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test.py:12: in <module>
    from plotly_resampler import FigureResampler
/usr/local/lib/python3.9/site-packages/plotly_resampler/__init__.py:8: in <module>
    from .figure_resampler import FigureResampler
/usr/local/lib/python3.9/site-packages/plotly_resampler/figure_resampler.py:28: in <module>
    from .downsamplers import AbstractSeriesDownsampler, LTTB
/usr/local/lib/python3.9/site-packages/plotly_resampler/downsamplers/__init__.py:5: in <module>
    from .downsamplers import LTTB, EveryNthPoint, AggregationDownsampler
/usr/local/lib/python3.9/site-packages/plotly_resampler/downsamplers/downsamplers.py:8: in <module>
    import lttbc
E   ImportError: numpy.core.multiarray failed to import

So I abandoned.

As I said, I am publishing this info here in case someone stumbles on a similar issue, so feel free to close. However, I saw that lttbc is a top-level dependency of plotly-resampler and is still in early stages (version <1) and has not been updated since 2020. So there is little chance its python wheels will be changed anytime soon. So I wonder, whether on the plotly-resampler side we could add a try-except for lttbc import and fall back onto another resampler if lttbc is unavailable for import? Or, perhaps, if you have any idea of how to install the lttbc dependency without gcc compiling, it would be much appreciated!

I understand this is not directly related to ploty-resampler. I have thought about posting in lttbc instead, but the repo does not seem to be actively maintained. Thanks again for the resampler. Great idea!

[jonasvdd]

Hi @Alexander-Serov,

I encountered the same error issue on my Linux-distro once I tried to update Plotly-Resampler's Python dependencies to lttbc=0.2.1, see #42.

Can you state which lttbc & numpy version you are using? (I think the issue lies in that direction).
Then I can further research this and adapt the pyproject accordingly.

Best regards,
Jonas

[Alexander-Serov]

Hey @jonasvdd,

Thanks for looking into it. Good idea, it may indeed be related to the versions of other packages in the repo. I already had a numpy-related problem on a different package (though I don't insist there is a direct connection). But just in case: louisabraham/pydivsufsort#30 (comment)

To answer your specific question, I see the pipelines installing the following versions:

• numpy ==1.21.5
• lttbc==0.2.0
  I have played around with putting specific versions of lttbc and linux-headers, but wasn't really successful getting another error you see in the description above. So for the moment, I don't install plotly-resampler for the tests. But it's kind of a dirty fix.

Let me know if you have further questions!

[jonasvdd]

Hi @Alexander-Serov!

In the coming months, we will look into using a new version of the LTTB algorithm!
keep you posted!

[jvdd]

Hi @Alexander-Serov, any updates on this?

PS: we added an adapted C implementation of LTTB in this repo #103 (hence the lttbc dependency is dropped now) - we did not create a new release yet (but you can try out this prerelease)

[Alexander-Serov]

Hi @Alexander-Serov, any updates on this?

PS: we added an adapted C implementation of LTTB in this repo #103 (hence the lttbc dependency is dropped now) - we did not create a new release yet (but you can try out this prerelease)

Hey @jvdd ,

thanks for the update! I will try to find a moment to give it a try in the coming days!

[jonasvdd]

Hi @Alexander-Serov,

Have you already found a moment to try it?
Fyi: We just released plotly-resampler==0.8.0, which contains an improved LTTB implementation.

[Alexander-Serov]

Hey @jonasvdd,

I am really sorry for the delay: I have been busy in July and the holiday season intervened in the testing.

In any case, I have finally managed to test the new version and it seems to work great, both locally and on the pipelines! Thanks for the rapid solution to this not-so-simple issue! Amazing. 😀 I can finally remove my try and catch clauses around the plotly-resampler imports.

Collecting plotly-resampler~=0.8.0
  Downloading plotly-resampler-0.8.0.tar.gz (43 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'

Successfully installed ... plotly-resampler-0.8.0 ...

There is one minor problem though. In one of my test plot functions, FigureResampler is now refusing to initialize from a go.Figure with the following message:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
in plot_average_year:
    fig = FigureResampler(go.Figure(), default_n_shown_samples=max_points)
/usr/local/lib/python3.9/site-packages/plotly_resampler/figure_resampler/figure_resampler.py:248: in __init__
    if is_figure(figure) and not is_fr(figure):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
figure = Figure({
    'data': [], 'layout': {'template': '...'}
})
    def is_figure(figure: Any) -> bool:
        """Check if the figure is a plotly go.Figure or a FigureResampler.
    
        .. Note::
            This method does not use isinstance(figure, go.Figure) as this will not work
            when go.Figure is decorated (after executing the
            ``register_plotly_resampler`` function).
    
        Parameters
        ----------
        figure : Any
            The figure to check.
    
        Returns
        -------
        bool
            True if the figure is a plotly go.Figure or a FigureResampler.
        """
    
>       return isinstance(figure, BaseFigure) and (not isinstance(figure, BaseFigureWidget))
E       TypeError: isinstance() arg 2 must be a type or tuple of types

It is as if I was not giving it a go.Figure as input. Weirdly, I only experience it on my Gitlab-CI setup. The same test passes correctly on my Windows machine. 🤔 I am investigating where this comes from. This does not seem to be related to this issue, but to some other changes in 0.8.0. I will let you know if I find anything.

P.S. And by the way, it's awesome to have static images in the saved notebooks now! Since I discovered the resampler, it has become my everyday tool. Keep up the good work!

[Alexander-Serov]

I seem to be launching the FigureResampler on a figure before it contains any data in it. And this seems to fail the resampler isinstance test. I will try to move resampling to the moment once everything has been drawn. I am sure this worked before (and seems to work on Windows), but maybe something has changed since v0.2.* :)

[jonasvdd]

Hi @Alexander-Serov,

Firstly, thank you for your kind words @jvdd and I really appreciate this! 🤗

I think I found your experienced issue on Windows; you do not seem to have IPywidgets installed; which is totally okay!
However, our is_figure check ⬇️ method, did not deal correctly with this

"""Utility functions for the figure_resampler submodule."""

import math
import pandas as pd

from plotly.basedatatypes import BaseFigure
try: # Fails when IPywidgets is not installed
    from plotly.basewidget import BaseFigureWidget
except (ImportError, ModuleNotFoundError):
    BaseFigureWidget = None  # <--- this is the bug: we compare with None; must be type(None)

...

def is_figure(figure: Any) -> bool:
    return isinstance(figure, BaseFigure) and (not isinstance(figure, BaseFigureWidget))

Which gives the following error when comparing with None:

I will fix this and release a new version later this day. Keep you posted!

Thank you so much for pointing out these bugs!
Together we improve the usability of packages like this one! 🤝