Issues with prefix/destdir and "Broken python installation detected" warnings #82
Comments
You need Meson master (or https://github.com/rgommers/meson/tree/scipy) for the build to succeed. Could you try that and see if it helps? I'm not sure it will, because I haven't tried virtualenv yet, but 0.59.2 is certainly too old. |
|
Yup, that was it! With the build succeeds. Running the tests now. The remaining annoyance is during the setup stage, and finally install is into |
|
One more issue, with stock BLAS/ATLAS. Many copies of But that goes under a different rubric, as in "improve blas/lapack detection" for openblas alternatives. |
|
Thanks! I haven't seen |
|
mesonbuild/meson#9388 (comment) indicates that the warning is probably going to stay as it's related to a heuristics meson applies to work around Debian/Ubuntu interaction with virtualenvs (and likely other virtual environments on debian derivatives). |
|
Ok, so the original problem was a user fluke, which is now fixed by the meson master, so this one can be closed. Or do you want me to keep this issue open for the openblas alternatives? |
|
Not for OpenBLAS, but let's leave it open for the "broken Python" warning - I'm seeing it on Arch Linux as well, so it's almost certainly incorrectly triggered. |
|
In fact the warning is only printed when meson evaluates the build setup and concludes that it believes |
|
That warning will be unexpected then. I expected development builds to do this standard, and also wheel builds - install to a custom prefix, then use Note that |
|
As for the install step, setup.py is blessed to run in a python only world, does not compute the standard GNU install directories, and cannot handle mixed-language code (it unofficially supports "data files" as long as they are installed to sys.prefix but the authors consider this deprecated, and cannot install outside that at all). In return, it's very good at installing to the correct location always. I suppose it wasn't considered important to implement a message during Pip on the other hand, will in fact warn you if it installs an entry point to a location that isn't on PATH. This seems reasonable. ... In-place builds aren't supported at all, so that's a bad comparison to make. The structured sources proposal in meson might solve this and make None of this makes the idea of ensuring that (I am pretty sure you don't actually want to do it that way, and you're just working around the fact that editable installs aren't implemented.) |
It may be, but
No, I disagree. To stay with Pip: pip will invoke the actual build system (Meson), and may ask it to do all its work in
Actually, we're mostly avoiding editable installs, because it makes the dev use an extra layer (like pip) which does unwanted things like hide the build log. So yeah, I'd rather not use PYTHONPATH but something like Possible solution If you want to keep it enabled by default, can we have a simple way to turn it off? I don't have a project where I'm producing both Python packages and non-Python package, nor do I need to care about GNU install directories. So I'd just like something like The |
|
@FFY00 pointed out that I likely should be using # On Arch Linux, using a Python from a conda env:
$ meson setup build
$ ninja -C build # warns about platlibdir etc.
# To test (this is wrong):
$ meson install -C build
<asks for permissions to run as super user because the default location is /usr - deny those>
# Now try --destdir:
$ meson install -C build --destdir installdir
... <many lines of output, last one is:>
Installing /home/rgommers/code/bldscipy/scipy/io/harwell_boeing/tests/test_hb.py to /home/rgommers/code/bldscipy/build/installdir/usr/local/lib/python3.9/site-packages/scipy/io/harwell_boeing/testsConclusion of the above: yes I should be using So it seems like I should do both here:
Does this work? It seems not: $ rm -rf build
$ which python
/home/rgommers/anaconda3/envs/scipy-meson/bin/python
$ meson setup build --prefix /home/rgommers/anaconda3/envs/scipy-meson
...
Program python3 found: YES (/home/rgommers/anaconda3/envs/scipy-meson/bin/python3.9)
WARNING: Broken python installation detected. Python files installed by Meson might not be found by python interpreter.
This warning can be avoided by setting "python.platlibdir" option.
WARNING: Broken python installation detected. Python files installed by Meson might not be found by python interpreter.
This warning can be avoided by setting "python.purelibdir" option.
...
User defined options
prefix: /home/rgommers/anaconda3/envs/scipy-meson
...Why is this still showing the warnings? If there is one targets out of the many I have that points to a path outside of the prefix, how do I find it? |
rgommers
changed the title
|
When I try to build in a virtual environment created using (scipy-meson) tirthasheshpatel@tirthap:~/Desktop/scipy_meson$ meson setup build --prefix=$PWD/install
...
scipy/meson.build:34:0: ERROR: Tried to form an absolute path to a source dir.
You should not do that but use relative paths instead.
To get include path to any directory relative to the current dir do
incdir = include_directories(dirname)
After this incdir will contain both the current source dir as well as the
corresponding build dir. It can then be used in any subdirectory and
Meson will take care of all the busywork to make paths work.
Dirname can even be '.' to mark the current directory. Though you should
remember that the current source and build directories are always
put in the include directories by default so you only need to do
include_directories('.') if you intend to use the result in a
different subdirectory.From To verify that this is the case, I replaced the diff --git a/scipy/meson.build b/scipy/meson.build
index 3f6698a73..cdd394a4a 100644
--- a/scipy/meson.build
+++ b/scipy/meson.build
@@ -31,7 +31,7 @@ incdir_numpy = run_command(py3,
check: true
).stdout().strip()
-inc_np = include_directories(incdir_numpy)
+inc_np = include_directories('/home/tirthasheshpatel/Desktop/scipy_meson/scipy-meson/lib/python3.9/site-packages/numpy/core/include')
incdir_f2py = incdir_numpy / '..' / '..' / 'f2py' / 'src'
inc_f2py = include_directories(incdir_f2py)and built using a local meson installation outside of the virtual env in my system and it fails with the same error. Is there any way I could use Steps to reproduce: # I use Python3.9
python3.9 -m virtualenv scipy-meson
source scipy-meson/bin/activate
python -m pip install -U pip
pip install numpy cython pythran pybind11 meson==0.60.1 ninja
meson setup build --prefix=$PWD/install |
|
@tirthasheshpatel is this happening with both meson |
Yes (tried with 0.60.0, 0.60.1, master) |
|
Also, what distro? |
|
I haven't been able to reproduce this yet, but it's the second time I've seen it. Maybe be related to scipy#14890, IIRC that's why we needed an absolute path - file writing seemed to need it, can't remember the details but likely because had to do with it having to do the right thing for both distutils and meson based builds, and those are writing files to different directories (in-place, or to the build dir). |
|
Hmm, not sure if it is related to Cython code generation. If I am not wrong, meson fails during the setup stage before generating any files... We call |
|
Ah I remember you mentioned this a week ago, and it's only broken in your Ubuntu virtualenv, outside of virtualenv it's fine: scipy#14847 (comment). I added this to the tracking issue. Debian/Ubuntu is known to make a giant mess out of virtualenvs (and Python sysconfig et al.), so I think I'd like to put this lower on the TODO list. We should add a CI job for this situation. It's also not the same issue as the For now I'd say, if you're on Ubuntu, avoid a virtualenv, or use conda. It may be a "bug" in Meson, but only because Debian is completely messed up - see mesonbuild/meson#8739. |
|
Thanks for your help! I will use conda in the meantime. |
Ugh, that's a bit of an advice :-). On my previous attempts, it did work despite an unhelpful warning . That was on meson master, pre-0.60. Virtualenv on ubuntu is a perfectly valid and mainstream use case. |
We'll fix it well before switching over to Meson as the default, no worries. I just want to finally merge support into SciPy master first, and ensure wheel build and Windows work - that's more important than this virtualenv-on-debian specific issue. If it was easy we'd just do it now, but it is not. |
@ev-br Did you install meson in your virtual env? I think meson could pick up on numpy C sources outside virtual envs if a local installation is used. Can you share your meson logs ( |
|
@tirthasheshpatel I just realized why I'm notified in this issue :-). My attempts are reported above, and I don't think I've kept the logs beyond what's up there. And I definitely installed everything into the virtualenv --- I don't think I even have numpy installed outside of a virtualenv. |
|
It seems that your virtualenv is a subdirectory of the scipy repository? |
No, that's the same "issue" as we've been discussion higher up (see #82 (comment)). It should be irrelevant here. @tirthasheshpatel can confirm that just a plain |
|
I do hope your work-in-progress |
|
My point is that this seems to be entirely unrelated to prefix. Prefix and how to handle it, has an effect on whether meson warns you that files will be installed outside of the PYTHONPATH. It's not actually a meson error, it's just bad defaults for installing. Virtualenvs being inside the source tree have an effect on whether string literals passed from subprocesses (that run numpy code and print header paths) to meson's include_directories, are absolute paths which meson objects for purposes of codegen, and actually raises a configure-time error. Imagine that numpy isn't involved, and some developer somewhere makes a project in Meson lints this sort of thing, and it doesn't have any way of knowing that One way to let meson know that this is a dependency is to go through the subproject system, build numpy with meson, and expose a meson.override_dependency() which builds a relative include path and exports it for use by the superproject. The other solution is just "don't do that", build your virtualenv one directory upward so it isn't a child node of the scipy source tree. If you have any ideas for a potential third solution I'm all ears. :) But I'm not sure how to allow it for numpy without allowing it for |
It's not in the source tree, the virtualenv is elsewhere. It's just that the in-tree install location is what is recommended by my install script. So you can do development work, and
Yes I agree. I just don't understand why it breaks in this Ubuntu virtualenv, and it works without virtualenvs.
We definitely cannot require building # NumPy include directory - needed in all submodules
incdir_numpy = run_command(py3,
[
'-c',
'import os; os.chdir(".."); import numpy; print(numpy.get_include())'
],
check: true
).stdout().strip()
inc_np = include_directories(incdir_numpy)Then we just need to build NumPy support into Meson, so |
|
(and no, |
Oh wait, it was?? Then I agree with "don't do that". |
Yes, it was. But I created a fresh venv in a different directory but still got the same error. Meson Log |
|
That actually appears to be a legit bug. Given @rgommers I'd be happy to mentor and/or review+merge a numpy dependency for https://mesonbuild.com/Dependencies.html#dependencies-with-custom-lookup-functionality (I may or may not spontaneously find time to implement that myself.) |
Thanks! Let me open a Meson issue so we keep track of it and do it all at once (e.g. we need |
|
I've seen exactly the same before and was able to fix it by uninstalling meson and reinstalling via pip. e.g. "python3 -m pip install meson" and then set the PATH to include "~/.local/bin". Now this pip installed "meson" understands python locations, and the warning is gone. Hopefully that applies to your case as well. |
|
To record the current state:
|
|
Yup, can confirm that the virtualenv on ubuntu focal (the OP issue) works fine with meson 0.61.1. I only tested 0.61.1 and only with virtualenv (no stdlib venv, no user installs outside of a virtualenv). Which is superb, and goes 99% of the way, I'd say. The warning is mostly harmless IMO (will need to be documented to be ignored, and duly ignored, I'd think). |
|
Thanks, let me close this then. |
|
Also tested on my ubuntu focal with meson 0.61.1 and virtualenv and everything works seamlessly. I love the speed: from 30-40 minutes of build time using distutils to 2-3 minutes (on 8 cores) using meson & ninja!! It would be a cheery on the cake to get mesonbuild/meson#9788 in 0.62.0! Thanks @rgommers and Meson team for working on this! |
|
The |
|
Awesome! Thanks again @eli-schwartz. When 0.62.0 is out I will make it the minimum required version and add |
Describe your issue.
Am trying to build the tip of the master branch with meson in a virtualenv on ubuntu focal. The setup works fine for the scipy master with setup.py / runtests.py / pip, but with meson fails with a nonsensical cannot find
<complex> hearderin scipy.special.Here's a full set of steps:
Reproducing Code Example
Error message
.SciPy/NumPy/Python version information
.
The text was updated successfully, but these errors were encountered: