TRACKER: Getting Started with Meson/Bug Reports #49683
Comments
lithomas1
added
Build
|
I propose a more gradual approach to meson (merge #49115).
At this point, almost everything in #49115 would be included. Then we could wait for the editable wheel and all the other necessary features/fixes on meson. When everyone is satisfied, we then modify the backend setting in pyproject.toml to make the final switch. |
|
I would not go forward with setuptools PEP 517, since it's just going to get replaced by meson, and you can't have two build systems in pyproject.toml. The reason it is safe to modify the backend setting in pyproject.toml now, is because that it doesn't break people with editable installs(pip is smart enough to fall back to setup.py develop).
The problem is that meson builds won't work until the build backend is set to pyproject.toml. I would also really not prefer waiting. It is really hard to keep the meson PR open and up to date, and even if you can't really use meson locally without the editable installs, I think the reduction in CI time is well worth it. |
|
Am trying to build pandas locally with meson. I'm on commit c96dbb7 here's what I've done: and here's what I get: has anyone come across this? |
|
Hm. I think the issue might be that the venv is stored in a folder within the pandas repo? In general, I don't think meson likes absolute paths if they are within the repo you're trying to build. We might need to steal scipy's code for this. |
This snippet of the error message was added by @rgommers to provide a hint about the problem. The relpath trick seems apropos here, indeed. (More generally, the long-term idea is to create some dependency interface for numpy, so people can just use that directly and not need to worry about running |
Indeed, that should fix it - you need a few more lines, down to: np_dep = declare_dependency(include_directories: inc_np) |
That's right - I've now created a virtualenv in a different folder ( Admittedly the pandas docs do say to do that, but I'm so used to |
|
I ran into this today. (pandas-dev) fangchenli@Fangchens-MacBook-Pro-2 pandas-fangchenli % pip install -e . --no-build-isolation -v
Using pip 23.1.2 from /Users/fangchenli/mambaforge/envs/pandas-dev/lib/python3.10/site-packages/pip (python 3.10)
Obtaining file:///Users/fangchenli/Workspace/pandas-fangchenli
Running command Checking if build backend supports build_editable
Checking if build backend supports build_editable ... done
Running command Preparing editable metadata (pyproject.toml)
+ meson setup --reconfigure /Users/fangchenli/Workspace/pandas-fangchenli /Users/fangchenli/Workspace/pandas-fangchenli/build/cp310 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --vsenv --native-file=/Users/fangchenli/Workspace/pandas-fangchenli/build/cp310/meson-python-native-file.ini
Cleaning... 0 files.
The Meson build system
Version: 1.0.1
Source dir: /Users/fangchenli/Workspace/pandas-fangchenli
Build dir: /Users/fangchenli/Workspace/pandas-fangchenli/build/cp310
Build type: native build
Project name: pandas
Project version: 2.1.0.dev0+901.g4d217a42f0
C compiler for the host machine: arm64-apple-darwin20.0.0-clang (clang 14.0.6 "clang version 14.0.6")
C linker for the host machine: arm64-apple-darwin20.0.0-clang ld64 609
C++ compiler for the host machine: arm64-apple-darwin20.0.0-clang++ (clang 14.0.6 "clang version 14.0.6")
C++ linker for the host machine: arm64-apple-darwin20.0.0-clang++ ld64 609
Cython compiler for the host machine: cython (cython 0.29.34)
Host machine cpu family: aarch64
Host machine cpu: aarch64
Program python found: YES (/Users/fangchenli/mambaforge/envs/pandas-dev/bin/python3.10)
Build targets in project: 54
pandas 2.1.0.dev0+901.g4d217a42f0
User defined options
Native files: /Users/fangchenli/Workspace/pandas-fangchenli/build/cp310/meson-python-native-file.ini
buildtype : release
b_ndebug : if-release
b_vscrt : md
Found ninja-1.11.1 at /Users/fangchenli/mambaforge/envs/pandas-dev/bin/ninja
+ /Users/fangchenli/mambaforge/envs/pandas-dev/bin/ninja
[1/79] Generating pandas/_libs/algos_common_helper_pxi with a custom command
[2/79] Generating pandas/_libs/hashtable_class_helper_pxi with a custom command
[3/79] Generating pandas/_libs/algos_take_helper_pxi with a custom command
[4/79] Generating pandas/_libs/khash_primitive_helper_pxi with a custom command
[5/79] Generating pandas/_libs/index_class_helper_pxi with a custom command
[6/79] Generating pandas/_libs/intervaltree_helper_pxi with a custom command
[7/79] Generating pandas/_libs/sparse_op_helper_pxi with a custom command
[8/79] Generating pandas/_libs/hashtable_func_helper_pxi with a custom command
[9/79] Linking target pandas/_libs/arrays.cpython-310-darwin.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[10/79] Linking target pandas/_libs/hashing.cpython-310-darwin.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[11/79] Linking target pandas/_libs/indexing.cpython-310-darwin.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[12/79] Linking target pandas/_libs/internals.cpython-310-darwin.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[13/79] Generating write_version_file with a custom command
[14/79] Compiling C object pandas/_libs/tslibs/timedeltas.cpython-310-darwin.so.p/meson-generated_pandas__libs_tslibs_timedeltas.pyx.c.o
FAILED: pandas/_libs/tslibs/timedeltas.cpython-310-darwin.so.p/meson-generated_pandas__libs_tslibs_timedeltas.pyx.c.o
arm64-apple-darwin20.0.0-clang -Ipandas/_libs/tslibs/timedeltas.cpython-310-darwin.so.p -Ipandas/_libs/tslibs -I../../pandas/_libs/tslibs -I../../../../mambaforge/envs/pandas-dev/lib/python3.10/site-packages/numpy/core/include -I../../pandas/_libs/include -I/Users/fangchenli/mambaforge/envs/pandas-dev/include/python3.10 -fvisibility=hidden -fcolor-diagnostics -DNDEBUG -w -std=c99 -O3 -DNPY_NO_DEPRECATED_API=0 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/fangchenli/mambaforge/envs/pandas-dev/include -D_FORTIFY_SOURCE=2 -isystem /Users/fangchenli/mambaforge/envs/pandas-dev/include -MD -MQ pandas/_libs/tslibs/timedeltas.cpython-310-darwin.so.p/meson-generated_pandas__libs_tslibs_timedeltas.pyx.c.o -MF pandas/_libs/tslibs/timedeltas.cpython-310-darwin.so.p/meson-generated_pandas__libs_tslibs_timedeltas.pyx.c.o.d -o pandas/_libs/tslibs/timedeltas.cpython-310-darwin.so.p/meson-generated_pandas__libs_tslibs_timedeltas.pyx.c.o -c pandas/_libs/tslibs/timedeltas.cpython-310-darwin.so.p/pandas/_libs/tslibs/timedeltas.pyx.c
pandas/_libs/tslibs/timedeltas.cpython-310-darwin.so.p/pandas/_libs/tslibs/timedeltas.pyx.c:755:10: fatal error: 'src/datetime/pd_datetime.h' file not found
#include "src/datetime/pd_datetime.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
[15/79] Compiling C object pandas/_libs/lib.cpython-310-darwin.so.p/src_parser_tokenizer.c.o
[16/79] Compiling C object pandas/_libs/index.cpython-310-darwin.so.p/meson-generated_pandas__libs_index.pyx.c.o
FAILED: pandas/_libs/index.cpython-310-darwin.so.p/meson-generated_pandas__libs_index.pyx.c.o
arm64-apple-darwin20.0.0-clang -Ipandas/_libs/index.cpython-310-darwin.so.p -Ipandas/_libs -I../../pandas/_libs -I../../../../mambaforge/envs/pandas-dev/lib/python3.10/site-packages/numpy/core/include -I../../pandas/_libs/include -I/Users/fangchenli/mambaforge/envs/pandas-dev/include/python3.10 -fvisibility=hidden -fcolor-diagnostics -DNDEBUG -w -std=c99 -O3 -DNPY_NO_DEPRECATED_API=0 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/fangchenli/mambaforge/envs/pandas-dev/include -D_FORTIFY_SOURCE=2 -isystem /Users/fangchenli/mambaforge/envs/pandas-dev/include -MD -MQ pandas/_libs/index.cpython-310-darwin.so.p/meson-generated_pandas__libs_index.pyx.c.o -MF pandas/_libs/index.cpython-310-darwin.so.p/meson-generated_pandas__libs_index.pyx.c.o.d -o pandas/_libs/index.cpython-310-darwin.so.p/meson-generated_pandas__libs_index.pyx.c.o -c pandas/_libs/index.cpython-310-darwin.so.p/pandas/_libs/index.pyx.c
pandas/_libs/index.cpython-310-darwin.so.p/pandas/_libs/index.pyx.c:756:10: fatal error: 'khash_python.h' file not found
#include "khash_python.h"
^~~~~~~~~~~~~~~~
1 error generated. |
|
You'll need to remove the I don't think meson tracked when the include directory changed, so didn't mark the affected files as needing a rebuild. |
|
When running in a docker container where there is no command
I get an error |
|
There are also some challenges getting the Cython debugger to work with meson. In setuptools you could hack this by adding |
Line 5 in 0c80f4e Meson itself doesn't run python here -- if this were
What happens if you add |
|
Ah cool thanks for the insight. So we should just make that python3 huh? For Cython, I got it to work doing similar to what you suggest: cython_args = ['--include-dir', meson.current_build_dir()]
if get_option('buildtype') == 'debug'
cython_args += ['--gdb']
endifThen passing cython args to the extension module The nuance from there is that you need to run |
|
After reading the docs I am unsure how I am supposed to rebuild the cython extensions once I modified a cython file. In times of setuptools I did something like |
|
It should rebuild automatically on import if you did an editable install. For reference, the new install command (it should only be necessary to run it once) is |
|
I tried to re-build pandas with meson yesterday. I had built it before via When trying to build pandas with meson I got the following error: I'm on an Apple M1 Pro computer and I'm on commit |
|
After starting again from scratch (removing my local pandas repo and cloning it again as well as creating a new dev environment) I was able to successfully build pandas with meson. The good old "Have you tried turning it off and on again?" did the trick. 😄 |
|
Building pandas through a virtual environment works well for me, but today I tried doing it using a Docker container but it failed, to setup (Windows), I ran: Then: Output: And in the log file it said something about Is there a better way to fix this issue? |
I don't know what you mean by this.
Why was it ever in Windows-style to begin with? This sounds like git was very badly misconfigured to cause |
Just wanted to mention that I have successfully built pandas using Meson build system before.
No clue, I don't ever remember playing around with my git installation |
|
When I say I don't know what you mean, I don't know what it means to build with Linux paths in docker, but with the word "Windows" in parentheses. |
|
I just wanted to highlight that I am on a Windows machine |
OS: Linux Debian I am not sure if this is related or not. But I just installed Pandas from source and faced these issues in order: Issue1Fixed: Issue2Fixed: Issue3Fixed: Issue4Fixed: Issue5missing Numpy fixed: After this it worked and I could import pandas and check the version. |
|
https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-no-build-isolation
So you have to manually install those build dependencies, which is rather awkward in the extreme. Back in the day, it just installed whichever ones you didn't yet have installed...
This is arguably a bug in the editable installs standard itself. |
|
My Linux Meson build fails successfully. I tried to run in it a fresh environment, no result so far. python -m pip install -ve . --no-build-isolation --config-settings editable-verbose=true
Using pip 24.0 from /home/USER/micromamba/envs/pandas-dev/lib/python3.10/site-packages/pip (python 3.10)
Obtaining file:///tmp/pandas
Running command Checking if build backend supports build_editable
Checking if build backend supports build_editable ... done
Running command Preparing editable metadata (pyproject.toml)
+ meson setup /tmp/pandas /tmp/pandas/build/cp310 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --vsenv --native-file=/tmp/pandas/build/cp310/meson-python-native-file.ini
The Meson build system
Version: 1.2.1
Source dir: /tmp/pandas
Build dir: /tmp/pandas/build/cp310
Build type: native build
Project name: pandas
Project version: 3.0.0.dev0+1079.g9e7abc84a1
C compiler for the host machine: /home/USER/micromamba/envs/pandas-dev/bin/x86_64-conda-linux-gnu-cc (gcc 12.3.0 "x86_64-conda-linux-gnu-cc (conda-forge gcc 12.3.0-7) 12.3.0")
C linker for the host machine: /home/USER/micromamba/envs/pandas-dev/bin/x86_64-conda-linux-gnu-cc ld.bfd 2.40
C++ compiler for the host machine: /home/USER/micromamba/envs/pandas-dev/bin/x86_64-conda-linux-gnu-c++ (gcc 12.3.0 "x86_64-conda-linux-gnu-c++ (conda-forge gcc 12.3.0-7) 12.3.0")
C++ linker for the host machine: /home/USER/micromamba/envs/pandas-dev/bin/x86_64-conda-linux-gnu-c++ ld.bfd 2.40
Cython compiler for the host machine: cython (cython 3.0.10)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python found: YES (/home/USER/micromamba/envs/pandas-dev/bin/python)
../../pandas/meson.build:1:15: ERROR: Command `/home/USER/micromamba/envs/pandas-dev/bin/python -c '
import os
import numpy as np
try:
# Check if include directory is inside the pandas dir
# e.g. a venv created inside the pandas dir
# If so, convert it to a relative path
incdir = os.path.relpath(np.get_include())
except Exception:
incdir = np.get_include()
print(incdir)
'` failed with status 1.
A full log can be found at /tmp/pandas/build/cp310/meson-logs/meson-log.txt
error: subprocess-exited-with-error
× Preparing editable metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/USER/micromamba/envs/pandas-dev/bin/python /home/USER/micromamba/envs/pandas-dev/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_editable /tmp/tmp9ki4obxi
cwd: /tmp/pandas
Preparing editable metadata (pyproject.toml) ... error
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details. |
|
Do you have a copy of numpy installed? |
|
@lithomas1 I do. Is this relevant somehow? What's the point of a virtual environment if it still relies on system packages? |
|
As @Aloqeely mentions, I reported the same issue in #59000. As I mentioned in that issue, I do not know how CRLF's got in the file. My suggestion is to add eol=lf to line 1 in .gitattributes, but I readily admit I am no expert in the proper way to configure line-endings in git. This, however, is the suggestion in the vscode dev container documentation for a different but related issue. |
|
I have the same error as @vkhodygo (#49683 (comment)) but on Windows. Was there ever a fix? To confirm, numpy is installed in the activated conda environment. This is the error on the meson log, not sure what to make of it: 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte
Unusable script 'G:\\Miniconda3\\envs\\pandas-dev\\python.exe'
Program python found: YES (G:\Miniconda3\envs\pandas-dev\python.exe)
Running command: G:\Miniconda3\envs\pandas-dev\python.exe -c "
import os
import numpy as np
try:
# Check if include directory is inside the pandas dir
# e.g. a venv created inside the pandas dir
# If so, convert it to a relative path
incdir = os.path.relpath(np.get_include())
except Exception:
incdir = np.get_include()
print(incdir)
"
--- stdout ---
--- stderr ---
Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initializedRunning that command at the command line works just fine, including with the provided -c script, making it all the more puzzling. |
Background
In #49115, we are adding support for a new build system meson. This build system addresses some of the flaws of the setuptools-based build system, such as race conditions on parallel builds. It is also much simpler and easier to understand, and we'll be able to clean up a bunch of build related code once it's merged.
How to build/test with meson
Timeline
- Mostly done. The debugging docs need to be updated
Ideally, meson build support and removal of setuptools should be fully complete by 2.1 (the next minor release).
Known Issues
N/A.
Purpose of this thread
I've opened this thread to get some feedback and to track any build issues users might be having with meson.
The text was updated successfully, but these errors were encountered: