cross build --target x86_64-pc-windows-gnu fails
#1445
Comments
oscar6echo
changed the title
cross build --target x86_64-pc-windows-gnu fails with " /usr/bin/x86_64-w64-mingw32-ld: cannot find -lpython310"cross build --target x86_64-pc-windows-gnu fails
oscar6echo
changed the title
cross build --target x86_64-pc-windows-gnu failscross build --target x86_64-pc-windows-gnu fails
|
There's a lot happening here. Is |
|
When you say 'in scope', you mean in the docker container meant to do the build or the host ? I would expect you mean in the container, right ? But then I don't really know the image. is mostly derived from a cross-rs image. The host is a Ubuntu without any knowledge about anything windows... |
|
I mean that the library needs to be accessible in I see that |
|
From inside the container.: root@45d1a52cf28c:/# ls -al /opt/python/cp310-cp310
lrwxrwxrwx 1 root root 30 Feb 8 18:54 /opt/python/cp310-cp310 -> /opt/_internal/cpython-3.10.13
root@45d1a52cf28c:/# ls -al /opt/_internal/cpython-3.10.13
total 24
drwxr-xr-x 6 root root 4096 Feb 8 18:54 .
drwxr-xr-x 16 root root 4096 Feb 8 18:55 ..
drwxr-xr-x 2 root root 4096 Feb 8 18:55 bin
drwxr-xr-x 3 root root 4096 Feb 8 18:54 include
drwxr-xr-x 4 root root 4096 Feb 8 18:54 lib
drwxr-xr-x 3 root root 4096 Feb 8 18:54 share
root@45d1a52cf28c:/# ls -al /opt/_internal/cpython-3.10.13/lib
total 16
drwxr-xr-x 4 root root 4096 Feb 8 18:54 .
drwxr-xr-x 6 root root 4096 Feb 8 18:54 ..
drwxr-xr-x 2 root root 4096 Feb 8 18:54 pkgconfig
drwxr-xr-x 36 root root 4096 Feb 8 18:55 python3.10
root@45d1a52cf28c:/# So yes py310 is installed. It was downloaded from https://github.com/indygreg/python-build-standalone/releases Is it correct ? I guess not because it breaks. |
|
You need a mingw python, so no See indygreg/python-build-standalone#14 for that source https://packages.msys2.org/package/mingw-w64-x86_64-python?repo=mingw64 has what you want I think. |
|
Ok I found the image and put it inside the container but it's getting increasingly confusing, actually. I can download and put a mingw64 distrib, say the latest 3.10 in the container. Then I see: root@5b5dd9bea925:/usr/local/mingw64# ls -al bin
total 3792
drwxr-xr-x 2 root root 4096 Jun 14 2023 .
drwxr-xr-x 6 root root 4096 Jun 14 2023 ..
-rwxr-xr-x 1 root root 104 Jun 14 2023 2to3
-rwxr-xr-x 1 root root 104 Jun 14 2023 2to3-3.10
-rwxr-xr-x 1 root root 102 Jun 14 2023 idle
-rwxr-xr-x 1 root root 102 Jun 14 2023 idle3
-rwxr-xr-x 1 root root 102 Jun 14 2023 idle3.10
-rwxr-xr-x 1 root root 3323336 Jun 14 2023 libpython3.10.dll
-rwxr-xr-x 1 root root 87 Jun 14 2023 pydoc
-rwxr-xr-x 1 root root 87 Jun 14 2023 pydoc3
-rwxr-xr-x 1 root root 87 Jun 14 2023 pydoc3.10
-rwxr-xr-x 1 root root 3776 Jun 14 2023 python-config
-rwxr-xr-x 1 root root 99407 Jun 14 2023 python.exe
-rwxr-xr-x 1 root root 3776 Jun 14 2023 python3-config
-rwxr-xr-x 1 root root 3776 Jun 14 2023 python3.10-config
-rwxr-xr-x 1 root root 99407 Jun 14 2023 python3.10.exe
-rwxr-xr-x 1 root root 99407 Jun 14 2023 python3.exe
-rwxr-xr-x 1 root root 97871 Jun 14 2023 python3w.exe
-rwxr-xr-x 1 root root 97871 Jun 14 2023 pythonw.exe
root@5b5dd9bea925:/usr/local/mingw64# But it's python.exe in a linux container. Then of course python is not found: # inside container:
root@5b5dd9bea925:/usr/local/mingw64# echo $PATH
/usr/local/mingw64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
root@5b5dd9bea925:/usr/local/mingw64# which python
root@5b5dd9bea925:/usr/local/mingw64# And the I'm a bit lost in fact. I started from this image used in this ci.yml, recommended by setuptools-rust in their doc. Working from there i was trying to adapt the docker file for a cross compile to windows. Note: Generally I think having a working example connecting pyo3/setuptools enabling to cross compile to windows (which seems to be trickier than linux to linux) would be useful. The info is maybe there, though I have not found it yet, but quite disseminated. |
|
why do you need a python.exe? if now, |
I just unpacked the mingw64 python distrib dowloaded. But now I only keep the dll and put it in the path. Which gives the following, with #12 [stage-1 7/10] RUN ls -al /usr/local/python/bin
#12 0.264 total 1152
#12 0.264 drwxr-xr-x 1 root root 4096 Feb 13 00:10 .
#12 0.264 drwxr-xr-x 1 root root 4096 Feb 12 23:59 ..
#12 0.264 lrwxrwxrwx 1 root root 9 Jan 1 2021 2to3 -> 2to3-3.10
#12 0.264 -rwxrwxr-x 1 root root 128 Jan 1 2021 2to3-3.10
#12 0.264 lrwxrwxrwx 1 root root 8 Jan 1 2021 idle3 -> idle3.10
#12 0.264 -rwxrwxr-x 1 root root 126 Jan 1 2021 idle3.10
#12 0.264 -rw-r--r-- 1 root root 1110920 Feb 13 00:10 libpython3.10.dll.a
#12 0.264 -rwxrwxr-x 1 root root 256 Jan 1 2021 pip
#12 0.264 -rwxrwxr-x 1 root root 256 Jan 1 2021 pip3
#12 0.264 -rwxrwxr-x 1 root root 256 Jan 1 2021 pip3.10
#12 0.264 lrwxrwxrwx 1 root root 9 Jan 1 2021 pydoc3 -> pydoc3.10
#12 0.264 -rwxrwxr-x 1 root root 111 Jan 1 2021 pydoc3.10
#12 0.264 lrwxrwxrwx 1 root root 10 Jan 1 2021 python3 -> python3.10
#12 0.264 lrwxrwxrwx 1 root root 17 Jan 1 2021 python3-config -> python3.10-config
#12 0.264 -rwxrwxr-x 1 root root 20944 Jan 1 2021 python3.10
#12 0.264 -rwxrwxr-x 1 root root 3093 Jan 1 2021 python3.10-config
#12 DONE 0.3sThen i get the same error: cross build --target x86_64-pc-windows-gnu
Compiling pyo3-poker-eval v0.1.0 (/project)
error: linking with `x86_64-w64-mingw32-gcc-posix` failed: exit status: 1
|
(...)
= note: /usr/bin/x86_64-w64-mingw32-ld: cannot find -lpython310
collect2: error: ld returned 1 exit statusIs there a Dockerfile that is tested to perform such cross compile from linux to win ? Here is the current Dockerfile: FROM ghcr.io/cross-rs/x86_64-pc-windows-gnu:edge
RUN apt install zstd -y
# download python version - https://github.com/indygreg/python-build-standalone/releases
ENV DATE=20240107
ENV VERSION=cpython-3.10.13
ENV URL_ASSET=https://github.com/indygreg/python-build-standalone/releases/download/$DATE/$VERSION+$DATE-x86_64-unknown-linux-gnu-install_only.tar.gz
RUN curl -L $URL_ASSET | tar -xz -C /usr/local
RUN ls -al /usr/local
ENV URL_ASSET=https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-python-3.10.12-1-any.pkg.tar.zst
RUN curl $URL_ASSET > /tmp/pkg.tar.zst && \
tar -axvf /tmp/pkg.tar.zst -C /tmp && \
rm /tmp/pkg.tar.zst
RUN cp /tmp/mingw64/lib/libpython3.10.dll.a /usr/local/python/bin/
RUN ls -al /usr/local/python/bin
ENV PATH=/usr/local/python/bin:$PATH
RUN python3 -VHow would you go about debugging this ? |
|
Put it in /usr/x86_64-w64-mingw32 |
|
Excuse me if my answers are short also, if you're not able to solve this soon I'll try to reproduce it myself and see what's wrong! |
It turns out I had to put it in /lib below and change its name: ENV URL_ASSET=https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-python-3.10.12-1-any.pkg.tar.zst
RUN curl $URL_ASSET > /tmp/pkg.tar.zst && \
tar -axvf /tmp/pkg.tar.zst -C /tmp && \
cp /tmp/mingw64/lib/libpython3.10.dll.a /usr/x86_64-w64-mingw32/lib/libpython310.dll.a && \
rm /tmp/pkg.tar.zst && \
rm -rf /tmp/mingw64Then in container - to be sure: docker run --rm --name cross -it cross-pyo3:x86_64-pc-windows-gnu /bin/bash
root@9e2b18f2f5c2:/# cd /tmp
root@9e2b18f2f5c2:/tmp# ls
home wine
root@9e2b18f2f5c2:/tmp# cd /usr/x86_64-w64-mingw32/lib
root@9e2b18f2f5c2:/usr/x86_64-w64-mingw32/lib# ls -al | grep python
-rw-r--r-- 1 root root 1110920 Feb 13 08:11 libpython310.dll.a
root@9e2b18f2f5c2:/usr/x86_64-w64-mingw32/lib# Then it works - or at least does not break - could not test wheel produced yet - as cross is part of the build managed by
You're all excused ! On the contrary, your help is both swift and critical 👍 BUT: I still think the doc lags in this respect: Unless I missed the relevant section, it does not cover this tricky "linux2win" cross compile config. I am willing to help document it (while it's hot in my mind). Do you think it's a good idea ? If so where would you see this addition ? Assuming it was written, it could then be linked from this pyo3 doc cross compile section. At the bottom there is a link to cross-rs/cross which is how I landed here in the first place. EDIT: |
|
Yes, I think documenting it is a good idea. We are currently however reworking the docs a bit, see #1438. The place to put it would be in the wiki or when that pr is merged in the docs folder. I have #1357 planned to improve the situation a bit. For now though, feel free to add to https://github.com/cross-rs/wiki_assets/tree/main/Recipes where we can then incorporate it |
|
Ok I will try to add a minimalistic example in a folder in the wiki recipes. In the meantime, the repo from which the above is taken is oscar6echo/pyo3-poker-eval. |
|
NOTE: For that I used a manylinux container. |
|
I'm not sure I follow, what would cross possibly be able to help with here? |
|
Well nothing I reckon now 🤔 Sorry for the noise - and thx again for the alway fast replies 🙏 |
|
I revisit this conversation because I have a similar case and reach an unexpected runtime error... I can create a wheel using this reproduceable procedure. I run it from ubuntu and expect to produce a wheel for windows, which I do. Everything ends up without error:
And the wheel is in dist folder:
But on Windows10, it crashes at runtime: polars.exception.ComputeError: error loading dynamic library: LoadLibrryExW failedSomehow the building was not good. FROM ghcr.io/cross-rs/x86_64-pc-windows-gnu:edge
# FROM ghcr.io/cross-rs/x86_64-pc-windows-gnu:0.2.5
# download linux python - https://github.com/indygreg/python-build-standalone/releases
# make it system python
# latest as of 6nov24
ENV DATE=20241016
# latest 3.10 as of 6nov24
ENV VERSION=cpython-3.10.15
# ENV VERSION=cpython-3.10.13
ENV URL_ASSET=https://github.com/indygreg/python-build-standalone/releases/download/$DATE/$VERSION+$DATE-x86_64-unknown-linux-gnu-install_only.tar.gz
RUN apt install zstd -y && \
curl -L $URL_ASSET | tar -xz -C /usr/local
# download mingw64 python - https://repo.msys2.org/mingw/mingw64/
# keep only dll
# latest 3.10 as of 6nov24
ENV URL_ASSET=https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-python-3.10.12-1-any.pkg.tar.zst
# debug cf issue https://github.com/cross-rs/cross/issues/1445
RUN curl $URL_ASSET > /tmp/pkg.tar.zst && \
tar -axvf /tmp/pkg.tar.zst -C /tmp && \
cp /tmp/mingw64/lib/libpython3.10.dll.a /usr/x86_64-w64-mingw32/lib/libpython310.dll.a && \
cp /usr/x86_64-w64-mingw32/lib/libpython310.dll.a /usr/x86_64-w64-mingw32/lib/libpython3.dll.a && \
rm /tmp/pkg.tar.zst && \
rm -rf /tmp/mingw64
ENV PATH=/usr/local/python/bin:$PATHHow would you go about debugging ?? |
|
I am investingating a higher level route to cross compile to win, using maturin. # ------- build windows wheel
# debian & co
sudo apt-get install mingw-w64
# redhat & co
sudo dnf install mingw64-gcc
maturin build --release --target x86_64-pc-windows-gnu --out distShould be able to test soon and hopefully close the issue. |
|
It works with maturin. There was a specific issue with lib polars. Cf. discussion. The method described previously was working just the same in fact. |
Checklist
Describe your issue
I have a pyo3 lib that compiles fine in linux.
I wanted to cross compile to windows, so I did:
with the following context:
And it fails with a very specific miss:
Note: I used the same
cargo build --target x86_64-pc-windows-gnuon a a hello world project and it works fine.Is this a bug ? Else what is missing to cross compile ?
What target(s) are you cross-compiling for?
No response
Which operating system is the host (e.g computer cross is on) running?
What architecture is the host?
What container engine is cross using?
cross version
cross 0.2.5
Example
See above
Additional information / notes
Hopefully this is enough info.
The text was updated successfully, but these errors were encountered: