Skip to content

libstdcxx version-bounds #243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Nov 20, 2022
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/src/pythoncall.md
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,23 @@ into it. If you want to use a pre-existing Conda environment, see the previous s
If `conda`, `mamba` or `micromamba` is not in your `PATH` you will also need to set
`JULIA_CONDAPKG_EXE` to its path.

#### If you installed a newer version of libstdc++
PythonCall injects a dependency to bound the allowed versions of the `libstdcxx-ng`
Conda package. It finds the bound by runtime discovery of the libstdcxx version. To
override this value, use:

```julia
[PythonCall]
ENV["JULIA_PYTHONCALL_LIBSTDCXX_VERSION_BOUND"] = ">=3.4,<=12"
```

To figure out installed version, run
```bash
strings /path/to/julia/lib/julia/libstdc++.so.6 | grep GLIBCXX
```
Then look at <https://gcc.gnu.org/onlinedocs/gcc-12.1.0/libstdc++/manual/manual/abi.html>
for the GCC version compatible with the GLIBCXX version.

## [Installing Python packages](@id python-deps)

Assuming you haven't [opted out](@ref pythoncall-config), PythonCall uses
Expand Down
59 changes: 42 additions & 17 deletions src/cpython/context.jl
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,47 @@ function _atpyexit()
return
end

# By default, ensure libstdc++ in the Conda environment is compatible with
# the one linked in Julia. This is platform/version dependent, so needs to
# occur at runtime.
#
# Allow the user to override the default. This is useful when the version
# of libstdcxx linked in Julia is customized in the local installation of
# Julia.
#
# To figure out cxx_version for a given Julia version, run
# strings /path/to/julia/lib/julia/libstdc++.so.6 | grep GLIBCXX
# then look at
# https://gcc.gnu.org/onlinedocs/gcc-12.1.0/libstdc++/manual/manual/abi.html
# for the highest GCC version compatible with the highest GLIBCXX version.
function get_libstdcxx_version_bound()
# This list comes from: https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html
# Start with GCC 4.8, as it's extremely difficult to build Julia with anything older
vers_mapping = Dict(
18 => v"4.8.0",
19 => v"4.8.3",
20 => v"4.9.0",
21 => v"5.1.0",
22 => v"6.1.0",
23 => v"7.1.0",
24 => v"7.2.0",
25 => v"8.1.0",
26 => v"9.1.0",
27 => v"9.2.0",
28 => v"9.5.0",
29 => v"11.3.0",
30 => v"12.2.0",
31 => v"13.1.0",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at https://gcc.gnu.org/develop.html#timeline there are more versions of GCC than in that other linked page. If I understand right, I think it means that we could bump the mappings to 28 => 9.5.0, 29 => 11.3.0, 30 => 12.2.0.

Copy link
Contributor Author

@samtkaplan samtkaplan Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went ahead and applied you suggestion. But, it isn't clear to me if the libstdc++ version is constant from, say, gcc 9.3.0->9.5.0

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ABI Policy page says "If a particular release is not listed, it has the same version labels as the preceding release." I take that to mean that all releases >=9.3.0 and <10.1.0 have the same version labels.

)
# Get the libstdcxx version that is currently loaded in this Julia process
loaded_libstdcxx_version = Base.BinaryPlatforms.detect_libstdcxx_version()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to gracefully handle the possibility that this returns nothing or a version other than 3.4.*.

Copy link
Contributor Author

@samtkaplan samtkaplan Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added some code in the last commit (e7c8dac) that attempts to address this, but I'm not sure what the best solution is. In particular, if Julia is not able to detect the version, then we try to get the version bound from the user environment variable, and if the user has not set a variable, then do not inject a version bound. Hopefully the last case will be rare. Note that if the version is not 3.4.*, then detect_libstdcxx_version returns nothing (if I understood the code correctly).

# Map it through to get a GCC version; if the version is unknown, we simply return
# the highest GCC version we know about, which should be a fairly safe choice.
max_version = get(vers_mapping, loaded_libstdcxx_version.patch, vers_mapping[maximum(keys(vers_mapping))])
cxx_version = ">=3.4,<=$(max_version.major).$(max_version.minor)"
get(ENV, "JULIA_PYTHONCALL_LIBSTDCXX_VERSION_BOUND", cxx_version)
end

function init_context()

CTX.is_embedded = haskey(ENV, "JULIA_PYTHONCALL_LIBPTR")
Expand Down Expand Up @@ -60,23 +101,7 @@ function init_context()
exe_path::String
else
if Sys.islinux()
# Ensure libstdc++ in the Conda environment is compatible with the one
# linked in Julia. This is platform/version dependent, so needs to occur at
# runtime.
#
# To figure out cxx_version for a given Julia version, run
# strings /path/to/julia/lib/julia/libstdc++.so.6 | grep GLIBCXX
# then look at
# https://gcc.gnu.org/onlinedocs/gcc-12.1.0/libstdc++/manual/manual/abi.html
# for the highest GCC version compatible with the highest GLIBCXX version.
if Base.VERSION <= v"1.6.2"
# GLIBCXX_3.4.26
cxx_version = ">=3.4,<9.2"
else
# GLIBCXX_3.4.29
# checked up to v1.8.0
cxx_version = ">=3.4,<11.4"
end
cxx_version = get_libstdcxx_version_bound()
CondaPkg.add("libstdcxx-ng", version=cxx_version, channel="conda-forge", temp=true, file=joinpath(@__DIR__, "..", "..", "CondaPkg.toml"), resolve=false)
end
# By default, we use Python installed by CondaPkg.
Expand Down
6 changes: 6 additions & 0 deletions test/context.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
@testitem "libstdc++ version" begin
ENV["JULIA_PYTHONCALL_LIBSTDCXX_VERSION_BOUND"] = ">=3.4,<=12"

cxxversion = PythonCall.C.get_libstdcxx_version_bound()
@test cxxversion == ">=3.4,<=12"
end