[SR-6317] Package Manager's .pc parsing makes it difficult to selectively shadow system libraries

[Lukasa]

Previous ID	SR-6317
Radar	rdar://problem/40235197
Original Reporter	@Lukasa
Type	Bug

Environment

Linux 1779df4f8699 4.9.49-moby #1 SMP Wed Sep 27 23:17:17 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Swift version 4.0.2 (swift-4.0.2-RELEASE)

Additional Detail from JIRA

Votes	0
Component/s	Package Manager
Labels	Bug
Assignee	None
Priority	Medium

md5: e849c854eac92b5e5de3cdf3e167fa11

Issue Description:

I'm running a Swift project where I want to link mostly against system libraries (e.g. the system copy of icu-uc), but want to selectively shadow one or two other system libraries with different versions. This should be something that can mostly be solved by using pkg-config, as pkg-config allows us to give SwiftPM library search paths. This means, for example, if you're using icu-uc.pc you should expect to see the following flags added:

$ pkg-config --libs icu-uc
-licuuc -licudata

However, SwiftPM's parsing of .pc files doesn't behave exactly like that. Instead it behaves like pkg-config with the optional PKG_CONFIG_ALLOW_SYSTEM_LIBS flags set:

$ env PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 pkg-config --libs icu-uc
-L/usr/lib/x86_64-linux-gnu -licuuc -licudata

This means that if some libraries are being added that shadow system libraries, and some are not, you will get both -L/my/shadow/path and -L/usr/lib/x86_64-linux-gnu in the command line. Depending on the order in which they appear this may mean that the attempt to shadow the system libraries fails, as the system library path may end up first in the search path again.

I'd propose that SwiftPM should behave more like pkg-config without the PKG_CONFIG_ALLOW_SYSTEM_LIBS environment variable set, and refuse to put the system library path in a -L flag.

[aciidgh]

We already do this for macOS by removing /usr/include and /usr/lib. I wonder if there is a way to ask pkg-config about the system libary paths so we can strip them off. However, that won't work if pkg-config is not installed on the system.

[Lukasa]

I don't think we can do that for pkg-config anyway: this seems to be data compiled into the binary and not exposed through any of its interfaces. However, you can use the following dirty hack to work out whether a given package is using the system library path:

root@e1f25860f73b:/code# pkg-config --libs openssl
-lssl -lcrypto
root@e1f25860f73b:/code# env PKG_CONFIG_SYSTEM_LIBRARY_PATH="" pkg-config --libs openssl
-L/usr/lib/x86_64-linux-gnu -lssl -lcrypto

That is, you can override the system library path with the empty string.

Of course, if we were going to rely on this or something like it then we'd have to ask ourselves why we weren't just shelling out to pkg-config directly in the first instance, which would clearly resolve the issue.

[Lukasa]

The easiest proposed fix, if you're interested in having it, would just be to defer to pkg-config directly on systems that have it installed. That is, if we can locate a pkg-config on the path then rather than use the custom logic in PkgConfig.swift we just call it directly.

The downside of that mechanism is that we will inevitably behave differently on different systems, which can lead to strange and difficult-to-debug issues.

Another proposal, then, is to just manually provide these for supported Linux distributions. They can be obtained by grabbing source packages for the supported pkg-config installers and looking in their build scripts. For example, Ubuntu 14.04's pkg-config package provides the following as the system library path:

/lib:/lib/i386-linux-gnu:/lib/x86_64-linux-gnu/4.8:/lib/x86_64-linux-gnux32:/lib32:/libx32:/usr/lib:/usr/lib/i386-linux-gnu:/usr/lib/x86_64-linux-gnu/4.8:/usr/lib/x86_64-linux-gnux32:/usr/lib32:/usr/libx32

This somewhat opaque string is the result of a build script that basically does this:

OPTS=$(gcc -print-multi-lib | sed -n -e's/.*;@/-/p')
for opt in $OPTS; do \
     gcc "$opt" -print-search-dirs | sed -n -e's/^libraries: =//p' \
     | sed -e's/:/\n/g' | xargs -n1 readlink -f | grep -v 'gcc\|/[0-9.]\+$$'; \
    done | sort -u | tr '\n' : | sed 's/:$$//'
echo ""

For Ubuntu 16.04 this is:

/lib:/lib/i386-linux-gnu:/lib/x86_64-linux-gnu/5:/lib/x86_64-linux-gnux32:/lib32:/libx32:/usr/lib:/usr/lib/i386-linux-gnu:/usr/lib/x86_64-linux-gnu/5:/usr/lib/x86_64-linux-gnux32:/usr/lib32:/usr/libx32

For Ubuntu 16.10 this is:

/lib:/lib/i386-linux-gnu:/lib/x86_64-linux-gnu/6:/lib/x86_64-linux-gnux32:/lib32:/libx32:/usr/lib:/usr/lib/i386-linux-gnu:/usr/lib/x86_64-linux-gnu/6:/usr/lib/x86_64-linux-gnux32:/usr/lib32:/usr/libx32

You'll note these strings are not quite identical: in particular, they contain paths that are versioned by what I presume to be the version of libc on the system. This does lead to the annoyance that we actually have to maintain three of these opaque strings. It gets worse if we want to support different architectures for these distributions, or different distributions: each entry in the arch/distro/version matrix gets its own opaque string. That will get annoying, fast.

That makes my vote for fixing this problem go towards just letting pkg-config do this when it's installed on the system, and having the Swift code available as a fallback for systems without a pkg-config.

[Lukasa]

After some discussion with my colleagues we've come up with an alternative: we can ask clang what it thinks the system library paths are. This will probably work a bit better, because 100% of systems compiling Swift with SwiftPM that require .pc parsing will have a clang installed (otherwise they won't be able to build), and they will use clang to do that build. The end result is that gcc's list might not quite match clang's list, and in that case we'd like to use clang's instead of gcc's.

It was even suggested that we might be able to do this at SwiftPM build time, to avoid the overhead of checking each time we run SwiftPM. I'll investigate both these options to see how viable they are.

[Lukasa]

Ok, so assuming we're using the regular ld linker, we can just ask ld. We get output like this (Ubuntu 14.04):

# ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012
SEARCH_DIR("/usr/x86_64-linux-gnu/lib64")
SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib/x86_64-linux-gnu")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib")
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")

So that would make a runtime check possible and fairly straightforward to parse.

[Lukasa]

So CMAKE_SYSTEM_LIBRARY_PATH should be the correct variable to pass, assuming we're built on the right system. I want to double-check though.

[Lukasa]

In point of fact, we'd need to replicate cmake's whole logic here, because it's non-trivial:

   <prefix>/lib/<arch> if CMAKE_LIBRARY_ARCHITECTURE is set, and
   <prefix>/lib for each <prefix> in CMAKE_SYSTEM_PREFIX_PATH
   CMAKE_SYSTEM_LIBRARY_PATH
   CMAKE_SYSTEM_FRAMEWORK_PATH

This probably gets us pretty close to the correct answer in most cases, as well. That said, asking ld is probably easier in general.

[Lukasa]

Hrm. The downside to "just ask ld" is that ld may not be the linker in use. Swift already supports ld and gold as Linkers on Linux, and there seems to be an open Swift PR that adds support for, amongst other things, lld. This complicates matters somewhat, and suggests that the build system may be the better place to do this work.

Probably the easiest approach is to ask the linker what its paths are as part of the build script. I'll investigate how tricky this is to do.

[Lukasa]

However, swiftc allows you to override the linker in use, which means SwiftPM does too (-Xswiftc -use-ld=gold, for example).

If we think lld, gold, and ld.bfd will all disagree on what the system library paths are, we would really have to do this dynamically to ensure we agree with the actual linker in use. If we think they won't disagree, it may be enough to just ask ld.bfd (or any of the other linkers) to get the answer.

[Lukasa]

I suppose while I'm enumerating all the options, we could also parse the output of ldconfig -v, which lets us ask the runtime linker what it thinks the search paths are.

[Lukasa]

Oh, I should note this problem also affects cflags: PKG_CONFIG similarly checks a PKG_CONFIG_ALLOW_SYSTEM_CFLAGS environment variable to determine whether to emit the default include path. I suspect that SwiftPM will encounter similar problems here.

[aciidgh]

@swift-ci create