Use pkgconfig for cross-compiling to a -pc-windows-gnu target

[micolous]

Using pkgconfig for -pc-windows-gnu targets on non-Windows hosts is currently blocked:

rust-openssl/openssl-sys/build/find_normal.rs

Lines 201 to 209 in 8909d3e

	// If we're going to windows-gnu we can use pkg-config, but only so long as
	// we're coming from a windows host.
	//
	// Otherwise if we're going to windows we probably can't use pkg-config.
	if target.contains("windows-gnu") && host.contains("windows") {
	env::set_var("PKG_CONFIG_ALLOW_CROSS", "1");
	} else if target.contains("windows") {
	return;
	}

It looks like it was originally entirely blocked on Windows in d465f6c

This was then changed to allow it for windows-gnu from a windows host: 43c951f#diff-b088de9b758aec6c173f92e9216ce8ce9597855d542a5d95d14b526d9259f334R140

There's probably a reason for this... but I don't know what, and this was a few years ago... so may not be needed anymore?

If the block on non-Windows hosts was removed:

    if target.contains("windows-gnu") {

Then it's possible to use a vcpkg-provided OpenSSL installation on a non-Windows host:

# the location where you checked out vcpkg to
export VCPKG_ROOT="$HOME/vcpkg"
export PKG_CONFIG_PATH_x86_64_pc_windows_gnu="$VCPKG_ROOT/installed/x64-mingw-static/lib/pkgconfig" 

vcpkg install openssl:x64-mingw-static
cargo build --target x86_64-pc-windows-gnu  ...
wine64 target/x86_64-pc-windows-gnu/debug/...

It may also be possible to use Linux distributions' lib*-mingw-w64 packages (example), if they packaged OpenSSL.

Separately to this, I'm trying to unblock using vcpkg-rs on -pc-windows-gnu targets, but that's blocked on a few other things right now.

[sfackler]

Yeah at this point I don't think that restriction makes sense, and we should allow pkg-config on windows.

Ideally we'd remove the PKG_CONFIG_ALLOW_CROSS setting as well, but I imagine that'd be breaking.

[micolous]

I'm pretty sure PKG_CONFIG_ALLOW_CROSS is required (either in openssl-sys or every downstream package), even on Windows (eg: building i686 binaries on an x86_64 host). Though that is a feature of pkg-config-rs rather than pkg-config itself:

https://github.com/rust-lang/pkg-config-rs/blob/80657406d633630af0f22a4b54dc95487d4f9beb/src/lib.rs#L426-L436

This environment variable was also added to openssl-sys in #464, maybe @alexcrichton can shed some light on this?

Following these recommendations of using PKG_CONFIG_SYSROOT_DIR (which would allow cross-compilation in pkg-config-rs) and PKG_CONFIG_LIBDIR gives incorrect paths (which appears to be a known bug) on a macOS host (with pkg-config 0.29.2) with Windows target:

% PKG_CONFIG_SYSROOT_DIR="$VCPKG_ROOT/installed/x64-mingw-dynamic" PKG_CONFIG_LIBDIR="$VCPKG_ROOT/installed/x64-mingw-dynamic/lib/pkgconfig:$VCPKG_ROOT/installed/x64-mingw-dynamic/share/pkgconfig" PKG_CONFIG_PATH="" pkg-config --libs --cflags openssl

-I/Users/me/vcpkg/installed/x64-mingw-dynamic/Users/me/vcpkg/installed/x64-mingw-static/lib/pkgconfig/../../include -L/Users/me/vcpkg/installed/x64-mingw-dynamic/Users/me/vcpkg/installed/x64-mingw-static/lib/pkgconfig/../../lib -lssl -lcrypto -lws2_32 -lgdi32 -lcrypt32

Changing PKG_CONFIG_LIBDIR to a relative path doesn't work:

% PKG_CONFIG_SYSROOT_DIR="$VCPKG_ROOT/installed/x64-mingw-dynamic" PKG_CONFIG_LIBDIR="lib/pkgconfig:share/pkgconfig" PKG_CONFIG_PATH="" pkg-config --libs --cflags openssl

Package openssl was not found in the pkg-config search path.
Perhaps you should add the directory containing `openssl.pc'
to the PKG_CONFIG_PATH environment variable
No package 'openssl' found

Using PKG_CONFIG_SYSROOT_DIR alone is also broken if the system-installed libraries have a different layout to the target:

% PKG_CONFIG_SYSROOT_DIR="$VCPKG_ROOT/installed/x64-mingw-dynamic" PKG_CONFIG_PATH="" pkg-config --libs --cflags openssl

-I/Users/me/vcpkg/installed/x64-mingw-dynamic/usr/local/Cellar/openssl@3/3.1.1_1/include -L/Users/me/vcpkg/installed/x64-mingw-dynamic/usr/local/Cellar/openssl@3/3.1.1_1/lib -lssl -lcrypto

Using PKG_CONFIG_PATH alone works:

% PKG_CONFIG_PATH="$VCPKG_ROOT/installed/x64-mingw-dynamic/lib/pkgconfig" pkg-config --libs --cflags openssl

-I/Users/me/vcpkg/installed/x64-mingw-dynamic/lib/pkgconfig/../../include -L/Users/me/vcpkg/installed/x64-mingw-dynamic/lib/pkgconfig/../../lib -lssl -lcrypto -lws2_32 -lgdi32 -lcrypt32

[alexcrichton]

Heh that PR is ... 7 years old at this point, so a bit of a stretch to remember specific context like this. That being said IIRC at the time mingw was not really at the point where this would "just work" for most folks, and instead it mostly caused things to not work. I recall that at the time by default pkg-config was used and by default that selects host libraries, meaning that cross-compiles were "broken by default" if the literal pkg-config executable was used. If that's been fixed in the meantime or pkg-config is more likely to work by default with external configuration or similar then I think the original rationale for this has been resolved.

[micolous]

No worries.

Cross-compiles are broken-by-default unless you set up PKG_CONFIG_PATH (etc.) environment variables, but the present state of affairs is such that it won't even try when targetting Windows from non-Windows hosts (ie: even if you have set things up properly, openssl-sys still blocks you).

There's no such restriction for other types of cross-compiles, so it seems pretty arbitrary at this point.

At this point, it sounds like there are no objections to dropping that check entirely, so I'll look at a way to make sure this path is tested properly so it can be dropped.

[micolous]

After fiddling with the CI a bit, it looks like the pkgconfig path is entirely untested for cross-compiles, because it sets OPENSSL_DIR:

rust-openssl/.github/workflows/ci.yml

Line 209 in 0f0bbe4

OPENSSL_DIR: /opt/openssl

So #1991 will just remove that restriction without touching CI.

[micolous]

Some notes on the state of Windows builds for when someone comes across this in 7 years' time 😉 :

• The logic is now that only MSVC targets skip running pkg-config: 4d2379f 41c03d6

  My assumption here is that MSVC targets will use vcpkg instead, and that MSVC will only be used on Windows hosts.

  vcpkg-installed OpenSSL provides pkg-config metadata as well, but the pkg-config binary (currently needed for pkg-config-rs) might not be installed on Windows.

• pkg-config-rs has its own logic for detecting whether a cross-compile is "properly configured". PKG_CONFIG_ALLOW_CROSS=1 will override those checks, but bugs in pkg-config may cause issues here (particularly for cross-OS compiles with different filesystem layouts).

  I haven't dug too much into this, but nothing here should have changed things.

• PKG_CONFIG_ALLOW_CROSS=1 will only be applied for Windows-GNU/Mingw targets compiled from a Windows host.

  At the moment it sounds like we don't know why this is needed on Windows hosts - maybe pkg-config-rs thinks its cross-compiling if you build with a rustup-installed *-pc-windows-gnu toolchain inside (one of the/a) mingw environment(s)?

  If you needed PKG_CONFIG_ALLOW_CROSS=1 before, you'll still need it. If you didn't need it, you'll still not need it (though this may change in future).

• I haven't investigated pkgconf's pkg-config compatibility wrapper, and whether it is bug-compatible.

• Allow running vcpkg on any Windows target #1982 unblocks using vcpkg on any Windows target (not just when the host is built with MSVC). Once Add support for mingw (-pc-windows-gnu) targets mcgoo/vcpkg-rs#52 is in, you'll be able to use it for *-pc-windows-gnu targets as well.

  While vcpkg supports non-Windows platforms, my assumption is that most people using vcpkg for rust-openssl are targeting Windows.

• rust-openssl's CI tests cross-compiling using OPENSSL_DIR, rather than pkg-config. I made an attempt at cross-compiling for x86_64-pc-windows-gnu on a Linux host (running tests in Wine), which mostly worked:

  • OpenSSL v1.0.2 doesn't build, suspect this is because of a conflicting type in the Win32 API. This version of OpenSSL is obsolete, so may not care about this.

  • OpenSSL v1.0.1 fails systests. This version of OpenSSL is obsolete.

  • BoringSSL fails to build because a test seems to be picking up a thread.h from OpenSSL rather than the C++ standard library (std::thread).

    It took quite a bit of fiddling with the build environment to get that far, and there's probably something else I've messed up.

    However, it looks like BoringSSL assumes it is building with MSVC for Windows, and the building docs state "Recent versions of GCC (6.1+) and Clang should work on non-Windows platforms, and maybe on Windows too". Given BoringSSL "is not intended for general use", I don't know how much to care about this. 😄

  It might be useful to have those as CI targets; but you'd need to exclude those three. The current state of CI for Windows only runs with the current version of OpenSSL from vcpkg, so these are already testing gaps.