Allow inclusion in static builds

[cromefire]

I'm trying to build a static version ffmpeg including rav1e, but it seems like one can't compile a static library that includes lgcc_s, because there's only a shard version of it.

gcc-10 -L<prefix>/lib -I<prefix>/lib -L/usr/lib/gcc/x86_64-linux-gnu/10 -static -Wl,--as-needed -Wl,-z,noexecstack -I<prefix>/include/rav1e -L<prefix>/lib -o /tmp/ffconf.EpJq5glV/test /tmp/ffconf.EpJq5glV/test.o -lrav1e -lgcc_s -lutil -lrt -lpthread -lm -ldl -lc -lgcc_s -lutil -lrt -lpthread -lm -ldl -lc -lm -lpthread
/usr/bin/ld: cannot find -lgcc_s
collect2: error: ld returned 1 exit status
ERROR: rav1e >= 0.1.0 not found using pkg-config

I'm not a lot into C so I'm not sure if that is something that even can be changed at all or if there's something I couldn't find that's allows this (although I searched quite thorough and all of the "solutions" simply didn't work or aren't applicable).

[lu-zero]

It seems a local problem. You seem to be missing a base library.

[cromefire]

Yeah I do have libgcc_s.so, but that's not going to work for a static (libgcc_s.a) build and that's doesn't seem to be available as static library (for technical reasons I guess)

[cromefire]

Ref: https://packages.ubuntu.com/search?searchon=contents&keywords=libgcc_s.a&mode=filename&suite=focal&arch=amd64 (the few results are only mingw, which probably isn't really helpful in this instance)

[cromefire]

If you look it up, you see a lot of people suggesting liking it dynamically, which isn't quite possible here, because it's defined via pkgconfig: https://stackoverflow.com/a/18803598/8220327

[cromefire]

For reference the generated .pc file looks like this:

prefix=/path/to/the/prefix
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: rav1e
Description: The fastest and safest AV1 encoder
Version: 0.4.0
Libs: -L${libdir} -lrav1e  -lgcc_s -lutil -lrt -lpthread -lm -ldl -lc
Cflags: -I${includedir}/rav1e
Libs.private:  -lgcc_s -lutil -lrt -lpthread -lm -ldl -lc

[barrbrain]

I see in https://askubuntu.com/a/1190778 that someone recommends removing the rav1e shared library to build ffmpeg. We should look into why this workaround is necessary for some configurations.

[cromefire]

I'm actually building without shared libraries:

RUSTFLAGS="-C target-cpu=native" cargo cinstall --release --locked --prefix "${prefix}" --library-type staticlib

I did not see any difference vs not doing that though

[cromefire]

To be clear: If I'm not linking it statically it's working just fine, it's only having problems while linking statically

[tdaede]

FWIW I double checked and on Fedora Linux and my .pc also has -lgcc_s.

I did notice there is a -static-libgcc option to gcc, however I've not tested it. (I'm also not sure why rav1e depends on libgcc_s...)

prefix=/usr/local
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: rav1e
Description: The fastest and safest AV1 encoder
Version: 0.5.0-alpha
Libs: -L${libdir} -lrav1e
Cflags: -I${includedir}/rav1e
Libs.private:  -lutil -ldl -lutil -lgcc_s -lutil -lrt -lpthread -lm -ldl -lc

[cromefire]

I tested -static-libgcc and it didn't work, I tried to remove the -lgcc_s and it did actually compile (it requires manually modifying the .pc file though), I'll check later if that still works during runtime

[barrbrain]

I note that media-autobuild_suite is also using this workaround for static builds:
https://github.com/m-ab-s/media-autobuild_suite/blob/962658b9/build/media-suite_compile.sh#L1077-L1088

[cromefire]

It seems to be target at Windows though so there might be a difference there (likely using mingw, which seems to bring a static version of libgcc_s)

[tdaede]

Apparently Rust's own unwinder normally depends on libgcc_s. I'm not sure if we can disable the unwinder (or would even want to - it probably requires abort on panic), though I did find this option for using libunwind instead, which might be easier to statically link. Maybe we should do this by default. rust-lang/rust#59089

[cromefire]

Even if not by default (one probably should benchmark it and look at the consequences for compilation and usage), it might make sense to add it as an option if that's possible

[lu-zero]

You should open the issue on the rust side, we receive the link line directly from the compiler.

[cromefire]

But is it the compiler though or the cargo-cinstall thingy (or rather the bindgen behind it)?

[cromefire]

So I tested and this works as a workaround (I haven't tested a panic though):

mv "${prefix}/lib/pkgconfig/rav1e.pc" "${prefix}/lib/pkgconfig/rav1e.pc.orig"
sed 's/ -lgcc_s/ /g' "${prefix}/lib/pkgconfig/rav1e.pc.orig" > "${prefix}/lib/pkgconfig/rav1e.pc"
rm "${prefix}/lib/pkgconfig/rav1e.pc.orig"

[lu-zero]

It is the compiler --print native-static-libs producing that link line.

[cromefire]

Okay do you want to report it or should I? Because I don't really have experience and probably can't really provide Infos to them

[tdaede]

Just so I can mess around with this a bit, are you statically linking libc as well? If so, which libc and/or target triple are you using?

[cromefire]

I'm not entirely sure, I think it is, but I can provide you with a little shell script that should be able to reproduce it

[cromefire]

If so, which libc and/or target triple are you using

libc is probably gcc default, so glibc6 and target triple is omitted, so it should be x86_64-unknown-linux-gnu

[cromefire]

You can download this, put it into an empty dir and execute it and it'll download ffmpeg and rav1e (into the folder of the script) and compile both locally with the prefix set to <folder of script>/prefix:

repro.sh

#!/bin/bash

set -e
cd "$(dirname "${0}")"

prefix="$(pwd)/prefix"

mkdir -p "${prefix}"

export CFLAGS="$CFLAGS -O3" CXXFLAGS="$CFLAGS"
export LDFLAGS="$LDFLAGS -L${prefix}/lib"
export LD_LIBRARY_PATH="${prefix}/lib"
export PKG_CONFIG_PATH="${prefix}/lib/pkgconfig"

ffmpeg_version="4.3.2"
rav1e_version="0.4.0"

echo "Fetching repositories..."
if [[ -d rav1e ]]; then
    pushd rav1e
    git fetch
else
    git clone https://github.com/xiph/rav1e.git rav1e
    pushd rav1e
fi
git checkout "v${rav1e_version}"
popd

if [[ -d ffmpeg ]]; then
    pushd ffmpeg
    git fetch
else
    git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
    pushd ffmpeg
fi
git checkout "n${ffmpeg_version}"
popd

echo "Building rav1e..."
pushd "rav1e"
echo " - Building..."
RUSTFLAGS="-C target-cpu=native" cargo cinstall --release --locked --prefix "${prefix}" --library-type staticlib
# tmp fix, because static build fails
#echo " - Fixing static build..."
#mv "${prefix}/lib/pkgconfig/rav1e.pc" "${prefix}/lib/pkgconfig/rav1e.pc.orig"
#sed 's/ -lgcc_s/ /g' "${prefix}/lib/pkgconfig/rav1e.pc.orig" >"${prefix}/lib/pkgconfig/rav1e.pc"
#rm "${prefix}/lib/pkgconfig/rav1e.pc.orig"
popd

echo "Building ffmpeg..."
pushd ffmpeg
echo " - Configuring..."
./configure \
    --prefix="${prefix}" \
    --extra-libs="-lm -lpthread" \
    --pkg-config-flags="--static" \
    --extra-cflags="-I${prefix}/include -static -w -O3" \
    --extra-ldflags="-static" \
    --disable-runtime-cpudetect \
    --disable-doc \
    --disable-pic \
    --disable-shared \
    --enable-static \
    --disable-autodetect \
    --disable-ffplay \
    --disable-ffprobe \
    --disable-debug \
    --enable-librav1e
echo " - Building..."
make clean
make -j "$(nproc)"
popd

The configuration log of ffmpeg (with that error) can be found at <folder of script>/ffmpeg/ffbuild/config.log. The workaround is also included, but commented out.

The resulting binary will be at <folder of script>/ffmpeg/ffmpeg

[cromefire]

With the workaround I get:

~$ ldd ffmpeg/ffmpeg
        not a dynamic executable

So I guess libc is statically linked.

[lu-zero]

Okay do you want to report it or should I? Because I don't really have experience and probably can't really provide Infos to them

Please do and also open an issue on cargo-c. I can probably add a workaround on the cargo-c side.

[cromefire]

Can you give me more info on where the compiler generates the list of libraries, so I can report something substantial to them?
Also what issue exactly do you want me to raise at cargo-c?

[lu-zero]

If you run on a crate with staticlib the following:

cargo rustc --lib -- --print native-static-libs

You'd get:

note: Link against the following native artifacts when linking against this static library. The order and any duplication can be significant on some platforms.

note: native-static-libs: -lutil -lutil -lgcc_s -lutil -lrt -lpthread -lm -ldl -lc

For rav1e you'd have to add the crate types to Cargo.toml like this:

diff --git a/Cargo.toml b/Cargo.toml
index 65b87ef4..fc1bfa52 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -138,6 +138,7 @@ required-features = ["binaries", "channel-api", "unstable"]
 bench = false

 [lib]
+crate-type = ["staticlib", "rlib", "cdylib"]
 bench = false

 [[bench]]

cargo-c automates this (and lots more), but the link line is produced by rustc and it is not correct for your use-case. We could discuss upstream if isn't better to use libgcc_eh.a and libgcc.a instead.

I confirm that panic=abort does not omit -lgcc_s from the link line so potentially it uses something more than the unwinder.

[cromefire]

Filed a report for the compiler, I'll see what's the result of that and then I think be can look into whether there's something to be done on the cargo-c part of it.

[lu-zero]

Can we consider the problem now solved?