AppImage builds: Exclude more system libs

[ferdnyc]

In #3010 a user reported crashing on startup, trying to run the latest Daily Builds under Ubuntu 19.04.

TL;DR

I had the GitLab builders build a standard AppImage from this PR branch, meaning it excluded the libxcb-* plugin libs and the font libs (see below). That build fixed the launch crashes in Ubuntu 19.04, for the user in #3010, and also runs fine on my Fedora 30 system. So that's 2 in the win column.

If anyone has a different distro/release they can test it on, though, you can download OpenShot-v2.4.4-dev2-1569159864-f8c180c1-f17a7ed4-x86_64.AppImage, which was built from this PR branch, here: https://drive.google.com/open?id=1bqSudb-MITqqNJoJrKZub1WHuN-eoicN

Bundled-vs-system library conflicts?

The traceback pointed to a segfault in libICE.so.6, which is a system (non-bundled) lib, but it was immediately after calls into libqxcb.so which is a bundled lib. My theory was that the system and bundled lib versions were separated by too much time and too many updates to still be fully compatible.

libxcb- plugin exclusions

I picked apart the dependency tree some, and noticed that while we're bundling libqxcb.so (which is a Qt plugin, so that makes sense), and we're excluding libSM, libICE, and libxcb (which are common system libs, so that also makes sense), we're bundling all of the other parts of libxcb — that's where I suspect we get into trouble.

For example, a Daily Build from around two months ago:

$ ls /tmp/.mount_1cXWx1/usr/bin/ -R|grep -i xcb
libX11-xcb.so.1
libxcb-dri2.so.0
libxcb-dri3.so.0
libxcb-glx.so.0
libxcb-icccm.so.4
libxcb-image.so.0
libxcb-keysyms.so.1
libxcb-present.so.0
libxcb-randr.so.0
libxcb-render.so.0
libxcb-render-util.so.0
libxcb-shape.so.0
libxcb-shm.so.0
libxcb-sync.so.1
libxcb-util.so.0
libxcb-xfixes.so.0
libxcb-xkb.so.1
libqxcb.so

So, I added any library filename that startswith("libxcb-") to the exclusion list, since those are all plugins that should really be kept in version sync with the main libxcb.so.1.

I didn't exclude libqxcb.so of course, and I didn't exclude libX11-xcb.so.1. (Though maybe I should've, since we do exclude libX11.so.6 and I believe they're also distributed together. I just wasn't entirely 100% sure whether Wayland systems would all be guaranteed to have libX11-xcb.so.1 installed.)

font library suite exclusions

I also added these additional libs to the exclusion list:

                            "libfreetype.so.6",
                            "libfontconfig.so.1",
                            "libcairo.so.2",
                            "libpango-1.0.so.0",
                            "libpangocairo-1.0.so.0",
                            "libpangoft2-1.0.so.0",
                            "libharfbuzz.so.0",
                            "libthai.so.0",

Every library there except the main libcairo.so.2 is part of an interdependent suite that makes up the Linux desktop font rendering system. (libthai.so.0 is part of Pango. libpangoft2 is Pango's FreeType2 interface. Harfbuzz is an OpenType layout implementation. Other related candidates for exclusion, if we were bundling them in the first place: FriBiDi (a bi-directional script handling library), libpangoxft, libpangomm. But we're already not bundling them.

We've been bundling all of the ones listed above for a long while... I believe unnecessarily. Most of them are on the pkg2appimage exclusion list (or they're dependencies of libs on that list), and AFAIK should be present on any current linux distro. As long as they're all excluded together, shouldn't be any problems, and it'll prevent issues like the startup logspam from that outdated copy of libfontconfig we've been bundling.

libcairo.so.2 is the only questionable one, but as the comments in that pkg2appimage list note, it's closely tied in to pango and bundling cairo without pango can cause problems. Plus, it should be present on basically any Linux system. libcairo.so.1 hasn't existed since 2006.

Fixes #3010, fixes #1775

[ferdnyc]

The more I think about it, the more I think that if we're excluding both libX11.so.6 AND libxcb.so.1, we should also exclude libX11-xcb.so.1, which the package on my Fedora system describes as the "libX11/libxcb interoperability library". It just doesn't make sense that we'd bundle any specific version of that, without bundling either of the libraries it makes interoperable. There's way too much danger of version mismatch problems, and it's just nonsensical that a system which contains both libX11.so.6 and libxcb.so.1 would be lacking libX11-xcb.so.1. I'm going to push in another commit adding that file to the exclusion list.

[wilx]

Not that I understand this fully but isn't it possible to take the other extreme approach and bundle in everything except glibc? X11 has to be on wire compatible AFAIK, so that should work, IMHO.

[ferdnyc]

Not that I understand this fully but isn't it possible to take the other extreme approach and bundle in everything except glibc? X11 has to be on wire compatible AFAIK, so that should work, IMHO.

In theory, yes, and in the Flatpak / Snap / etc. model that's basically what's done, packages are practically run in fully-isolated containers. In practice with AppImages, though, that doesn't work out particularly well, because the executable is still running inside the host system. That implies a certain generation of libraries being used by other processes it has to communicate with — like the display server, window manager, etc. — as well as a system tuned to those libraries.

(The fontconfig thing is a perfect example. Fontconfig released a new version a year or two ago which expanded its config file grammar. When an older version of libfontconfig.so is loaded by an application running on a system that was installed with the newer version, the older libfontconfig complains mightily about all of the /etc/fonts.d/ files that contain XML tags/attributes it doesn't know how to parse. ...Fortunately, all it does is complain, and not crash, but you see my point. Unless you're truly running completely isolated from the host system, you're better off using the libraries installed on it, at least the ones you can depend on being available pretty much universally.)

[jonoomph]

This looks good to me, but I'm not 100% sure if this is more or less compatible that what we currently use. Have we verified this PR works with the latest version of Distros (at least the few big ones)? If so, I'll happily merge it. 😉

[ferdnyc]

Conflict resolved, so that's the first step.

This looks good to me, but I'm not 100% sure if this is more or less compatible that what we currently use.

Basically, the way I've come to understand the situation is: Bundling fewer libs is generally MORE compatible (counterintuitively), if and only if those libs are "standard equipment" that can be assumed to be present on pretty much any Linux system. (Or any Linux system capable of running the app, at least, since we require a GUI and a desktop session as part of the existing environment, so e.g. a headless server install doesn't count.)

So, the question becomes: Are the libs being excluded really all "standard" libs?

IMHO the answer is a resounding "yes" in most cases, and a fairly safe gamble worth taking on the rest. We've seen the issues that a bundled libfontconfig can cause, for instance, and I challenge anyone to find a desktop Linux build anywhere that doesn't already provide its own libfontconfig. No-brainer, excluding that one.

The ones I admit to considering questionable, myself, are the edge cases like the pango and cairo components. I believe those can be treated as "universally available", at this point, but I won't deny there's a certain amount of finger-crossing involved on that.

Have we verified this PR works with the latest version of Distros (at least the few big ones)?

Ubuntu yes, Fedora yes, but so far that's all. I'd love to have a wider sample, but that requires users of other distros to step up and do the testing. Results from Arch, Mint, Manjaro, MX, and Solus would be great, and they'd really round out the "popular kids" category nicely.

Results from some older CentOS/RHEL, Ubuntu LTS, and Debian versions would be helpful as well, since the intention here is to remain compatible with even older releases/installs, while not messing up newer ones by bundling outdated versions of system libs we could be pulling from the host environment. But I'm less worried about latest-version compatibility (in the sense that I consider it a near-certainty this will improve things on that front) than I am about possibly breaking things for users in less bleeding-edge scenarios.

But, ultimately "Not extensively" would be the short answer to this question:

Have we verified this PR works with the latest version of Distros (at least the few big ones)?

No, this hasn't been tested as widely or as much as I'd like it to be tested. However, anecdotal evidence so far is positive, and it's been tested to the full extent of my own abilty to test it, current.

[jonoomph]

Okay, I'm convinced! 😜 I like your thought on "less is more". If this has widespread breakages, I suppose we could always make a tweak, and do a quick bug-fix release. I might also have time this week to test on a few non-Ubuntu distros in your list above.

[ferdnyc]

Basically, the way I've come to understand the situation is: Bundling fewer libs is generally MORE compatible (counterintuitively), if and only if those libs are "standard equipment" that can be assumed to be present on pretty much any Linux system.

I guess an even better way to phrase/conceptualize that, really, is: With an AppImage, the packaged executable is still running in the host environment (not containerized/virtualized).

Which means whenever a library is bundled that also will be present and in use on/by the host system (think libwayland, libfreetype, or libglib, which are going to be loaded from the host system even if they're bundled, because the Xorg and mutter and gnome-shell and etc. processes have already loaded them), the bundling adds another dimension to the matrix of versions/dependencies that all have to be at least nominally compatible between the package and the host system(s). So, each added "system" library creates a set of additional opportunities to encounter some incompatibility with a host environment.

(That's not true for things like, say, the Qt libs. Even if Qt is installed on the host system, it'll just be ignored as long as the necessary components are all correctly bundled. So, it's safe, and desirable, and the point of AppImages, to bundle that kind of stuff.)

So, what you want to bundle are any and all application libraries, plus any system libraries that need to be included because at least some of the target platforms the app is expected to run in will lack that library. And then you want to stop right there, bundling only the minimal set of those libraries, and nothing more. (Because of the whole "bundling more == deeper compatibility matrix with more opportunities for breakage" thing.)