Skip to content

Desktop Linux Platform Issues

Jump to bottom
probonopd edited this page Jun 6, 2018 · 57 revisions

TL;DR: Stop making "Desktop Linux" a moving target by agreeing on a minimal baseline that third-party applicationa can take for granted to exist on each Desktop Linux system.

To make "Desktop Linux" a viable platform (as in: Windows, macOS) to develop against, we need to insist on backward compatibility and either need to find the "least common denominator" by experimentation (as we have done for over a decade now), or get the main Linux distributions (Ubuntu, Fedora, openSUSE, Debian, etc.) to agree on a certain set of infrastructure that can be expected to be "there", in a consistant way, without unnecessary differences.

TL;DR: A guaranteed minimal set of infrastructure (available in the default installation of all distributions) with guaranteed backward compatibility is required for "Desktop Linux" to become viable as a platform.

Your thoughts are highly welcome. Please join the discussion here. However, please do not make substantial edits to this page before reaching consensus in the discussion on https://github.com/AppImage/AppImageKit/issues/445.

Credit: MuseScore (CC BY 2.0)

openSUSE Chairman Richard Brown pointed out the need for this in a talk at openSUSE Conference 2017:

This is not to say that the "least common denominator" approach does not work. In fact, the arguably two most successful "Desktop Linux" applications, LibreOffice and Firefox, are built on old build systems and bundle many libraries in order to be able to deliver one set of binaries to (almost) all "Desktop Linux" systems, independent of distributions, and have been doing this for quite some time now. But given some sane platform decisions, life could be made much easier for application developers who want to target the "Desktop Linux" platform.

This page is here to collect ideas on what would be worthwhile to standardize on. It is meant as a basis to get the discussion started rather than finalized recommendations. You are welcome to edit this page to add factual information, but if you fundamentally disagree with the direction of this page please talk to us first before editing this page.

Kernel

In the over a decade experience with application bundles the kernel has proven never to be an issue, since the interface between the kernel and userland is stable, and Linus Torvalds is heavily defending a stable userland interface. So no action required as far as AppImage is concerned. Applications should ensure they can run on a vanilla kernel without extra patches.

Paths

Library filenames

The names of libraries should reflect the version as given by upstream. Currently it is not, resulting in the necessity for each application using it to bundle a private version, since the application cannot rely on the system-provided one.

Example:

  • Ubuntu 12.04 has libarchive.so.12 desipte it being version 3.x
  • Ubuntu 14.04 has libarchive.so.13 desipte it being version 3.x

This should be changed so that every distribution that carries any version of libarchive version 3.x would contain libarchive.so.3.xxx and a symlink to libarchive.so.3, so that applications could rely on being able to link against libarchive.so.3.

The libarchive upstream developers say they can do nothing about this since the library filename is decided by distributions.

We might be able to make the number more consistent across platforms, but it would require someone to do the research into how cmake and autoconf generate build numbers on the many different platforms.

References:

Other example:

  • Mageia has /usr/lib/pulseaudio/libpulsecommon-10.0.so
  • App wants libpulsecommon-4.0.so

Library paths

Despite there being the FHS, filesystem paths are different on different reason for no apparent reason. They should be standardized between all distributions.

Example:

  • Debian, Ubuntu: /lib/i386-linux-gnu/, /lib/x86_64-linux-gnu/
  • openSUSE: /lib/, /lib64/

References:

Certificates

Should be installed to a standard location that is the same in all distributions. Currently this is not the case and where certificates go is apparently entirely random by distribution, as of mid-2017:

"/etc/ssl/certs/ca-certificates.crt", // Debian/Ubuntu/Gentoo etc.
"/etc/pki/tls/certs/ca-bundle.crt",   // Fedora/RHEL
"/etc/ssl/ca-bundle.pem",             // OpenSUSE
"/etc/pki/tls/cacert.pem",            // OpenELEC
"/etc/ssl/certs",               // SLES10/SLES11, https://golang.org/issue/12139
"/system/etc/security/cacerts", // Android

Source: https://serverfault.com/questions/62496/ssl-certificate-location-on-unix-linux

The result is that if we bundle something like libgnutls, then it fails to find the certificates if the target system happens to store them in a different place.

Workaround: Patch whatever consumes certificates (e.g., libgnutls) to look in all known places. Example: https://github.com/darealshinji/vlc-AppImage/issues/1#issuecomment-321041496

Basic libraries

Applications should be able to rely on certain basic infrastructure libraries to be provided by all distributions in the default installation. They should be standardized between all distributions.

glibc

Ideally this library would be frozen indefinitely. Why is it necessary to still keep adding symbols to it, making software compiled against later versions fail on earlier versions? If freezing this is not realistic, then at least build systems should use older versions than what is available in the oldest still-supported available distributions, and the library should be only very infrequently and only if there are severe reasons that really require an update. Applications written against earlier versions of these should be guaranteed to still run with later versions of these.

  • ld-linux.so.2
  • ld-linux-x86-64.so.2
  • libanl.so.1
  • libBrokenLocale.so.1
  • libcidn.so.1
  • libcrypt.so.1
  • libc.so.6
  • libdl.so.2
  • libm.so.6
  • libmvec.so.1
  • libnsl.so.1
  • libnss_compat.so.2
  • libnss_db.so.2
  • libnss_dns.so.2
  • libnss_files.so.2
  • libnss_hesiod.so.2
  • libnss_nisplus.so.2
  • libnss_nis.so.2
  • libpthread.so.0
  • libresolv.so.2
  • librt.so.1
  • libthread_db.so.1
  • libutil.so.1

These files are all part of the GNU C Library which should never be bundled. List was generated from a fresh build of glibc 2.25.

References: https://www.gnu.org/software/libc/

But seemingly not all distribution can even seem to agree what is part of glibc and can be taken for granted, vs. what is not. As this example shows:

libnsl.so.1 is part of glibc (and hence can be assumed to be "just there") in Fedora 27 but not in Fedora 28!

libnsl.so.1 is part of glibc in Fedora 27:

podman run -it fedora:27 /bin/bash                                                                
[root@0ac2ed5ccbf5 /]# dnf provides libnsl.so.1
Fedora 27 - x86_64 - Updates                                                    992 kB/s |  24 MB     00:24    
Fedora 27 - x86_64                                                               19 MB/s |  58 MB     00:03    
Last metadata expiration check: 0:00:18 ago on Tue Jun  5 21:02:14 2018.
glibc-2.26-28.fc27.i686 : The GNU libc libraries
Repo        : updates
Matched from:
Provide    : libnsl.so.1

libnsl.so.1 is not part of glibc in Fedora 28 (and hence we cannot be sure that it will be installed by default on every Fedora 28 system):

glibc-2.26-15.fc27.i686 : The GNU libc libraries
Repo        : fedora
Matched from:
Provide    : libnsl.so.1
podman run -it fedora:28 /bin/bash
[root@9251485ee3e2 /]# dnf provides libnsl.so.1
Fedora 28 - x86_64 - Updates                                                     11 MB/s |  14 MB     00:01    
Fedora 28 - x86_64                                                               29 MB/s |  60 MB     00:02    
Last metadata expiration check: 0:00:04 ago on Tue Jun  5 21:02:31 2018.
libnsl-2.27-15.fc28.i686 : Legacy support library for NIS
Repo        : updates
Matched from:
Provide    : libnsl.so.1

libnsl-2.27-8.fc28.i686 : Legacy support library for NIS
Repo        : fedora
Matched from:
Provide    : libnsl.so.1

Distributions should agree on what is part of the GNU C Library and can be taken for granted, vs. what is not. Distributions should not make parts of what could be taken for granted optional (and, as a result, missing from e.g., Fedora 28 Atomic Workstation) all of a sudden.

libstdc++.so.6

Ideally this library would be frozen indefinitely. Why is it necessary to still keep adding symbols to it, making software compiled against later versions fail on earlier versions? If freezing this is not realistic, then at least build systems should use older versions than what is available in the oldest still-supported available distributions, and the library should be only very infrequently and only if there are severe reasons that really require an update. Applications written against earlier versions of it should be guaranteed to still run with later versions of this.

  • libstdc++.so.6

References: https://gcc.gnu.org/wiki/Libstdc++

OpenSSL

This should be abstracted in a way so that software compiled against OpenSSL would work against whatever version of OpenSSL comes with the system. Currently software compiled against a certain "letter" version (e.g., "f", "l") of OpenSSL will fail when another version of OpenSSL is available on the system. Applications written against earlier versions of it should be guaranteed to still run with later versions of this.

Take Firefox, for example. Mozilla distributes binaries that are supposed to run on most systems. If they cannot trust that whatever OpenSSL is in the operating system will continue to work, they are essentially forced to bundle a private copy, which means that a) there is more duplication, eating more system resources b) Firefox will not benefit from security updates in OpenSSL that the operating system might do. Same for Chrome and any other application that a) uses OpenSSL and b) is distributed outside of the Linux distribution.

Currently applications cannot rely on these:

  • libssl.so.1
  • libssl.so.1.0.0
  • libcrypto.so.1
  • libcrypto.so.1.0.0

They should be standardized across all distributions. Then applications could rely on these from the system rather than having to privately bundle their own copies.

References:

According to https://twitter.com/Det_Conan_Kudo/status/896508995044880384 OpenSSL, the upstream project "bumps the soname on every version release". According to the source, "Downstream, we fix this. This provides the added value of literally being able to upgrade OpenSSL in a relatively safe way without busting the world in the process".

Especially worrisome: upstream seems to be fully aware that going from 1.0 to 1.1 they broke applications compiled for 1.0, yet they don't seem to care: https://wiki.openssl.org/index.php/OpenSSL_1.1.0_Changes#No_longer_works. Newer versions of openSSL (or any library for that matter) should never break applications compiled against older versions - at least within the same major version of the library.

Looking at the Qt source code confirms that there are issues:

// (...) OpenSSL is a well-known case of binary-compatibility breakage. To // avoid such problems, many system integrators and Linux distributions change // the soname of the binary, letting the full version number be the soname. So // we'll find libssl.so.0.9.7, libssl.so.0.9.8, etc. in the system. For that // reason, we will search a few common paths (see findAllLibSsl() above) in hopes // we find one that works.

libgnutls

An application compiled against libgnutls26 should continue to work on systems that are using subsequent versions, such as libgnutls28 and libgnutls30. Possibly the versioning scheme of libgnutls would have to be changed for this, with new major versions (with ABI-breaking changes) coming out as rarely as possible (every 5-7 years or less frequently), and distributions should carry not only the recent version but also the version before to give sufficient time to developers to update.

Currently it looks like libgnutls26, libgnutls28 and libgnutls30 are all in use and not compatible.

Kerberos

The following libraries currently cannot be safely assumed to be part of every distribution:

  • libkrb5.so.26
  • libkrb5.so.3
  • libkrb5support.so.0

Distributions should agree on libkrb.so.5 being the name for Kerberos 5. Then applications could rely on these from the system rather than having to privately bundle their own copies.

Workaround: Patch libcurl so that it does not need Kerberos. https://github.com/AppImage/AppImages/blob/c9e6b6b365fcd97e2980c0844c478904c030a25a/recipes/meta/Recipe#L194

X11 libraries, Xlib, XCB

X11 libraries, Xlib, and the X protocol C-language Binding (XCB) should be handled by the distribution and they should provide a clearly defined interface for applications to develop against. Build systems should use older versions than what is available in the oldest still-supported available distributions. Applications written against earlier versions of these should be guaranteed to still run with later versions of these.

  • libX11.so.6 (uncertain if really all distributions bundle it)
  • libxcb.so.1 (uncertain if really all distributions bundle it)
  • libxcb-dri3.so.0 (uncertain if really all distributions bundle it)
  • libSM.so.6
  • libXp.so.6 (currently not available on Fedora default installations)
  • Many other libX...

OpenGL and Direct Rendering Manager (DRM)

OpenGL and the and Direct Rendering Manager (DRM) should be handled by the distribution and they should provide a clearly defined interface for applications to develop against. Build systems should use older versions than what is available in the oldest still-supported available distributions. Applications written against earlier versions of these should be guaranteed to still run with later versions of these.

  • libGL.so.1
  • libdrm.so.2

References:

ALSA and PulseAudio

Sound should be handled by the distribution and they should provide a clearly defined interface for applications to develop against. Build systems should use older versions than what is available in the oldest still-supported available distributions. Applications written against earlier versions of these should be guaranteed to still run with later versions of these.

  • libasound.so.2
  • PulseAudio libraries (tbd)

Example:

  • Mageia has /usr/lib/pulseaudio/libpulsecommon-10.0.so
  • App wants libpulsecommon-4.0.so

Gtk+ and Gdk

Gtk+ and Gdk should be handled by the distribution and they should provide a clearly defined interface for applications to develop against. Build systems should use older versions than what is available in the oldest still-supported available distributions. Applications written against earlier versions of these should be guaranteed to still run with later versions of these.

Ideally, there would be a defined library to develop against, e.g., libgtk.so.2 for Gtk+2 and libgtk.so.3 and applications would be compiled against those instead of a multitude of individual libraries that make up "Gtk+".

Example:

  • libgdk_pixbuf-2.0.so.0
  • Many more (are they defined somewhere)

Versioning scheme

GTK+ should follow semver.

According to https://blog.gtk.org/2016/09/01/versioning-and-long-term-stability-promise-in-gtk/, "Updates within long-term stable series will be ABI stable". This is good, but in order to minimize developer confusion about which GTK+ version can be relied on, only such long-term stable series should get major version numbers.

nkiesel nails it in https://lwn.net/Articles/691131/

However, there is the psychological problem of such applications always using the "previous" release. So innocent bystanders will see "GTK 5.0 released with all those new cool features", followed by "app FOO upgraded from GTK 3 to GTK 4". Therefore, GTK should make it very clear in it's press release that GTK 5.0 is just the first release candidate of GTK 5 and not intended to be used by external applications.

In this example, I would propose "GTK 5" to be called that only after it has a stable API and should be used by external applications. This would be more in line with the idea of a sane, dependable platform to build on. Everything that is still a moving target and has no API guarantees should not be worthy of a new major version number.

GLib

GLib should be handled by the distribution and they should provide a clearly defined interface for applications to develop against. Build systems should use older version than what is available in the oldest still-supported available distributions, and the library should be only very infrequently and only if there are severe reasons that really require an update. Applications written against earlier versions of this should be guaranteed to still run with later versions of this.

  • libglib-2.0.so.0

Fonts and font rendering

Fonts and font rendering libraries should be handled by the distribution and they should provide a clearly defined interface for applications to develop against. Build systems should use older versions than what is available in the oldest still-supported available distributions. Applications written against earlier versions of these should be guaranteed to still run with later versions of these.

  • libfontconfig.so.1

These would ideally never have to be linked against directly:

  • libpangoft2-1.0.so.0
  • libpangocairo-1.0.so.0
  • libpango-1.0.so.0
  • libpangomm-1.4.so.1

With increasing frequency, we are running into symbol lookup error: /usr/lib64/libpangoft2-1.0.so.0: undefined symbol: hb_buffer_set_cluster_level. Why?

HarfBuzz

Bundling HarfBuzz seems to cause symbol lookup error: /usr/lib64/libpangoft2-1.0.so.0: undefined symbol: hb_buffer_set_cluster_level (suggesting that the version of HarfBuzz that is being bundled is too old for the lib64/libpangoft2-1.0.so.0 on the system; which means that we need to bundle HarfBuzz then we also must bundle libpangoft2-1.0.so.0). Not bundling HarfBuzz seems to cause symbol lookup error: /lib64/libharfbuzz.so.0: undefined symbol: FT_Get_Var_Blend_Coordinates on Solus 3, indicating that the HarfBuzz from the system needs a newer FreeType than what we are bundling

HarfBuzz should be handled by the distribution and they should provide a clearly defined interface for applications to develop against. Build systems should use older versions than what is available in the oldest still-supported available distributions. Applications written against earlier versions of these should be guaranteed to still run with later versions of these.

References:

Image formats

Currently every distribution names these differently:

  • libtiff (Fedora and openSUSE lack libtiff.so.4)
  • libpng
  • libjpeg

They should be named the same in all distributions. Then applications could rely on these from the system rather than having to privately bundle their own copies.

NSS

Currently distributions ship no or incompatible versions of:

  • libnss3.so
  • libnssutil3.so
  • libsmime3.so
  • libssl3.so
  • libfreebl3.so
  • libfreeblpriv3.so
  • libnssckbi.so
  • libnssdbm3.so
  • libnsssysinit.so
  • libsoftokn3.so

They should be standardized across all distributions. Then applications could rely on these from the system rather than having to privately bundle their own copies.

References:

Also, some distributions (e.g., Debian/Ubuntu-type distributions) put these libraries into non-standard locations such as /usr/lib/x86_64-linux-gnu/nss/ (notice the extra last component of the path) which applications/libraries compiled on other distributions have no idea exists. This results e.g., in Qt applications built using the official Qt binary distribution from qt.io being unable to run on Debian/Ubuntu-type distributions.

They should be placed into the same locations (ideally, /usr/lib/x86_64-linux-gnu/) in all distributions. Then applications could rely on these from the system rather than having to privately bundle their own copies or search in "arbitrary" subdirectories.

Reference: https://github.com/probonopd/linuxdeployqt/issues/281#issuecomment-382983171

PCRE

libpcre should be handled by the distribution and they should provide a clearly defined interface for applications to develop against. Build systems should use older versions than what is available in the oldest still-supported available distributions. Applications written against earlier versions of this should be guaranteed to still run with later versions of this

  • openSUSE (as of 2017) lacks libpcre.so.3 but provides libpcre.so.1
  • Ubuntu (as of 2017) lacks libpcre.so.1 but provides libpcre.so.3

According to https://twitter.com/Det_Conan_Kudo/status/896508995044880384 PCRE sonames are wrong in debian, and hence, probably Ubuntu. According to the source, it cannot be fixed "because now it's the ABI contract of Debian." According to https://github.com/AppImage/AppImages/issues/83#issuecomment-322044692, "libpcre.so.3 is a historical Debian/Ubuntu screwup. Everywhere else it is libpcre.so.1."

Weak dependencies

Certain distribution functionality/libraries should be purely optional. Software compiled on distributions should never require these as hard dependencies, but should rather check whether these are available, and fall back gracefully if they are not there.

  • libselinux.so.1. Currently some software compiled on CentOS requires libselinux.so.1 even if SELinux is not used or wanted
  • libapparmor.so.1. According to https://twitter.com/Det_Conan_Kudo/status/896733648619270144, this equally applies to AppArmor. According to the source, apparently binaries on Ubuntu, SUSE and Debian link to it

Workaround: dummy libraries that resolves all of their symbols but just do nothing, e.g., https://github.com/AppImage/AppImages/issues/83#issuecomment-321966519

ELF handling in desktop environments

Desktop environments should handle ELF files with missing executable bits more gracefully. https://github.com/AppImage/AppImageKit/issues/374

SELinux

In the realm of security, distribution vendors are going different paths (Red Hat using NSA-originated SELinux and others using AppArmor or other technologies. Which is fine, as long as these systems don't spill over into each and every application, as is the case with libselinux.so.1, which gets linked into each application compiled on Red Hat based systems like CentOS, while it is not available on some other systems.

Security enhancements in the operating system should not require applications to link to libraries like libselinux.so.1.

Workaround: Bundle libselinux.so.1 with every application that needs it (but since it depends on libpcre.so.3 we need also bundle that because distributions can't be relied on to ship the correct version under the same name). Or, better, replace it by a dummy library that resolves all of its symbols but just does nothing, e.g., https://github.com/AppImage/AppImages/issues/83#issuecomment-321966519

FUSE

Although working out-of-the box in >95% of desktop distributions in the default installation, there are still 5% where this is not the case.

Distributions should ensure that FUSE is working out of the box and that all permissions are set correctly for regular users.

Compilers

Ideally, which version of a compiler was used to compile a binary should not determine which version of libraries the compiled binaries need. However, according to https://stackoverflow.com/a/42705277, the GCC version used determines the minimum version of libstdc++.so.6:

Sometimes you don't control the target machine (e.g. your library needs to run on a locked-down enterprise system). In such a case you will need to recompile your code using the version of GCC that corresponds to their GLIBCXX version. In that case, you can do the following:

  1. Look up the latest version of GLIBCXX supported by the target machine: strings /usr/lib/libstdc++.so.6 | grep GLIBC ... Say the version is 3.4.19.
  2. Use https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html to find the corresponding GCC version. In our case, this is [4.8.3, 4.9.0).

So, in practice, if you want your binary to run on say, all still-supported Ubuntu LTS releases (14.04 at the point of this writing), then you must not use a compiler newer than GCC 4.8.3.

Question: Can the compiler version be decoupled from the libraries used by the resulting binaries by using a cross-compiler? If so, how?

GStreamer

GStreamer is a library for constructing graphs of media-handling components. Ideally we could rely on this being "there" in the base operating system. However, different distributions insist on scattering around its files in the filesystem at different and crazy looking paths:

  • Debian, Ubuntu: Some parts (e.g., gst-plugin-scanner) in /usr/lib/x86_64-linux-gnu/gstreamer1.0/gstreamer-1.0, other parts (e.g., plugins) in /usr/lib/x86_64-linux-gnu/gstreamer1.0/
  • Red Hat, CentOS, Fedora: Some parts (e.g., gst-plugin-scanner) in /usr/libexec/gstreamer-1.0/, other parts (e.g., plugins) in /usr/lib64/gstreamer-1.0

Distributions should ensure that GStreamer is working out of the box and is installed in the same, upstream-determined, location on every distribution.

Xorg and Wayland

Some distributions are beginning to ship Wayland as the default display server in place of the older Xorg. Distributions should ensure that X11 applications run without manual intervention, e.g., by installing and configuring XWayland by default. Workaround: Install and configure XWayland manually, see https://github.com/AppImage/AppImageKit/wiki/Wayland

Individual distributions' platform promises

Some distributions make their own platform promises. Whether this can be part of the solution or adds to the complexity remains to be determined. Of course, this is insufficient. We need platform promises for "Desktop Linux".

Red Hat Enterprise Linux

https://access.redhat.com/articles/rhel-abi-compatibility describes the Red Hat Enterprise Linux platform guarantees.

Limit linking applications to the core libraries at compatibility level 1. The core libraries (see Appendix A) are intended to preserve binary compatibility across three consecutive major releases.

Provide compatibility libraries for applications that have been built with libraries that are not at the desired compatibility level, provided the bundled libraries themselves only use the interfaces provided by the core libraries.

Compatibility level 1: APIs and ABIs are stable across three major releases (starting with RHEL 7)

alsa-lib elfutils-libelf glibc glibc-utils gtk2 hesiod krb5-libs libgcc libgfortran libgomp libstdc++ libtbbmalloc_proxy.so libtbbmalloc.so libtbb.so libusb libvirt-client libxml2 libxslt mesa-libGL mesa-libGLU motif pam SDL

This means that we can trust that if you build against these libraries and bundle everything else privately, then your application is expected to work at least in RHEL 7, 8, and 9.

Note that gtk3 at this point is in "Compatibility level 2: APIs and ABIs are stable within one major release (RHEL 7)". Which means that a gtk3 application might break in RHEL 8. Which also means that application developers should not use it before it also enters compatibility level 1, or else they cannot be sure that their application will continue working in the foreseeable future.

Relevant forums

Where should we discuss this? Please add ideas here.

References

Clone this wiki locally