Debug Symbols

[felixge]

Problem

The postgres binaries shipped with the docker images in this repo seem to be stripped:

root@0d42c3d8cd06:/# postgres --version
postgres (PostgreSQL) 12.1 (Debian 12.1-1.pgdg100+1)
root@0d42c3d8cd06:/# file $(which postgres)
/usr/lib/postgresql/12/bin/postgres: ELF 64-bit LSB shared object, x86-64, version 1
(SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux
3.2.0, BuildID[sha1]=e775eb7dcaa0306329f0624056df0fa892d48f03, stripped
-----------------------------------------------------------------^

This makes it difficult to use tools such as gdb or perf to debug problems as the resulting stack traces will often be useless.

See my tweet here for a practical example of what can be done when symbols are present.

Proposal

It'd be great if this image would start to ship with debug symbols by default, e.g. by compiling with ./configure CFLAGS="-fno-omit-frame-pointer -ggdb".

If not, it'd be great to document how to derive an image that includes symbols. My first attempts at doing so below have failed. I still get the same output for file $(which postgres) as above, and readelf -s $(which postgres) | wc -l gives me the same ~9k symbols contained in this image, rather than the ~30k symbols usually available for a non-stripped postgres binary.

FROM postgres
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y \
  debian-goodies \
  binutils
RUN apt-get install -y $(find-dbgsym-packages $(which postgres))

I suspect the image above doesn't work because postgres is pulled from apt.postgresql.org, but find-dbgsym-packages is not smart enough to search that repo for debug package, if those even exist.

I'd be happy to try to help with creating a PR to improve the debugging/symbol situation once the direction is clear.

From my PoV the only downside to including symbols is image size, so maybe they shouldn't go into the alpine flavor of this image.

[tianon]

I did some digging on find-dbgsym-packages to try and figure out why it didn't recognize the -dbgsym packages that are published in the apt.postgresql.org repo we use, and I found that it does, but it fails to find a lot of symbols for dependencies in Debian proper because the debian-debug repo isn't added (see https://manpages.debian.org/buster/debian-goodies/find-dbgsym-packages.1.en.html):

$ docker run -it --rm postgres bash
root@ed30d6289e53:/# apt-get update
...
root@ed30d6289e53:/# apt-get install debian-goodies
...
root@ed30d6289e53:/# which postgres
/usr/lib/postgresql/12/bin/postgres
root@ed30d6289e53:/# find-dbgsym-packages /usr/lib/postgresql/12/bin/postgres
W: No dbg package for source 'cyrus-sasl2'
W: Cannot find debug package for /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (e5a6c1b52e9b18f9b6b1b54bc006507409683ce7)
W: No dbg package for source 'nettle'
W: Cannot find debug package for /usr/lib/x86_64-linux-gnu/libnettle.so.6 (696c145020fc52f49a604b409e80c0f604514cbe)
...
W: Cannot find debug package for /lib/x86_64-linux-gnu/libsystemd.so.0 (6793bb7adf4f0ec3b4e32e8fa455f8f404670c9a)
libc6-dbg libffi6-dbg libgcc1-dbg libkrb5-dbg libstdc++6-8-dbg libxml2-dbg postgresql-12-dbgsym zlib1g-dbg

Worth noting, that last line does include postgresql-12-dbgsym (which is the appropriate package from apt.postgresql.org, so at least that bit's working).

I think https://wiki.debian.org/HowToGetABacktrace#Installing_the_debugging_symbols is a better reference than the find-dbgsym-packages man page because it notes what's necessary for a particular release and how to make sure it includes at least some of the release updates (from the proposed-updates repo), which will otherwise not be included properly.

After adding those, I get a much longer list (libaudit1-dbgsym libc6-dbg libcap-ng0-dbgsym libcom-err2-dbgsym libffi6-dbg libgcc1-dbg libgcrypt20-dbgsym libgmp10-dbgsym libgnutls30-dbgsym libgpg-error0-dbgsym libhogweed4-dbgsym libicu63-dbgsym libidn2-0-dbgsym libkeyutils1-dbgsym libkrb5-dbg libldap-2.4-2-dbgsym liblz4-1-dbgsym liblzma5-dbgsym libnettle6-dbgsym libp11-kit0-dbgsym libpam0g-dbgsym libsasl2-2-dbgsym libssl1.1-dbgsym libstdc++6-8-dbg libsystemd0-dbgsym libtasn1-6-dbgsym libunistring2-dbgsym libxml2-dbg postgresql-12-dbgsym zlib1g-dbg) and no warnings/errors, but several are then not installable due to version mismatches, which would be due to Debian security updates (which AFAIK, still do not publish debug symbol packages anywhere):

The following packages have unmet dependencies:
 libcom-err2-dbgsym : Depends: libcom-err2 (= 1.44.5-1+deb10u3) but 1.44.5-1+deb10u2 is to be installed
 libgnutls30-dbgsym : Depends: libgnutls30 (= 3.6.7-4+deb10u2) but 3.6.7-4 is to be installed
 libsystemd0-dbgsym : Depends: libsystemd0 (= 241-7~deb10u3) but 241-7~deb10u2 is to be installed
E: Unable to correct problems, you have held broken packages.

So, I think something like this is probably a better Dockerfile for doing this in a semi-automated way:

FROM postgres:12

RUN set -eux; \
	release="$(awk '/^deb / { print $3; exit }' /etc/apt/sources.list)"; \
	{ \
		echo "deb http://deb.debian.org/debian-debug $release-debug main"; \
		echo "deb http://deb.debian.org/debian-debug $release-proposed-updates-debug main"; \
	} | tee /etc/apt/sources.list.d/debug.list; \
	apt-get update; \
	apt-get install -y --no-install-recommends \
		debian-goodies \
# "Recommends" of debian-goodies we need for running "find-dbgsym-packages"
		libipc-system-simple-perl \
		libfile-which-perl \
		elfutils \
		dctrl-tools \
	; \
	postgres="$(command -v postgres)"; \
	dbgsym="$(find-dbgsym-packages "$postgres")"; \
	apt-get install -y --no-install-recommends $dbgsym; \
	rm -rf /var/lib/apt/lists/*

This will need some adjustment to overcome the security updates issue (either forcing APT to install the older versions that don't have the security fixes somehow or skipping installing the debugging symbols packages for those packages).

As to including these packages in the Debian-based images by default, IMO that's not really tenable: the postgresql-12-dbgsym package alone is ~20MB compressed (~22MB uncompressed), which isn't even accounting for any of these others. 😅

[otbutz]

A dedicated debug image would really be appreciated.

[felixge]

@tianon thank you so much for your research on this!

What about a dedicated debug image as suggested by @otbutz ? Would you consider to include something like this in this repo?

FWIW, I'd still prefer including symbols by default, even if it blows up the image size, and providing non-symbol images as the alternative. But I don't feel too strongly about it, especially if the README tells people about the tradeoffs that are available.

[otbutz]

Debugging the database itself is not what most users of this image are trying to achieve. I think a separate image and thus an opt-in approach is more appropriate.

[felixge]

Debugging the database itself is not what most users of this image are trying to achieve. I think a separate image and thus an opt-in approach is more appropriate.

Well, arguably almost nobody wants to debug their database ... but when your production DB has an issue, you might have a come-to-jesus when it comes to image size vs debuggability.

You'll be under time pressure, and restarting your db container on a new debug-image while praying that it won't corrupt your data due to incompatible libc collations or similar may not seem like an attractive option ...

That being said, as somebody who already has those scars and stories to tell, I'll end up with a better setup either way. I'm just trying to save others from future PTSD inducing adventures ; )

Anyway, I'd still to love to see a debug image to become part of this distribution, even if optional. "Doing the right thing" (TM) should be the default, but the second best option is to not make it excessively difficult for those who care.

[felixge]

but several are then not installable due to version mismatches, which would be due to Debian security updates (which AFAIK, still do not publish debug symbol packages anywhere):

FWIW, having full symbols for postgres itself and perhaps libc is 80% of the battle IMO. The other libs would be nice, but if it's super hard to get correct symbols for them, it might be reasonable to consider them out-of-scope for now.