"crash" does not support memory structure in 5.17+ linux kernels

[xmkg]

"crash" fails with the more recent (5.17+) linux kernels with the following error:

UBUNTUTOOLS_UBUNTU_DDEBS_MIRROR= python3 hotkdump.py -c 1 -d /workspace/case-data/example-crash-dumps/lathiat/kdump-demos/jammy/x86_64/5.19.0-35-generic/dump.202303240831 
initializing hotkdump, SF#1, vmcore: /workspace/case-data/example-crash-dumps/lathiat/kdump-demos/jammy/x86_64/5.19.0-35-generic/dump.202303240831
kernel version: 5.19.0-35-generic
created /tmp/tmp_mga1ww5 temporary directory for the intermediary files
The .ddeb file linux-image-unsigned-5.19.0-35-generic-dbgsym_5.19.0-35.36~22.04.1_amd64.ddeb already exists, re-using it
Executing command: `dpkg -x linux-image-unsigned-5.19.0-35-generic-dbgsym_5.19.0-35.36~22.04.1_amd64.ddeb /tmp/tmp_mga1ww5/ddeb-root`
Loading `vmcore` file /workspace/case-data/example-crash-dumps/lathiat/kdump-demos/jammy/x86_64/5.19.0-35-generic/dump.202303240831 into `crash`, please wait..
Executing command: `/workspace/github/hotkdump/crash -i /tmp/tmp_mga1ww5/crash_commands -s /workspace/case-data/example-crash-dumps/lathiat/kdump-demos/jammy/x86_64/5.19.0-35-generic/dump.202303240831 /tmp/tmp_mga1ww5/ddeb-root/usr/lib/debug/boot/vmlinux-5.19.0-35-generic`

crash: invalid structure member offset: kmem_cache_s_num
       FILE: memory.c  LINE: 9619  FUNCTION: kmem_cache_init()

[/workspace/github/hotkdump/crash] error trace: 56184f3b969e => 56184f38d2f4 => 56184f45b11b => 56184f45b09c

It's fixed upstream(1), but it still hasn't landed in the Ubuntu series yet(2). In order to be able to use the "latest and the greatest" version of "crash", we may:

• build the crash ourselves, ship hotkdump as a snap bundled with the source-built crash
• build the crash ourselves, containerize the hotkdump,
• containerize the hotkdump, use a rolling distro as base
• ?? ... any other ideas

https://bugzilla.redhat.com/show_bug.cgi?id=2041702

[joalif]

This is a known problem with crash in ubuntu releases, most of the times is outdated.
Looking at the snapped version of crash, it is last updated on May 2017.
I haven't tested at all but I dare to say it is unmaintained.
My suggestion is to git clone crash and build it. This can be either done from hotkdump code, or during the snap build process if we decide to snap-ify hotkdump .

[xmkg]

Yes, it seems to be unmaintained. It's published by an ex-Canonical employee, so I'll ask whether he'll want to keep maintainership, or maybe grant us publish rights so we can update it too.

[xmkg]

I have thought about this and I think it makes sense for us to ship pre-built crash inside our snap, so we can have more control over which version is available for hotkdump. We can allow the user to specify their crash executable path to override it to provide flexibility.

[xmkg]

We now have a snap which also includes a bundled version of the crash (which is built from the upstream 8.0.1 version).