Heads panic (unable to mount root fs) before menu shows up

[marmarek]

Please identify some basic details to help process the report

A. Provide Hardware Details

1. What board are you using? (Choose from the list of boards here)

2. Does your computer have a dGPU or is it iGPU-only?

   • dGPU (Distinct GPU other then internal GPU)
   • iGPU-only (Internal GPU, normally Intel GPU)

3. Who installed Heads on this computer?

   • Insurgo (Issues to be reported at https://github.com/linuxboot/heads/issues)
   • Nitrokey (Issues to be reported at https://github.com/Nitrokey/heads/issues)
   • Purism (Issues to be reported at https://source.puri.sm/firmware/pureboot/-/issues)
   • Novacustom (Issues to be reported at https://github.com/Dasharo/dasharo-issues)
   • HardnenedVault (Issues to be reported at https://github.com/hardenedvault/vaultboot/issues)
   • Other provider
   • Self-installed

4. What PGP key is being used?

   • Librem Key (Nitrokey Pro 2 rebranded)
   • Nitrokey Pro
   • Nitrokey Pro 2
   • Nitrokey 3 NFC
   • Nitrokey 3 NFC Mini
   • Nitrokey Storage
   • Nitrokey Storage 2
   • Yubikey
   • Other

5. Are you using the PGP key to provide HOTP verification?

   • Yes
   • No
   • I don't know

B. Identify how the board was flashed

1. Is this problem related to updating heads or flashing it for the first time?

   • First-time flash
   • Updating heads

2. If the problem is related to an update, how did you attempt to apply the update?

   • Using the Heads menus
   • Flashrom via the Recovery Shell
   • External flashing

3. How was Heads initially flashed?

   • External flashing
   • Internal-only / 1vyprep+1vyrain / skulls
   • Don't know

4. Was the board flashed with a maximized or non-maximized/legacy rom?

   • Maximized
   • Non-maximized / legacy
   • I don't know

5. If Heads was externally flashed, was IFD unlocked?

   • Yes
   • No
   • Don't know

C. Identify the rom related to this bug report

1. Did you download or build the rom at issue in this bug report?

   • I downloaded it
   • I built it

2. If you downloaded your rom, where did you get it from?

   • Heads CircleCi
   • Purism
   • Nitrokey
   • Dasharo DTS (Novacustom)
   • Somewhere else (please identify)

heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1

3. If you built your rom, which repository:branch did you use?

   • Heads:Master
   • Other (please identify)

4. What version of coreboot did you use in building?
   { You can find this information from github commit ID or once flashed, by giving the complete version from Sytem Information under Options --> menu}

5. In building the rom, where did you get the blobs?

   • No blobs required
   • Provided by the company that installed Heads on the device
   • Extracted from a backup rom taken from this device
   • Extracted from another backup rom taken from another device (please identify the board model)
   • Extracted from the online bios using the automated tools provided in Heads
   • I don't know

Please describe the problem

Describe the bug

After one of many reinstalls (this is a CI system), Heads doesn't start anymore. Could be Heads initramfs broken?

To Reproduce
Steps to reproduce the behavior:

1. Reinstall Qubes OS many times, including resetting TPM, regenerating HOTP keys etc
2. ...
3. See error

Expected behavior

No panic

Screenshots

A screen with the message: Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Additional context

I can do a flash dump with external programmer (if that would be helpful), but only in a few days.

[tlaurion]

@marmarek a full dump could help, but it looks like something spi got corrupted during internal flash. The error means that initramfs could not be found, after coreboot extracting payload in ram and trying to jump to its initramfs. Seems like spi content got corrupted somehow. Weird anyhow, since internal flashing either used flashrom/flashprog so it it should have failed if incapable of writing a block and should have failed verification step from prior rom version, leading you to recovery shell in case an erase/write operation on spi failed from prior rom flashing spi to new version.

Can you provide source rom version as well if you kept trace? cd683b1 is pretty old (2 months old)? On phone, will get back to this on computer later and inspects diffs, but as said previously flash op should have failed verification step, where rollback to previous rom could have mitigated leaving you in recovery shell in case verifying flashed rom content failing, and leaving you to old rom forever in case spi is worn out for whatever reason.

I'm mostly interested into external spi backup dump content, as well as if your test machine can boot from an externally flashed roms from master, or better, feature freeze PR branch.

Next step is to diagnose if spi chips are worn out. And if not, trying to explain what happened to result in corrupted spi flash otherwise.

[marmarek]

Can you provide source rom version as well if you kept trace? cd683b1 is pretty old (2 months old)? On phone, will get back to this.

From before the update? I don't have it. But it did worked for a while after the update. Last working date is 2024-12-14.

[tlaurion]

@marmarek a full dump could help, but it looks like something spi got corrupted during internal flash. The error means that initramfs could not be found, after coreboot extracting payload in ram and trying to jump to its initramfs. Seems like spi content got corrupted somehow. Weird anyhow, since internal flashing either used flashrom/flashprog so it it should have failed if incapable of writing a block and should have failed verification step from prior rom version, leading you to recovery shell in case an erase/write operation on spi failed from prior rom flashing spi to new version.

Can you provide source rom version as well if you kept trace? cd683b1 is pretty old (2 months old)? On phone, will get back to this on computer later and inspects diffs, but as said previously flash op should have failed verification step, where rollback to previous rom could have mitigated leaving you in recovery shell in case verifying flashed rom content failing, and leaving you to old rom forever in case spi is worn out for whatever reason.

I'm mostly interested into external spi backup dump content, as well as if your test machine can boot from an externally flashed roms from master, or better, feature freeze PR branch.

Next step is to diagnose if spi chips are worn out. And if not, trying to explain what happened to result in corrupted spi flash otherwise.

@marmarek ping because edited post on phone.

[marmarek]

External dump of broken state:
bad-rom.zip

I've dumped it twice and got the same result, so at least it is not very worn out flash...

[marmarek]

better, feature freeze PR branch.

which one specifically?

[tlaurion]

better, feature freeze PR branch.

which one specifically?

#1875

[tlaurion]

External dump of broken state: bad-rom.zip

I've dumped it twice and got the same result, so at least it is not very worn out flash...

@marmarek before investing time into inspecting provided rom, flashing a new rom resulted in platform to boot?
Being able to dump rom content from flash doesn't mean host is actually able to get access to same rom content. legs of spi to motherboard could be at fault here unfortunately. Another user reported symbols missing, but next boot was successful.

Hopefully flashing a new rom externally will permit host to boot. If it does, I will take time inspecting rom dump and rebuilt that commit for comparison.

[marmarek]

Yes, flashing new rom externally fixed it. But still interesting why it happened. When flash is written? Does "reset TPM" procedure involve some flash write, or only /boot modification?

[tlaurion]

Yes, flashing new rom externally fixed it. But still interesting why it happened. When flash is written? Does "reset TPM" procedure involve some flash write, or only /boot modification?

That's what is puzzling. What writes in firmware:

• memory (re)training (but unreleated to payload)
• factory reset for keyring + trustdb generated by importing pubkey (tpm reset writes to tpm nvram), also unrelated to payload
• config setting overrides from GUI (config.user written in CBFS), also unrelated to payload

Otherwise everything else is hashes or signatures from runtime generated artifacts resulting in digest, measured with sha256 and verified against kexec.sig under /boot.

For root (initramfs) to fail to decompress and be available to run init script from memory unpacked runtime env, something needs to corrupt coreboot's payload in cbfs. Unfortunately, with this corner case situation, nothing is in place to provide logs that can be properly extracted in absence of serial output or again cbmem logs written in spi (which we don't turn on in coreboot config for spi wear and slowing down boot). If initramfs rootfs cannot be mounted, nothing Heads could be run; no trace can be exported for debugging.

Attempting to rebuild cd683b1 now and will try to figure out what got corrupted. But as said, hypothesis is that something prevented host to read it properly. Observation paradox: connecting external flash programmer might have fixed the issue to be investigated (booting spi available cd683b1 + config.user + pubring + trustdb) might have worked without flashing new rom). Best hypothesis is flaky spi connection at this point, but investigating top+bottom differences might shed some lights on what changed, not what caused changes. Will see.

[marmarek]

I think the broken version was this binary:
heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.zip

[tlaurion]

Well, diffoscope doesn't help without way more time in investigating.
Prepared v0.2.5-test docker image containing full diffoscope, which when tested locally (use docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:v0.2.5-test to use it/jump into it):

cat bad-bottom.rom bad-top.rom > bad-12mb.rom #heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.rom built locally by git checkout cd683b1, building clean with docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:v0.2.4 make BOARD=x230-hotp-maximized_usb-kb
docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:v0.2.5-test
bash-5.2# diffoscope bad-12mb.rom build/x86/x230-hotp-maximized_usb-kb/heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.rom --html-dir 12mb/ --debug
2025-01-07 17:35:41 D: diffoscope.main: Starting diffoscope 267
2025-01-07 17:35:41 D: diffoscope.presenters.formats: Will generate the following presenter formats: html_directory
2025-01-07 17:35:41 D: diffoscope.environ: Normalising locale, timezone, etc. PATH is /nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/bin:/nix/store/35yzzhnm6g844lb2fib3l4phgpafrhr0-diffoscope-267/bin:/nix/store/wxivhpy40zpriv3rkwkzr62s1kagmr8v-acl-2.3.2-bin/bin:/nix/store/4f5l3xfv9286ylzyf2ysqw7isb1bwjg6-binutils-2.41/bin:/nix/store/mxcq77rlan82dzpv3cgj0fh6qvv8ncil-bzip2-1.0.8-bin/bin:/nix/store/c15864g1ibs52zr11zf2gfc45mh2yjcl-cdrkit-1.1.11/bin:/nix/store/hrqj70idz1x367v2vhdzwcmnql2vw6y8-colordiff-1.0.21/bin:/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/kzdb7vjm8s7nad3mkynyl06k4vjkykdg-cpio-2.15/bin:/nix/store/jm14ghwq245i0wliygx57hzhyl7fxa3a-db-5.3.28-bin/bin:/nix/store/i34mknsjgrfyy71k2h79gda0bvagzc2j-diffutils-3.10/bin:/nix/store/s9iapa6l26glddd87id7nin4n17bddaj-dtc-1.7.0/bin:/nix/store/fskqxhk5kx3wliw4lm2cp2ccbcx448z6-e2fsprogs-1.47.0-bin/bin:/nix/store/cz28fainhxq78r4l4wgr140izc9hxsxp-file-5.45/bin:/nix/store/jjcsr5gs4qanf7ln5c6wgcq4sn75a978-findutils-4.9.0/bin:/nix/store/9wc153f41kbamp3bqb0nvd8rd7xffv9p-fontforge-20230101/bin:/nix/store/z14w63wj0gk97hr06l76d4s723ynhr2k-gettext-0.21.1/bin:/nix/store/rik7p68cq7yzlj5pmfpf4yv6jnrpvlgf-gnutar-1.35/bin:/nix/store/j5chw7v1x3vlmf3wmdpdb5gwh9hl0b80-gzip-1.13/bin:/nix/store/cc3nd6krrwizlzppc45n8v77jjj2c3zh-html2text-1.3.2a/bin:/nix/store/93lih8knkk2wb760ixsp8za08r57hbhh-libarchive-3.7.4/bin:/nix/store/7xjwqvb6argz4qn5j55x4hch17mqykp7-libxmlb-0.3.19/bin:/nix/store/wfnfsglf5zw13slpplsg3nqqlazri1vk-lz4-1.9.4-bin/bin:/nix/store/gjw4qwp03mdkv2qvwqf86h0rlfb1dliq-lzip-1.24.1/bin:/nix/store/qannz09m66qpcy3ny1f4nkl4ql0g71ks-openssl-3.0.13-bin/bin:/nix/store/c1rrlgm2js6i5b7q4hklrqfv5yb31lnb-pgpdump-0.36/bin:/nix/store/lmjakskd949i20dmvfwgz289rzy03y6i-sng-1.1.0/bin:/nix/store/rgg0dav9j9wyxwbzvzlx019v3szdk5sy-sqlite-3.45.3-bin/bin:/nix/store/i6mwg9kmnl9rz50h4cq1a7vwxviv5990-squashfs-4.6.1/bin:/nix/store/c9ankq44wixyl11szsg2g6c2rs008ny8-unzip-6.0/bin:/nix/store/8ihyhjpkngfc35nlfpxqjdkzib50jngq-xxd-vim-9.1.0377/bin:/nix/store/6i4xxaa812vsbli9jkq4mksdddrk27lw-xz-5.4.6-bin/bin:/nix/store/ig56yz6lgkmlwjvkrdz51pp94212kwyf-zip-3.0/bin:/nix/store/hxhdhc0zddbhmqvv99nldq2gfmy9983x-zstd-1.5.6-bin/bin:/nix/store/abnch2ab1jfh3kvlrf1fshnx4i2p7kdf-python3.11-argcomplete-3.3.0/bin:/nix/store/pj2ylnxmzwbajq5cyas8ca7xqxcbmc3p-python3.11-chardet-5.2.0/bin:/nix/store/ln2ll458vb7rzn0nh3mnyahzfmb6h5p4-python3.11-jsbeautifier-1.15.1/bin:/nix/store/fi2cgccibl4g04ffc6642sazpnwjspvl-python3.11-editorconfig-0.12.4/bin:/nix/store/94v50jmnjj69pymxiwkzl56nh1abk0ag-python3.11-jsondiff-2.0.0/bin:/nix/store/x6k54msbs5ci6jgdzk9fhi8z5d9xn50x-rpm-4.18.1/bin:/nix/store/nqhf1w10rl8r7czrj6ibi832kk32p8rz-abootimg-0.6/bin:/nix/store/bf3pzqg6l9qqmhkjjz9r5l549k58m64k-apksigcopier-1.1.1/bin:/nix/store/ivwzg8aykn8603nm1xwvv2ajid5nbch5-apksigner-34.0.5-unstable-2024-03-06/bin:/nix/store/p6kkc8s37ybyw6lf7ii0v7lrg2z62xci-apktool-2.9.3/bin:/nix/store/97nfzjijjrmpjq0bsv4s5x54hmmvb703-cbfstool-24.05/bin:/nix/store/r9ijdg4f5dbi27ni53ms73j3cii3knd7-colord-1.4.6/bin:/nix/store/64cix55ynjmp1j5m3i1h9z2m9mrzdip6-enjarify-1.0.3/bin:/nix/store/3zc5jbvqzrn8zmva4fx5p0nh4yy03wk4-ffmpeg-6.1.1-bin/bin:/nix/store/rn8irkhixy7z0vvs9q3x8vwh5ghcgwqj-fpc-3.2.2/bin:/nix/store/0rnmmwnlrfssbz6s4f96jc9mi1w1ggj5-ghc-9.6.5/bin:/nix/store/r4qnpfh6s2s6d0kw938syll5pp3gy76c-ghostscript-with-X-10.02.1/bin:/nix/store/j7jpz6vabnfjx11nm6cm988chh2835jx-giflib-5.2.2/bin:/nix/store/h7qn6nl04qplrsnqlgwxp72rsg4cil9r-gnupg-2.4.5/bin:/nix/store/1izdxwml9nsifjrh53rdfiglhjmrnx2s-imagemagick-7.1.1-32/bin:/nix/store/ya0wj4dz2778xbhdayzphvbz9c04gryr-openjdk-8u362-ga/bin:/nix/store/jh070x4pmiik3s57h6kw7amr8xjlkhwb-libcaca-0.99.beta20-bin/bin:/nix/store/xhkqzjmzpsrzyjfpvwgpaqb5gvqkhmyv-llvm-17.0.6/bin:/nix/store/i3y3yxxhs1jrml5pvxzdwlcfnr57fw8z-mono-6.12.0.182/bin:/nix/store/2dd9wc8msl3v38972xqmwsw3d50yans5-ocaml-5.1.1/bin:/nix/store/c2ijq9crgaiypanq8751qpzgchwrcl6s-odt2txt-0.5/bin:/nix/store/0g1s8yd0biawp32fl3i7kdbi219jx6aq-openssh-9.7p1/bin:/nix/store/7bx3cvg5r7xj1rifhfsh1bhjqd8h3jdi-pdftk-3.3.3/bin:/nix/store/1f2vbia1rg1rh5cs0ii49v3hln9i36rv-poppler-utils-24.02.0/bin:/nix/store/40f94zajxkd9r4bvj9k48k461anp6lyh-procyon-0.6.0/bin:/nix/store/ck1jf678yi3x1n4idmr4q1c0bdj2s5x5-qemu-8.2.4/bin:/nix/store/ijmjjnhnyzyfp6f63v3xhll27pf86899-R-4.3.3/bin:/nix/store/p16nknin6274dnb8f9lsaakb9wzi40sz-tcpdump-4.99.4/bin:/nix/store/k19qwa8pln1bwiahqa0ngf3w12wlsdy2-uboot-tools-only_defconfig-2024.04/bin:/nix/store/bwxs6pcbibz3sdx6id2hsnb0jvp1wzds-wabt-1.0.35/bin:/nix/store/drk5ym85v3qdrp4nklgxn6kdvwgnkw9b-xmlbeans-5.1.1-20220819/bin:/nix/store/gz20ka85pagp3fq5vqp4x8ly6kwz8k8r-python3.11-androguard-3.4.0a1/bin:/nix/store/h9aqfjqx95vangljkimfpgyw1nw8fqsw-python3.11-future-1.0.0/bin:/nix/store/vkmcav3nrdg8d2mw54iacsmkqk3x5kr3-python3.11-ipython-8.24.0/bin:/nix/store/cid5kxmrjdgikliwbhry3wq6j0117595-python3.11-pygments-2.17.2/bin:/nix/store/ylvsb39l0wq5q6xdphrq35an6dxqhjw9-python3.11-numpy-1.26.4/bin:/nix/store/b4ykqkilhvvk5hq5ywk0yqkvwizg0jf2-python3.11-fonttools-4.51.0/bin:/nix/store/afrh3b5nkyp2vlgavf9x313hwsbglj7i-python3.11-binwalk-2.3.4/bin:/nix/store/j6yz1h0i58a3i54szjv40mdz4gssb391-p7zip-17.05/bin:/nix/store/22q379b6f18qk05f2dnw3hj90ag5j7mn-cabextract-1.11/bin:/nix/store/64kal2vckiwjdg6bqrvvm2axwx9infl4-cramfsprogs-1.1/bin:/nix/store/dwxxkdn6znbi1w7j48gkjf4c48fhdyfa-cramfsswap-1.4.2/bin:/nix/store/qfcvsm1vw7a9qhy9jvz1qjdkv4j9jcr5-sasquatch-4.4/bin:/nix/store/9ibi757d3ldh4hj5l0vaydng4ffj6drk-libguestfs-1.50.1/bin:/nix/store/2g1cjsajwqpma7v5wzfyh521n4p9ic0r-python3.11-pdfminer-six-20231228/bin:/nix/store/jrr6l56xssk4szz6xxk9mxhk8pxwghhg-python3.11-charset-normalizer-3.3.2/bin:/nix/store/lgfcx6wqcg7jfqvndw5kbc7kgd9m0i7j-oggvideotools-0.9.1/bin:/nix/store/bh92a52c07iiarn1fcg2j035flc1hq5b-gnumeric-1.12.57/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-01-07 17:35:41 D: diffoscope.main: Starting comparison
2025-01-07 17:35:41 D: diffoscope.comparators.decompile: radare2 not found, disabling decompiler
2025-01-07 17:35:41 D: diffoscope.comparators: Loaded 92 comparator classes
2025-01-07 17:35:41 D: diffoscope.comparators.cbfs: looking for header at offset: 3022c
2025-01-07 17:35:41 D: diffoscope.comparators.utils.specialize: Using cbfs.CbfsFile for bad-12mb.rom
2025-01-07 17:35:41 D: diffoscope.comparators.cbfs: looking for header at offset: 3022c
2025-01-07 17:35:41 D: diffoscope.comparators.utils.specialize: Using cbfs.CbfsFile for build/x86/x230-hotp-maximized_usb-kb/heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.rom
2025-01-07 17:35:41 D: diffoscope.comparators.utils.compare: Comparing bad-12mb.rom (CbfsFile) and build/x86/x230-hotp-maximized_usb-kb/heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.rom (CbfsFile)
2025-01-07 17:35:41 D: diffoscope.comparators.utils.file: has_same_content(bad-12mb.rom, build/x86/x230-hotp-maximized_usb-kb/heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.rom)
2025-01-07 17:35:41 D: diffoscope.comparators.utils.file: Executing: cmp -s bad-12mb.rom build/x86/x230-hotp-maximized_usb-kb/heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.rom
2025-01-07 17:35:41 D: diffoscope.comparators.utils.command: Executing cbfstool {} print
2025-01-07 17:35:41 D: diffoscope.comparators.utils.command: Executing cbfstool {} print
2025-01-07 17:35:41 D: diffoscope.tempfiles: Created top-level temporary directory: /tmp/diffoscope_vtq274q8_12mb (free space: 48.31 GiB)
2025-01-07 17:35:41 D: diffoscope.diff: Running diff -a -U7 /tmp/diffoscope_vtq274q8_12mb/tmp59u8srme/fifo1 /tmp/diffoscope_vtq274q8_12mb/tmp59u8srme/fifo2
2025-01-07 17:35:41 D: diffoscope.diff: diff -a -U7 /tmp/diffoscope_vtq274q8_12mb/tmp59u8srme/fifo1 /tmp/diffoscope_vtq274q8_12mb/tmp59u8srme/fifo2: returncode 1, parsed True
2025-01-07 17:35:41 D: diffoscope.comparators.utils.file: Instantiating a cbfs.CbfsContainer for bad-12mb.rom
2025-01-07 17:35:41 D: diffoscope.comparators.utils.file: Instantiating a cbfs.CbfsContainer for build/x86/x230-hotp-maximized_usb-kb/heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.rom
2025-01-07 17:35:41 D: diffoscope.comparators.utils.command: Calling external command: cbfstool bad-12mb.rom print -k
2025-01-07 17:35:41 D: diffoscope.comparators.utils.archive: Unpacking cbfs_master_header from bad-12mb.rom to /tmp/diffoscope_vtq274q8_12mb/tmp9ajxhpw5_CbfsContainer
2025-01-07 17:35:41 D: diffoscope.comparators.cbfs: cbfstool extract cbfs_master_header to /tmp/diffoscope_vtq274q8_12mb/tmp9ajxhpw5_CbfsContainer/cbfs_master_header
2025-01-07 17:35:41 D: diffoscope.comparators.utils.command: Calling external command: cbfstool bad-12mb.rom extract -n cbfs_master_header -f /tmp/diffoscope_vtq274q8_12mb/tmp9ajxhpw5_CbfsContainer/cbfs_master_header
2025-01-07 17:35:41 D: diffoscope.comparators.utils.archive: Unpacking fallback/romstage from bad-12mb.rom to /tmp/diffoscope_vtq274q8_12mb/tmp3t1z6h0h_CbfsContainer
2025-01-07 17:35:41 D: diffoscope.comparators.cbfs: cbfstool extract fallback/romstage to /tmp/diffoscope_vtq274q8_12mb/tmp3t1z6h0h_CbfsContainer/romstage
2025-01-07 17:35:41 D: diffoscope.comparators.utils.command: Calling external command: cbfstool bad-12mb.rom extract -n fallback/romstage -f /tmp/diffoscope_vtq274q8_12mb/tmp3t1z6h0h_CbfsContainer/romstage
2025-01-07 17:35:41 D: diffoscope.comparators.utils.command: Executing xxd {}
2025-01-07 17:35:42 D: diffoscope.comparators.utils.command: Executing xxd {}
2025-01-07 17:35:42 D: diffoscope.diff: Running diff -a -U7 /tmp/diffoscope_vtq274q8_12mb/tmpsux__srz/fifo1 /tmp/diffoscope_vtq274q8_12mb/tmpsux__srz/fifo2
2025-01-07 17:35:49 D: diffoscope.diff: diff -a -U7 /tmp/diffoscope_vtq274q8_12mb/tmpsux__srz/fifo1 /tmp/diffoscope_vtq274q8_12mb/tmpsux__srz/fifo2: returncode 1, parsed True
2025-01-07 17:35:49 D: diffoscope.presenters.formats: Generating 'html_directory' output at '12mb/'
2025-01-07 17:35:49 D: diffoscope.presenters.html.html: new unified-diff subpage, parent page went over 128 lines
2025-01-07 17:35:50 D: diffoscope.presenters.formats: Generated 'html_directory' output at '12mb/' (size: n/a)
2025-01-07 17:35:50 D: root: # Profiling output for: /nix/store/35yzzhnm6g844lb2fib3l4phgpafrhr0-diffoscope-267/bin/diffoscope bad-12mb.rom build/x86/x230-hotp-maximized_usb-kb/heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.rom --html-dir 12mb/ --debug
2025-01-07 17:35:50 D: root: ## close_archive (total time: 0.000s)
2025-01-07 17:35:50 D: root:        0.000s      2 calls    diffoscope.comparators.cbfs.CbfsContainer
2025-01-07 17:35:50 D: root: ## command (total time: 16.037s)
2025-01-07 17:35:50 D: root:        9.435s     10 calls    xxd
2025-01-07 17:35:50 D: root:        6.583s      2 calls    diff
2025-01-07 17:35:50 D: root:        0.015s     13 calls    cbfstool
2025-01-07 17:35:50 D: root:        0.002s      1 call     cmp
2025-01-07 17:35:50 D: root:        0.002s      1 call     cmp (external)
2025-01-07 17:35:50 D: root:        0.000s      1 call     cmp (internal)
2025-01-07 17:35:50 D: root: ## compare_files (cumulative) (total time: 7.541s)
2025-01-07 17:35:50 D: root:        7.541s      1 call     abc.CbfsFile
2025-01-07 17:35:50 D: root: ## container_extract (total time: 0.014s)
2025-01-07 17:35:50 D: root:        0.014s      1 call     diffoscope.comparators.cbfs.CbfsContainer
2025-01-07 17:35:50 D: root: ## diff (total time: 0.031s)
2025-01-07 17:35:50 D: root:        0.031s     12 calls    linediff
2025-01-07 17:35:50 D: root: ## has_same_content_as (total time: 0.002s)
2025-01-07 17:35:50 D: root:        0.002s      1 call     abc.CbfsFile
2025-01-07 17:35:50 D: root: ## main (total time: 8.897s)
2025-01-07 17:35:50 D: root:        8.897s      2 calls    outputs
2025-01-07 17:35:50 D: root:        0.000s      1 call     cleanup
2025-01-07 17:35:50 D: root: ## open_archive (total time: 0.000s)
2025-01-07 17:35:50 D: root:        0.000s      2 calls    diffoscope.comparators.cbfs.CbfsContainer
2025-01-07 17:35:50 D: root: ## output (total time: 1.143s)
2025-01-07 17:35:50 D: root:        1.143s      1 call     html_directory
2025-01-07 17:35:50 D: root: ## recognizes (total time: 0.027s)
2025-01-07 17:35:50 D: root:        0.027s    142 calls    diffoscope.comparators.binary.FilesystemFile
2025-01-07 17:35:50 D: root: ## specialize (total time: 0.000s)
2025-01-07 17:35:50 D: root:        0.000s      1 call     specialize

As can be seen, some errors of diffocsocpe (full, as per constructed v0.2.5-test docker image) fails parsing cbfs content :(

Output of diffoscope states in header of html file:

bad-12mb.rom vs.
build/x86/x230-hotp-maximized_usb-kb/heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.rom
Command `cbfstool bad-12mb.rom extract -n fallback/romstage -f «TEMP»/diffoscope_vtq274q8_12mb/tmp3t1z6h0h_CbfsContainer/romstage` failed with exit code 1. (No output) 

diffoscope output:
bad-12mb.tar.gz

Next iteration if some other similar report given with rootfs failing to boot

• remove 12mb cbfs content from @marmarek
• compare roms again
• update flake.lock so that packages pinned are latest (for some reason nix requires building ceph with latest packages (no cache?!?!) and docker image was resulting in more then 12gb size with full diffoscope, I gave up maybe too soon)

[tlaurion]

• update flake.lock so that packages pinned are latest (for some reason nix requires building ceph with latest packages (no cache?!?!) and docker image was resulting in more then 12gb size with full diffoscope, I gave up maybe too soon)

Building v0.2.5-test2 with flake.lock updated to build full diffoscope (and bump all other nix packages to latest) and will push testting docker image when ready. It builds ceph from source and will take a while. Hopefully, cbfstool won't error and comparison of roms will show what's up otherwise more manual steps are needed. Edit: flake.lock updated with nix flake update has ceph building errors. I will put aside for now, trying bookworm available diffoscope (probably really old) without much hope.

ceph> [  7%] Built target jerasure_utils
ceph> make[2]: *** [src/msg/CMakeFiles/common-msg-objs.dir/build.make:90: src/msg/CMakeFiles/common-msg-objs.dir/Message.cc.o] Error 1
ceph> make[1]: *** [CMakeFiles/Makefile2:3148: src/msg/CMakeFiles/common-msg-objs.dir/all] Error 2
ceph> -- qatlib_ext build command succeeded.  See also /build/ceph-19.2.0/build/qatlib_ext-prefix/src/qatlib_ext-stamp/qatlib_ext-build-*.log
ceph> [  7%] Performing install step for 'qatlib_ext'
ceph> -- qatlib_ext install command succeeded.  See also /build/ceph-19.2.0/build/qatlib_ext-prefix/src/qatlib_ext-stamp/qatlib_ext-install-*.log
ceph> [  7%] Completed 'qatlib_ext'
ceph> [  7%] Built target qatlib_ext
ceph> [  8%] Built target common-options-objs
ceph> make: *** [Makefile:146: all] Error 2
error: builder for '/nix/store/07p234xd8l9914svh88vi1wv2bjai47n-ceph-19.2.0.drv' failed with exit code 2
error: 1 dependencies of derivation '/nix/store/0s6j637s0k08pzxp3mmgyb56k6rw9klr-qemu-9.2.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/srlx7qj137plm1x35vdgy84bi22kal9l-nix-shell-env.drv' failed to build

So newer qemu provided by nix is not yet cached so not downloadable, could hack into providing old dep but... how much should I put into this issue? OS cannot write to SPI (PR0 enforced), Heads doesn't write to SPI on daily operations, otherwise CBFS content changing and measurements failing (see cbmem -L output, all taken into consideration into PCR2)....

@marmarek as said, hypothesis is misconnection at this point more than anything else:

But it did worked for a while after the update. Last working date is 2024-12-14.

So it was not a misconstruction firmware from build(worked for a while). Not a bad flash (not flashrom/flashprog related). It worked for a while and stopped working with rootfs error (payload->extraction to ram, kernel pointing to ramfs failed meaning intramfs part corrupted, no why here without logs)

And as said earlier, there is nothing writing in CBFS that can corrupt payload(kernel+initramfs in CBFS. Occam's razor best explanation is spi connection to motherboard flakyness, because way too many external connection to SPI? Where flashing externally made this successful.

[tlaurion]

bash-5.2# build/x86/coreboot-24.02.01/util/cbfstool/cbfstool /home/user/heads/bad-12mb.rom print
FMAP REGION: COREBOOT
Name                           Offset     Type           Size   Comp
cbfs_master_header             0x0        cbfs header        32 none
fallback/romstage              0x80       stage          106800 none
cpu_microcode_blob.bin         0x1a240    microcode       26624 none
fallback/ramstage              0x20a80    stage          151309 LZMA (333096 decompressed)
config                         0x45a00    raw              3383 LZMA (10736 decompressed)
revision                       0x46780    raw               724 none
build_info                     0x46a80    raw               101 none
bootsplash.jpg                 0x46b40    bootsplash      43282 none
fallback/dsdt.aml              0x51480    raw             14715 none
vbt.bin                        0x54e40    raw              1433 LZMA (4281 decompressed)
cmos_layout.bin                0x55440    cmos_layout      2012 none
fallback/postcar               0x55c80    stage           39660 none
fallback/payload               0x5f7c0    simple elf    7803690 none
heads/initrd/.gnupg/pubring.kbx 0x7d0b40   raw              2872 none
heads/initrd/.gnupg/trustdb.gpg 0x7d16c0   raw              1360 none
heads/initrd/etc/config.user   0x7d1c80   raw               170 none
(empty)                        0x7d1d80   null          4151960 none
bootblock                      0xbc7840   bootblock       34176 none
bash-5.2# build/x86/coreboot-24.02.01/util/cbfstool/cbfstool /home/user/heads/bad-12mb.rom remove -n heads/initrd/.gnupg/pubring.kbx
bash-5.2# build/x86/coreboot-24.02.01/util/cbfstool/cbfstool /home/user/heads/bad-12mb.rom remove -n heads/initrd/.gnupg/trustdb.gpg
bash-5.2# build/x86/coreboot-24.02.01/util/cbfstool/cbfstool /home/user/heads/bad-12mb.rom remove -n heads/initrd/etc/config.user   
bash-5.2# build/x86/coreboot-24.02.01/util/cbfstool/cbfstool /home/user/heads/bad-12mb.rom print
FMAP REGION: COREBOOT
Name                           Offset     Type           Size   Comp
cbfs_master_header             0x0        cbfs header        32 none
fallback/romstage              0x80       stage          106800 none
cpu_microcode_blob.bin         0x1a240    microcode       26624 none
fallback/ramstage              0x20a80    stage          151309 LZMA (333096 decompressed)
config                         0x45a00    raw              3383 LZMA (10736 decompressed)
revision                       0x46780    raw               724 none
build_info                     0x46a80    raw               101 none
bootsplash.jpg                 0x46b40    bootsplash      43282 none
fallback/dsdt.aml              0x51480    raw             14715 none
vbt.bin                        0x54e40    raw              1433 LZMA (4281 decompressed)
cmos_layout.bin                0x55440    cmos_layout      2012 none
fallback/postcar               0x55c80    stage           39660 none
fallback/payload               0x5f7c0    simple elf    7803690 none
(empty)                        0x7d0b40   null          4156644 none
bootblock                      0xbc7840   bootblock       34176 none

Resulting rom with deleted cbfs files (but not zeroed!!!! resulting in diff):
bad-12mb.tar.gz

Diffoscope comparison, showing positional diff for romstage when extracted with cbfstool, then FF space for clean rom vs above rom:
bad-12mb_cbfs_files_removed-compared_with_clean_rom.tar.gz

cbfstool doesn't provide ways of cleaning. truncate gives same result as above.

After removing user related files, only a positional offset changed under romstage and basic analysis (doing forensic takes long) shows the following changes (your linked rom was the exact bit by bit repro of docker image created rom locally):

[tlaurion]

@marmarek found the discrepency. Seems like payload base address got corrupted in rom?!!?

repro:

docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:v0.2.5-test3
cat bad-bottom.rom bad-top.rom > bad-12mb.rom
bash-5.2# cbfstool bad-12mb.rom print
FMAP REGION: COREBOOT
Name                           Offset     Type           Size   Comp
cbfs_master_header             0x0        cbfs header        32 none
fallback/romstage              0x80       stage          106800 none
cpu_microcode_blob.bin         0x1a240    microcode       26624 none
fallback/ramstage              0x20a80    stage          151309 LZMA (333096 decompressed)
config                         0x45a00    raw              3383 LZMA (10736 decompressed)
revision                       0x46780    raw               724 none
build_info                     0x46a80    raw               101 none
bootsplash.jpg                 0x46b40    bootsplash      43282 none
fallback/dsdt.aml              0x51480    raw             14715 none
vbt.bin                        0x54e40    raw              1433 LZMA (4281 decompressed)
cmos_layout.bin                0x55440    cmos_layout      2012 none
fallback/postcar               0x55c80    stage           39660 none
fallback/payload               0x5f7c0    simple elf    7803690 none
heads/initrd/.gnupg/pubring.kbx 0x7d0b40   raw              2872 none
heads/initrd/.gnupg/trustdb.gpg 0x7d16c0   raw              1360 none
heads/initrd/etc/config.user   0x7d1c80   raw               170 none
(empty)                        0x7d1d80   null          4151960 none
bootblock                      0xbc7840   bootblock       34176 none
bash-5.2# cbfstool bad-12mb.rom extract -n fallback/payload -f bad_payload -m x86
Found file fallback/payload at 0x5f7c0, type simple elf, compressed 7803690, size 7803690
bash-5.2# cbfstool heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1.rom extract -n fallback/payload -f payload -m x86
Found file fallback/payload at 0x5f7c0, type simple elf, compressed 7803690, size 7803690

diffoscope bad_payload payload --markdown output:

Comparing bad_payload & payload

• Format-specific differences are supported for ELF binaries but no file-specific differences were detected; falling back to a binary diff. file(1) reports: ELF 32-bit LSB executable, Intel i386, version 1 (SYSV), statically linked, stripped

@@ -241406,15 +241406,15 @@
 003aefd0: 1f25 0e4a cea6 cab5 b438 d712 60a3 550b  .%.J.....8..`.U.
 003aefe0: f3d9 476a 9aee b02f 7f48 96e6 c7f2 6cbd  ..Gj.../.H....l.
 003aeff0: 5fea ade8 2635 afd4 304e 5d20 f746 8455  _...&5..0N] .F.U
 003af000: abf8 9cd9 8004 9d24 cfa3 7d2c ffa8 b92f  .......$..},.../
 003af010: 189f 9e01 462e c681 d805 28aa b78e 5ea4  ....F.....(...^.
 003af020: 77fb 8ca5 3892 1d52 0de7 202b 28a2 2888  w...8..R.. +(.(.
 003af030: 9d80 f37a 6c30 6f50 4d29 bc3b 9990 0c1a  ...zl0oPM).;....
-003af040: 1ebb e561 9067 42ac ea6e 86bd 8289 d8cf  ...a.gB..n......
+003af040: 1ebb e561 9067 42ac ea6e 86bd 8288 d8cf  ...a.gB..n......
 003af050: cadf 7669 4a5b a458 9695 da59 c8d2 9513  ..viJ[.X...Y....
 003af060: 9a33 8617 bc7f 3b1c 8cc3 ba9f a771 734a  .3....;......qsJ
 003af070: 7a33 eb0e bc3b 3875 3201 7c9d 5bb3 f78e  z3...;8u2.|.[...
 003af080: aba2 648b ef05 2e3a 4cc8 b578 995b 7141  ..d....:L..x.[qA
 003af090: f6cd 76c2 40dd df1a 98cd be90 f191 9b74  ..v.@..........t
 003af0a0: 48d6 c37e 9790 c25a 6ad2 45d1 1bae d9b3  H..~...Zj.E.....
 003af0b0: 52a5 d637 01d1 fcc1 ea9a f648 d4f7 5841  R..7.......H..XA

diffoscope output of positional error of diffoscope html-dir:
bad_payload_diffoscope.tar.gz

[marmarek]

So, one bit was flipped. This very well may be a flash wear... This machine in general was reflashed several times, but more like tens than hundreds... Maybe Heads does write something to flash at some of the steps during OS reinstall? Or it's simply very old system and not that reliable anymore...

Anyway, thanks for analyzing!