Clang THINLTO BTRFS balance dmesg regression with RAID

[ptr1337]

Hi,

Im reporting this issue for @loathingKernel .
He is currently facing a issue, that the kernel built with thin lto is having a dmesg regression.
This issue is according him not present with a GCC built kernel.
The kernel version is 6.1.11-zen.
Here is a trace from dmesg:

${color8}[ +13.095108] ------------[ cut here ]------------
${color8}[  +0.000006] WARNING: CPU: 0 PID: 1288 at fs/btrfs/relocation.c:4096 btrfs_relocate_block_group+0x4c
              3/0x4e0 [btrfs]
${color8}[  +0.000163] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device nft_reject_ipv4 nf_
              reject_ipv4 nft_reject nft_ct nft_masq nft_chain_nat nf_tables nfnetlink nf_nat_h323 nf_conntrack_h323 nf_nat_
              pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_sip nf_conntrack_sip nf_nat_irc nf_conntrack_irc n
              f_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard ip6_udp_tunnel udp_tunn
              el libchacha20poly1305 poly1305_x86_64 chacha_x86_64 libchacha curve25519_x86_64 libcurve25519_generic bridge
              stp llc nct6775 hwmon_vid nct6775_core jc42 nvidia_drm(POE) nvidia_modeset(POE) intel_rapl_msr intel_rapl_comm
              on x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul polyva
              l_clmulni polyval_generic vboxnetflt(OE) vboxnetadp(OE) gf128mul ghash_clmulni_intel ath9k at24 ath9k_common i
              TCO_wdt sha512_ssse3 ath9k_hw intel_pmc_bxt vboxdrv(OE) iTCO_vendor_support mei_pxp mei_hdcp aesni_intel mac80
              211 usbip_host
${color8}[  +0.000109]  usbip_core crypto_simd snd_hda_codec_realtek snd_hda_codec_generic mxm_wmi cryptd ledt
              rig_audio libarc4 rapl snd_hda_codec_hdmi ath uas intel_cstate usb_storage usbhid xpad snd_hda_intel ff_memles
              s nvidia_uvm(POE) snd_intel_dspcfg snd_soc_rt5640 i2c_i801 intel_uncore i2c_smbus snd_soc_rl6231 mei_me lpc_ic
              h cfg80211 snd_hda_codec snd_hwdep snd_soc_core snd_hda_core snd_compress rfkill mei e1000e snd_pcm btrfs snd_
              timer raid6_pq snd video xor libcrc32c soundcore wmi acpi_pad mac_hid nvidia(POE) sg vhba crypto_user fuse ip_
              tables x_tables ext4 crc32c_generic mbcache crc16 jbd2 crc32c_intel xhci_pci xhci_pci_renesas
${color8}[  +0.000090] CPU: 0 PID: 1288 Comm: btrfs Tainted: P        W  OE      6.1.11-zen1-1-zen-akasha ::1
              f56d26afe801eeea62421aa2a24beb96cdbb3e0a
${color8}[  +0.000010] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.60 0
              3/06/2018
${color8}[  +0.000003] RIP: 0010:btrfs_relocate_block_group+0x4c3/0x4e0 [btrfs]
${color8}[  +0.000139] Code: 8b 24 24 e9 fe fe ff ff 0f 0b 49 8b 04 24 48 83 78 30 00 0f 84 d9 fe ff ff 0f 0b
              49 8b 04 24 48 83 78 38 00 0f 84 d3 fe ff ff <0f> 0b 45 85 ff 0f 85 d1 fe ff ff 45 31 ff e9 d2 fe ff ff 00 00
              00
${color8}[  +0.000006] RSP: 0018:ffffb78c43c0fb88 EFLAGS: 00010206
${color8}[  +0.000006] RAX: ffff94bb555aa000 RBX: ffff94bb42479000 RCX: 0000000000000000
${color8}[  +0.000004] RDX: 0000000000000000 RSI: ffffe33c0e808540 RDI: ffff94bb42479330
${color8}[  +0.000004] RBP: ffff94bb42479330 R08: 0000000080240024 R09: 0000000080240023
${color8}[  +0.000003] R10: ffffe33c0e808540 R11: ffff94bdcd987028 R12: ffff94bb67c43000
${color8}[  +0.000003] R13: 0000000000000401 R14: ffff94bb42479010 R15: 0000000000000000
${color8}[  +0.000004] FS:  00007f17ea24c900(0000) GS:ffff94be5ec00000(0000) knlGS:0000000000000000
${color8}[  +0.000005] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
${color8}[  +0.000004] CR2: 00007f974ddd3580 CR3: 000000015d000003 CR4: 00000000001706f0
${color8}[  +0.000004] Call Trace:
${color8}[  +0.000004]  <TASK>
${color8}[  +0.000005]  btrfs_relocate_chunk+0x43/0x150 [btrfs b81c5614352b2c8e4ef453a022ee2db9aac40728]
${color8}[  +0.000176]  __btrfs_balance+0x736/0x930 [btrfs b81c5614352b2c8e4ef453a022ee2db9aac40728]
${color8}[  +0.000147]  btrfs_balance+0x9b8/0xac0 [btrfs b81c5614352b2c8e4ef453a022ee2db9aac40728]
${color8}[  +0.000136]  ? slab_post_alloc_hook+0x95/0x370
${color8}[  +0.000014]  ? slab_post_alloc_hook+0x50/0x370
${color8}[  +0.000010]  btrfs_ioctl_balance+0x235/0x300 [btrfs b81c5614352b2c8e4ef453a022ee2db9aac40728]
${color8}[  +0.000136]  __se_sys_ioctl+0x6f/0xb0
${color8}[  +0.000008]  do_syscall_64+0x64/0x90
${color8}[  +0.000012]  ? exc_page_fault+0x77/0x1a0
${color8}[  +0.000011]  entry_SYSCALL_64_after_hwframe+0x46/0xb0
${color8}[  +0.000011] RIP: 0033:0x7f17ea39353f
${color8}[  +0.000043] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d
              44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 18 48 8b 44 24 18 64 48 2b 04 25 28 00
              00
${color8}[  +0.000005] RSP: 002b:00007ffefe5c3520 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
${color8}[  +0.000006] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f17ea39353f
${color8}[  +0.000003] RDX: 00007ffefe5c3620 RSI: 00000000c4009420 RDI: 0000000000000003
${color8}[  +0.000003] RBP: 0000000000000000 R08: 0000000000000013 R09: 00005620a8fa9e0c
${color8}[  +0.000004] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffefe5c55ed
[config-6.1.11.txt](https://github.com/ClangBuiltLinux/linux/files/10731267/config-6.1.11.txt)

${color8}[  +0.000003] R13: 00007ffefe5c3620 R14: 0000000000000000 R15: 0000000000000000
${color8}[  +0.000006]  </TASK>
${color8}[  +0.000002] ---[ end trace 0000000000000000 ]---

The kernel has been build in a chroot with clang 15.0.7 from archlinux (https://archlinux.org/packages/extra/x86_64/clang/).
Kernel config is attached.

[ptr1337]

config-6.1.11.txt

[nathanchance]

Is the issue consistently reproducible? All of my Arch Linux machines use btrfs with clang built kernels using ThinLTO and I have never seen this message but I do not use RAID, so that could be related. What is the best way to configure that, so I can try to reproduce the problem?

[loathingKernel]

Hi, I would like to give some more information now that btrfs balance has finished without any issues on the same RAID array with the LTO-less kernel.

The issue appeared when running a btrfs balance operation. I have noticed it for a number of kernel version built with LTO_THIN (I have not tried LTO_FULL) but I do not have logs. The operation didn't stop though as a result of the failure recorded in the trace.

It is a BTRFS RAID 0 array (not mdadm) spanning 4 disk partitions. The structure is as shown below

Label: 'array_a'  uuid: 8efae01a-8d18-48c6-85d9-42d5b338e69e
	Total devices 4 FS bytes used 846.43GiB
	devid    1 size 232.88GiB used 212.03GiB path /dev/sdc1
	devid    2 size 232.88GiB used 213.00GiB path /dev/sdd1
	devid    3 size 232.88GiB used 213.00GiB path /dev/sde1
	devid    4 size 232.88GiB used 212.03GiB path /dev/sdf1

The following kernel configs resulted in a number of traces like the one posted in the initial report
config-6.1.10-1-localmod.txt
Built with clang 15.0.7 on Arch, with CONFIG_LTO_CLANG_THIN, CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE_O3, and CONFIG_MHASWELL
config-6.1.11-1-localmod.txt
Built with clang 15.0.7 on Arch, with CONFIG_LTO_CLANG_THIN, CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE, and CONFIG_MHASWELL

The following config completed the balance operation on the same RAID 0 array without issues.
config-6.1.11-2-localmod.txt
Built with clang 15.0.7 on Arch, with CONFIG_LTO_NONE, CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE, and CONFIG_MHASWELL

All kernels are from the zen-kernel source tree, using the linux-zen package config as a base
The 6.1.11 configs are built on the exact same source of ArchLinux's zen-kernel, the pkgrel bump was for my own repo.

If you need anything more or need clarifications, testing, etc, I am more than happy to provide them.

[nickdesaulniers]

The warning from the dump doesn't make much sense to me, and ./scripts/decodecode < oops.txt is printing some odd stuff claiming the faulting instruction is a mov %eax,%edx which doesn't sound right.

@loathingKernel if you have the exact kernel image that produced that text, could you throw it somewhere we can fetch it to disassemble it?

[loathingKernel]

No I do not have it any more sadly. Although I can let the kernel build again with the previous config, see if that reproduces it and report back.

[nickdesaulniers]

Yes please. We'd need a matching dmesg with the matched kernel image to debug further.

The kernel version is 6.1.11-zen.
${color8}[ +0.000006] WARNING: CPU: 0 PID: 1288 at fs/btrfs/relocation.c:4096

For example, fs/btrfs/relocation.c:4096 doesn't line up with any WARN_ON in mainline. That file has had multiple commits in v6.2-rc1. Nothing with a Fixes tag between 6.1 and 6.2, but I only looked at commit messages to fs/btrfs/relocation.c.

[loathingKernel]

That got me wondering so I looked at the zen-kernel tree for 6.1 and it lines up with this https://github.com/zen-kernel/zen-kernel/blob/v6.1.11-zen1/fs/btrfs/relocation.c#L4094-L4096 if I understood you correctly.

[nathanchance]

I have tried to reproduce this in QEMU using an Arch Linux virtual machine to no avail, but it is possible that I need more data, as I have just been streaming a random number of bytes from /dev/urandom to a series of files on the filesystem.

# btrfs filesystem show /mnt/btrfs
Label: none  uuid: 9658dc4c-51be-43c1-a6be-3c52300ed2a6
        Total devices 4 FS bytes used 4.06GiB
        devid    1 size 25.00GiB used 2.27GiB path /dev/vda1
        devid    2 size 25.00GiB used 2.27GiB path /dev/vdb1
        devid    3 size 25.00GiB used 2.27GiB path /dev/vdc1
        devid    4 size 25.00GiB used 2.27GiB path /dev/vdd1

# btrfs balance /mnt/btrfs
Done, had to relocate 4 out of 4 chunks

# dmesg
[    0.000000] Linux version 6.1.12-00003-gecb0be182b6e (nathan@dev-arch.thelio-3990X) (clang version 15.0.7, LLD 15.0.7) #1 SMP PREEMPT_DYNAMIC Tue Feb 14 17:08:26 MST 2023
...
[  247.097026] BTRFS info (device vda1): balance: start -d -m -s
[  247.097251] BTRFS info (device vda1): relocating block group 49699356672 flags system|raid0
[  247.169555] BTRFS info (device vda1): relocating block group 48625614848 flags metadata|raid0
[  247.312287] BTRFS info (device vda1): found 991 extents, stage: move data extents
[  247.416687] BTRFS info (device vda1): relocating block group 44330647552 flags data|raid0
[  247.572496] BTRFS info (device vda1): found 179 extents, stage: move data extents
[  247.668997] BTRFS info (device vda1): found 179 extents, stage: update data pointers
[  247.736244] BTRFS info (device vda1): relocating block group 40035680256 flags data|raid0
[  254.560449] BTRFS info (device vda1): found 16168 extents, stage: move data extents
[  254.956949] BTRFS info (device vda1): found 16168 extents, stage: update data pointers
[  255.108228] BTRFS info (device vda1): balance: ended with status: 0

[loathingKernel]

Sorry for the radio silence, this is a work machine and haven't had the opportunity to tinker with it. I have been unable to reproduce the issue locally too after running that first successful balance.

I recently have had a few segfaults in the NTFS3 driver that I though might be related (for which I have collected logs) but it turns out those were present in the same kernel compiled with gcc too, so they are probably irrelevant.

That being said, I do have a feeling that the kernel panics might be related to the integrity of the filesystems, as in they are triggered because of a faulty filesystem, as these RAID arrays are also shared with Windows, under which they operate using the WinBtrfs driver. I have not used these filesystems under Windows since, so I don't know if that would actually trigger it.

At this point, what happens with this issue is up to you. I am trying to reproduce it but I have been unsuccessful so far, although it used to appear quite consistently. If you wouldn't mind, we could keep this open for now in case it appears again.