EL9.6: "Unpatched return thunk in use"

[tonyhutter]

System information

Type	Version/Name
Distribution Name	Almalinux
Distribution Version	9.6
Kernel Version	5.14.0-570.12.1
Architecture	x86_64
OpenZFS Version	master (a6f2025) or zfs-2.2.7

Describe the problem you're observing

When you load the zfs module on Almalinux 9.6, you will get a Unpatched return thunk in use. This should not happen! warning. Added here: torvalds/linux@4461438. Maybe it was backported to the EL 9.6 kernel?

Describe how to reproduce the problem

Load module on Almalinux 9.6

Include any warning/errors/backtraces from the system logs

[  186.636443] zfs: module license taints kernel.
[  186.970115] ------------[ cut here ]------------
[  186.970127] Unpatched return thunk in use. This should not happen!
[  186.970140] WARNING: CPU: 16 PID: 1694 at arch/x86/kernel/cpu/bugs.c:3108 __warn_thunk+0x26/0x30
[  186.970145] Modules linked in: zfs(POE+) spl(OE) nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink intel_rapl_msr intel_rapl_common nfit libnvdimm sunrpc qxl kvm_intel drm_ttm_helper binfmt_misc kvm ttm drm_kms_helper i2c_i801 i2c_smbus pcspkr joydev virtio_balloon drm xfs libcrc32c sd_mod ahci sg libahci crct10dif_pclmul libata crc32_pclmul crc32c_intel virtio_net ghash_clmulni_intel net_failover virtio_blk virtio_console virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse
[  186.970229] CPU: 16 PID: 1694 Comm: insmod Kdump: loaded Tainted: P           OE     -------  ---  5.14.0-570.12.1.el9_6.x86_64 #1
[  186.970231] Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.16.0-4.module+el8.9.0+19570+14a90618 04/01/2014
[  186.970232] RIP: 0010:__warn_thunk+0x26/0x30
[  186.970234] Code: 90 90 90 90 0f 1f 44 00 00 80 3d ff d3 2f 02 00 74 05 c3 cc cc cc cc 48 c7 c7 30 68 85 98 c6 05 ea d3 2f 02 01 e8 fa 1d 0c 00 <0f> 0b c3 cc cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90
[  186.970235] RSP: 0018:ffff9a38c0d77b70 EFLAGS: 00010286
[  186.970236] RAX: 0000000000000000 RBX: ffffffffc156bf90 RCX: 0000000000000000
[  186.970236] RDX: ffff88adfbc2d980 RSI: ffff88adfbc208c0 RDI: ffff88adfbc208c0
[  186.970237] RBP: ffff9a38c0d77bc0 R08: 0000000000000000 R09: ffff9a38c0d77a18
[  186.970237] R10: ffff9a38c0d77a10 R11: ffffffff993e17c8 R12: 0000000000000005
[  186.970238] R13: 0000561b814a9962 R14: ffff88ad825a2338 R15: ffffffff9a06de08
[  186.970238] FS:  00007f263d186740(0000) GS:ffff88adfbc00000(0000) knlGS:0000000000000000
[  186.970239] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  186.970239] CR2: 0000561b8149ebd0 CR3: 0000000115e22000 CR4: 0000000000752ef0
[  186.970241] PKRU: 55555554
[  186.970242] Call Trace:
[  186.970243]  <TASK>
[  186.970244]  ? show_trace_log_lvl+0x1c4/0x2df
[  186.970246]  ? show_trace_log_lvl+0x1c4/0x2df
[  186.970248]  ? warn_thunk_thunk+0x16/0x30
[  186.970250]  ? __warn_thunk+0x26/0x30
[  186.970251]  ? __warn+0x7e/0xd0
[  186.970253]  ? __warn_thunk+0x26/0x30
[  186.970254]  ? report_bug+0x100/0x140
[  186.970256]  ? prb_read_valid+0x17/0x20
[  186.970259]  ? handle_bug+0x3c/0x70
[  186.970260]  ? exc_invalid_op+0x14/0x70
[  186.970261]  ? asm_exc_invalid_op+0x16/0x20
[  186.970264]  ? __warn_thunk+0x26/0x30
[  186.970265]  warn_thunk_thunk+0x16/0x30
[  186.970267]  benchmark_raidz+0x40/0x1b0 [zfs]
[  186.970410]  ? __pfx_init_module+0x10/0x10 [zfs]
[  186.970479]  vdev_raidz_math_init+0xa/0xa0 [zfs]
[  186.970587]  spa_init+0x140/0x1d0 [zfs]
[  186.970703]  zfs_kmod_init+0x25/0xd0 [zfs]
[  186.970816]  ? __pfx_init_module+0x10/0x10 [zfs]
[  186.971960]  openzfs_init+0x38/0xdc7 [zfs]
[  186.972051]  do_one_initcall+0x41/0x210
[  186.972054]  do_init_module+0x64/0x230
[  186.972057]  init_module_from_file+0x8b/0xd0
[  186.972059]  idempotent_init_module+0x17e/0x230
[  186.972061]  __x64_sys_finit_module+0x5a/0xb0
[  186.972062]  do_syscall_64+0x5c/0xe0
[  186.972064]  ? do_syscall_64+0x6b/0xe0
[  186.972065]  ? __count_memcg_events+0x4f/0xb0
[  186.972067]  ? mm_account_fault+0x6c/0x100
[  186.972068]  ? handle_mm_fault+0x123/0x250
[  186.972070]  ? do_user_addr_fault+0x362/0x620
[  186.972072]  ? clear_bhb_loop+0x25/0x80
[  186.972074]  ? clear_bhb_loop+0x25/0x80
[  186.972075]  ? clear_bhb_loop+0x25/0x80
[  186.972076]  ? clear_bhb_loop+0x25/0x80
[  186.972077]  ? clear_bhb_loop+0x25/0x80
[  186.972078]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[  186.972080] RIP: 0033:0x7f263c90764d
[  186.972081] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 9b 17 0f 00 f7 d8 64 89 01 48
[  186.972082] RSP: 002b:00007ffda4c68eb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[  186.972084] RAX: ffffffffffffffda RBX: 0000561b81f257f0 RCX: 00007f263c90764d
[  186.972085] RDX: 0000000000000000 RSI: 0000561b814a9962 RDI: 0000000000000003
[  186.972085] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  186.972085] R10: 0000000000000003 R11: 0000000000000246 R12: 0000561b814a9962
[  186.972086] R13: 0000561b81f257c0 R14: 0000561b814a8550 R15: 0000561b81f25900
[  186.972087]  </TASK>
[  186.972087] ---[ end trace 0000000000000000 ]---
[  187.579670] ZFS: Loaded module v2.3.99-358_ga6f20250d (DEBUG mode), ZFS pool version 5000, ZFS filesystem version 5

[tonyhutter]

Looks like it's the curr_impl->is_supported() call:

(gdb) list *(benchmark_raidz+0x40)
0x231950 is in benchmark_raidz (/home/hutter/zfs/module/zfs/vdev_raidz_math.c:464).
459			curr_impl = (raidz_impl_ops_t *)raidz_all_maths[i];
460	
461			if (curr_impl->init)
462				curr_impl->init();
463	
464			if (curr_impl->is_supported())
465				raidz_supp_impl[c++] = (raidz_impl_ops_t *)curr_impl;
466		}
467		membar_producer();		/* complete raidz_supp_impl[] init */
468		raidz_supp_impl_cnt = c;	/* number of supported impl */

[tonyhutter]

It seems that adding .o files to OBJECT_FILES_NON_STANDARD_* in module/Kbuild.in is a workaround. It's whack-a-mole to see which .o files need to be added though..

[tonyhutter]

The big hammer silences a lot of these:

diff --git a/module/Kbuild.in b/module/Kbuild.in
index edb475c7d..bdfe0eaea 100644
--- a/module/Kbuild.in
+++ b/module/Kbuild.in
@@ -2,6 +2,8 @@
 # first.  This ensures its module initialization function is run before
 # any of the other module initialization functions which depend on it.
 
+OBJECT_FILES_NON_STANDARD := y
+
 ZFS_MODULE_CFLAGS += -std=gnu99 -Wno-declaration-after-statement
 ZFS_MODULE_CFLAGS += -Wmissing-prototypes
 ZFS_MODULE_CFLAGS += @KERNEL_DEBUG_CFLAGS@  @NO_FORMAT_ZERO_LENGTH@

Even with that in place, I'm still seeing warn_thunk_thunk() get invoked by cv_timedwait_idle_hires() and __cv_timedwait_idle() though.

[AttilaFueloep]

As a quick guess I'd say that the retbleed mitigation isn't happy with our raidz assembly. Have to dig further to say more.

[AttilaFueloep]

Some background cited from https://lkml.org/lkml/2023/10/11/1460:

With CONFIG_RETHUNK enabled, the compiler replaces every RET with a tail call to a return thunk ('JMP__x86_return_thunk'). Objtool annotates all such return sites so they can be patched during boot by apply_returns().

The implementation of __x86_return_thunk() is just a bare RET. It's only meant to be used temporarily until apply_returns() patches all return sites with either a JMP to another return thunk or an actual RET.

The Linux commit Tony cited above added a check and a warning for unpatched return thunks.

So either we fail to annotate the return sites with objtool during module build or the patching during module load fails. Most likely it's the first. I've no idea how to fix this yet, need to do more research.

[AttilaFueloep]

This seems to be a AlmaLinux 9.6 issue. The bug is described in [1]. After applying the patch in [2] to scripts/Makefile.modfinal the warning goes away. [3] mentions a different approach I didn't search for, but this issue should be fixed in newer kernels.

Guess AlmaLinux needs to apply a patch, not sure how to proceed.

[1]
[2]
[3]

[root@alma9 ~]# diff -u /usr/src/kernels/5.14.0-570.12.1.el9_6.x86_64/scripts/Makefile.modfinal~ /usr/src/kernels/5.14.0-570.12.1.el9_6.x86_64/scripts/Makefile.modfinal
--- /usr/src/kernels/5.14.0-570.12.1.el9_6.x86_64/scripts/Makefile.modfinal~	2025-05-13 12:22:42.000000000 +0200
+++ /usr/src/kernels/5.14.0-570.12.1.el9_6.x86_64/scripts/Makefile.modfinal	2025-05-24 23:46:17.437340115 +0200
@@ -23,7 +23,7 @@
 part-of-module = y
 
 quiet_cmd_cc_o_c = CC [M]  $@
-      cmd_cc_o_c = $(CC) $(filter-out $(CC_FLAGS_CFI) $(CFLAGS_GCOV), $(c_flags)) -c -o $@ $<
+      cmd_cc_o_c = $(CC) $(filter-out $(CC_FLAGS_CFI) $(CFLAGS_GCOV) $(CFLAGS_KCSAN), $(c_flags)) -c -o $@ $<
 
 %.mod.o: %.mod.c FORCE
 	$(call if_changed_dep,cc_o_c)

Edit: AlmaLinux of course, old habits.

[AttilaFueloep]

To summarize the issue: AlmaLinux 9.6 comes with CONFIG_KCSAN enabled by default but misses a kernel patch. While building the module, one of the final steps is to generate a zfs.mod.c file and compile that. Due to the bug the compiler adds a function to .text.startup which never gets annotated since objtool has already been run, but ends up in the module. On module load time it can't be patched due to the missing annotation and triggers the warning.

All of that of course IIUC

[andrewlukoshko]

CONFIG_KCSAN is not enabled in AlmaLinux 9.6
https://git.almalinux.org/rpms/kernel/src/commit/c1585a853adf564dfd4c4717358965063d08178b/SOURCES/kernel-x86_64-rhel.config#L2770

[mdc-webdb]

Also disabled on upstream.

grep CONFIG_KCSAN /boot/config-5.14.0-570.17.1.el9_6.x86_64
# CONFIG_KCSAN is not set

[AttilaFueloep]

Yes, right. I could've sworn I'v seen it enabled in .config (and being stunned), seems I was too tired yesterday, sorry.

Still, with the original Makefile.modfinal the produced module triggers the warning on module load while the one build with the patched Makefile does not. Just double checked. Not sure what to make out of this, seems to be a variation of the issue.

[ 3367.424346] ---[ end trace 0000000000000000 ]---
[ 3368.302909] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5
[ 3434.310906] ZFS: Unloaded module v2.3.2-1
[ 4599.115239] systemd-rc-local-generator[492664]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 4661.588808] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5
[root@alma9 scripts]#

[AttilaFueloep]

Scratch what I wrote earlier, I missed that this is a WARN_ONCE. After echo 1 > /sys/kernel/debug/clear_warn_once the warning is back.

Sorry for the confusion.

[tonyhutter]

I'm not claiming this is the right fix, but this does silence the error for me:

diff --git a/module/Kbuild.in b/module/Kbuild.in
index edb475c7d..e51648ea0 100644
--- a/module/Kbuild.in
+++ b/module/Kbuild.in
@@ -6,6 +6,11 @@ ZFS_MODULE_CFLAGS += -std=gnu99 -Wno-declaration-after-statement
 ZFS_MODULE_CFLAGS += -Wmissing-prototypes
 ZFS_MODULE_CFLAGS += @KERNEL_DEBUG_CFLAGS@  @NO_FORMAT_ZERO_LENGTH@
 
+# Workaround for https://github.com/openzfs/zfs/issues/17364
+OBJECT_FILES_NON_STANDARD := y
+ZFS_MODULE_CFLAGS += -mfunction-return=thunk
+KCPPFLAGS =+ -mfunction-return=thunk
+
 ifneq ($(KBUILD_EXTMOD),)
 zfs_include = @abs_top_srcdir@/include
 icp_include = @abs_srcdir@/icp/include