Kernel panic switching netvm on Linux 5.16.5

[pgerber]

How to file a helpful issue

Qubes OS release

$ cat /etc/qubes-release 
Qubes release 4.0 (R4.0)

Brief summary

Kernel panic trying to switch netvm on Linux kernel 5.16.5-1.fc25.qubes.x86_64 from kernel-latest-qubes-vm package.

Steps to reproduce

1. start vm (qvm-start example)
2. switch netvm (qvm-prefs example netvm example-netvm)

Expected behavior

VM is attached to newly selected netvm.

Actual behavior

Kernel crashes:

[ 2017.626976] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 2017.626997] #PF: supervisor read access in kernel mode
[ 2017.627009] #PF: error_code(0x0000) - not-present page
[ 2017.627022] PGD 0 P4D 0 
[ 2017.627030] Oops: 0000 [#1] PREEMPT SMP PTI
[ 2017.627041] CPU: 0 PID: 63 Comm: xenwatch Tainted: G           O      5.16.5-1.fc25.qubes.x86_64 #1
[ 2017.627062] RIP: 0010:free_netdev+0xb2/0x180
[ 2017.627077] Code: 8b 45 50 48 8b 08 49 39 c4 48 8d b8 a0 fe ff ff 48 8d 99 a0 fe ff ff 74 27 e8 3a fe ff ff e8 45 c4 ff ff 48 8d 83 60 01 00 00 <48> 8b 93 60 01 00 00 48 89 df 4c 39 e0 48 8d 9a a0 fe ff ff 75 d9
[ 2017.627117] RSP: 0018:ffffc90000357cf0 EFLAGS: 00010282
[ 2017.627129] RAX: 0000000000000000 RBX: fffffffffffffea0 RCX: 0000000000000002
[ 2017.627147] RDX: 0000000000000000 RSI: ffffc90000357c20 RDI: 00000000ffffffff
[ 2017.627164] RBP: ffff8880a73a3000 R08: 0000000000000001 R09: 0000000000000000
[ 2017.627182] R10: ffffffff82865480 R11: 0000000000000000 R12: ffff8880a73a3050
[ 2017.627199] R13: ffffffffc03ce180 R14: dead000000000100 R15: 0000000000000000
[ 2017.627217] FS:  0000000000000000(0000) GS:ffff8882e3200000(0000) knlGS:0000000000000000
[ 2017.627236] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2017.627250] CR2: 0000000000000000 CR3: 00000000d59b2002 CR4: 00000000003706f0
[ 2017.627272] Call Trace:
[ 2017.627280]  <TASK>
[ 2017.627289]  xennet_remove+0x275/0x2f0 [xen_netfront]
[ 2017.627308]  xenbus_dev_remove+0x6e/0xf0
[ 2017.627320]  device_release_driver_internal+0xfa/0x1d0
[ 2017.627335]  bus_remove_device+0xd8/0x140
[ 2017.627349]  device_del+0x18d/0x3c0
[ 2017.627359]  ? _raw_spin_unlock+0x16/0x30
[ 2017.627370]  ? klist_iter_exit+0x14/0x20
[ 2017.627382]  ? xenbus_probe_devices+0x120/0x120
[ 2017.627395]  device_unregister+0x13/0x60
[ 2017.627406]  xenbus_dev_changed+0xce/0x210
[ 2017.627417]  xenwatch_thread+0xdd/0x1b0
[ 2017.627428]  ? remove_wait_queue+0x60/0x60
[ 2017.627438]  ? find_watch+0x40/0x40
[ 2017.627448]  kthread+0x14b/0x190
[ 2017.627459]  ? set_kthread_struct+0x40/0x40
[ 2017.627470]  ret_from_fork+0x22/0x30
[ 2017.627483]  </TASK>
[ 2017.627489] Modules linked in: nf_conntrack_netlink xt_addrtype br_netfilter bridge stp llc ip6t_REJECT nf_reject_ipv6 ip6_tables ipt_REJECT nf_reject_ipv4 xt_state xt_conntrack nft_counter nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel xen_netfront snd_pcm snd_timer snd soundcore pcspkr xen_netback u2mfn(O) xen_gntdev xen_gntalloc xen_blkback xen_evtchn parport_pc ppdev lp parport drm fuse sunrpc xenfs xen_privcmd bpf_preload ip_tables overlay xen_blkfront
[ 2017.627629] CR2: 0000000000000000
[ 2017.627641] ---[ end trace d83f7ae915afd12e ]---
[ 2017.627653] RIP: 0010:free_netdev+0xb2/0x180
[ 2017.627667] Code: 8b 45 50 48 8b 08 49 39 c4 48 8d b8 a0 fe ff ff 48 8d 99 a0 fe ff ff 74 27 e8 3a fe ff ff e8 45 c4 ff ff 48 8d 83 60 01 00 00 <48> 8b 93 60 01 00 00 48 89 df 4c 39 e0 48 8d 9a a0 fe ff ff 75 d9
[ 2017.627706] RSP: 0018:ffffc90000357cf0 EFLAGS: 00010282
[ 2017.627719] RAX: 0000000000000000 RBX: fffffffffffffea0 RCX: 0000000000000002
[ 2017.627736] RDX: 0000000000000000 RSI: ffffc90000357c20 RDI: 00000000ffffffff
[ 2017.627753] RBP: ffff8880a73a3000 R08: 0000000000000001 R09: 0000000000000000
[ 2017.627771] R10: ffffffff82865480 R11: 0000000000000000 R12: ffff8880a73a3050
[ 2017.627788] R13: ffffffffc03ce180 R14: dead000000000100 R15: 0000000000000000
[ 2017.627806] FS:  0000000000000000(0000) GS:ffff8882e3200000(0000) knlGS:0000000000000000
[ 2017.627823] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2017.627839] CR2: 0000000000000000 CR3: 00000000d59b2002 CR4: 00000000003706f0
[ 2017.627856] Kernel panic - not syncing: Fatal exception
[ 2017.628402] Kernel Offset: disabled

[marmarek]

Patch posted: https://lore.kernel.org/xen-devel/20220220134202.2187485-1-marmarek@invisiblethingslab.com/T/#u

[DemiMarie]

I get what looks like the same panic on Linux 5.10.96

[marmarek]

I get what looks like the same panic on Linux 5.10.96

Indeed the bad commit was backported to 5.10.94 too