Linux stable >= 6.9.0 & Linux longterm >= 6.4.1: monitor mode and frame injection broken on mt76 series devices

[olivervbk]

I have a Raspberry Pi 4b and a wifi USB dongle Alfa AWUS036ACM

When I run:
hcxdumptool -i wlan1 -F
and once it has initialized the interface, last message is:

   LAST   E 2 MAC-AP-ROGUE   MAC-CLIENT   ESSID (last seen on top)
-----------------------------------------------------------------------------------------

and the Pi stops responding. Checking in dmesg, I see this:

Jul 21 05:40:39 rpi4b-aarch kernel: mt76x2u 2-2:1.0 wlan1: entered promiscuous mode
Jul 21 05:40:45 rpi4b-aarch kernel: Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
Jul 21 05:40:45 rpi4b-aarch kernel: Mem abort info:
Jul 21 05:40:45 rpi4b-aarch kernel:   ESR = 0x0000000096000044
Jul 21 05:40:45 rpi4b-aarch kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Jul 21 05:40:45 rpi4b-aarch kernel:   SET = 0, FnV = 0
Jul 21 05:40:45 rpi4b-aarch kernel:   EA = 0, S1PTW = 0
Jul 21 05:40:45 rpi4b-aarch kernel:   FSC = 0x04: level 0 translation fault
Jul 21 05:40:45 rpi4b-aarch kernel: Data abort info:
Jul 21 05:40:45 rpi4b-aarch kernel:   ISV = 0, ISS = 0x00000044, ISS2 = 0x00000000
Jul 21 05:40:45 rpi4b-aarch kernel:   CM = 0, WnR = 1, TnD = 0, TagAccess = 0
Jul 21 05:40:45 rpi4b-aarch kernel:   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Jul 21 05:40:45 rpi4b-aarch kernel: user pgtable: 4k pages, 48-bit VAs, pgdp=0000000041300000
Jul 21 05:40:45 rpi4b-aarch kernel: [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
Jul 21 05:40:45 rpi4b-aarch kernel: Internal error: Oops: 0000000096000044 [#1] PREEMPT SMP

---

OS information: Archlinux ARM AArch64

$ uname -a
Linux rpi4b-aarch 6.9.10-1-aarch64-ARCH #1 SMP PREEMPT_DYNAMIC Sat Jul 20 06:30:37 MDT 2024 aarch64 GNU/Linux
$ dmesg | grep mt76
[    8.578070] mt76x2u 2-2:1.0: ASIC revision: 76120044
[    8.620950] mt76x2u 2-2:1.0: ROM patch build: 20141115060606a
[    8.806949] mt76x2u 2-2:1.0: Firmware Version: 0.0.00
[    8.812104] mt76x2u 2-2:1.0: Build: 1
[    8.815829] mt76x2u 2-2:1.0: Build Time: 201507311614____
[    9.549762] usbcore: registered new interface driver mt76x2u
$ hcxdumptool -v
hcxdumptool 6.3.4 (C) 2024 ZeroBeat
running on Linux kernel 6.9.10-1-aarch64-ARCH
running GNU libc version 2.39
compiled by gcc 12.1.0
compiled with Linux API headers 6.7.0
compiled with GNU libc headers 2.35
enabled REALTIME DISPLAY
enabled GPS support
enabled BPF compiler

---

Some attempts at debugging this:

$ sudo hcxdumptool -L
Requesting physical interface capabilities. This may take some time.
Please be patient...

available wlan devices:

phy idx hw-mac       virtual-mac  m ifname           driver (protocol)
---------------------------------------------------------------------------------------------
  0   3 yyyyyyyyyyyy yyyyyyyyyyyy - wlan0            brcmfmac (NETLINK)
  1   4 xxxxxxxxxxxx xxxxxxxxxxxx * wlan1            mt76x2u (NETLINK)

* active monitor mode available (reported by driver - do not trust it)
+ monitor mode available (reported by driver)
- no monitor mode available

$ sudo hcxdumptool -i wlan1
Requesting physical interface capabilities. This may take some time.
Please be patient...

interface information:

phy idx hw-mac       virtual-mac  m ifname           driver (protocol)
---------------------------------------------------------------------------------------------
  1   4 xxxxxxxxxxxx xxxxxxxxxxxx * wlan1            mt76x2u (NETLINK)


available frequencies: frequency [channel] tx-power of Regulatory Domain: CA

  2412 [  1] 23.0 dBm     2417 [  2] 23.0 dBm     2422 [  3] 23.0 dBm     2427 [  4] 23.0 dBm
  2432 [  5] 23.0 dBm     2437 [  6] 23.0 dBm     2442 [  7] 23.0 dBm     2447 [  8] 23.0 dBm
  2452 [  9] 23.0 dBm     2457 [ 10] 23.0 dBm     2462 [ 11] 23.0 dBm     2467 [ 12] disabled
  2472 [ 13] disabled     2484 [ 14] disabled     5180 [ 36] 20.0 dBm     5200 [ 40] 20.0 dBm
  5220 [ 44] 20.0 dBm     5240 [ 48] 20.0 dBm     5260 [ 52] 20.0 dBm     5280 [ 56] 20.0 dBm
  5300 [ 60] 20.0 dBm     5320 [ 64] 20.0 dBm     5500 [100] 20.0 dBm     5520 [104] 20.0 dBm
  5540 [108] 20.0 dBm     5560 [112] 20.0 dBm     5580 [116] 20.0 dBm     5600 [120] disabled
  5620 [124] disabled     5640 [128] disabled     5660 [132] 20.0 dBm     5680 [136] 20.0 dBm
  5700 [140] 20.0 dBm     5720 [144] 20.0 dBm     5745 [149] 20.0 dBm     5765 [153] 20.0 dBm
  5785 [157] 20.0 dBm     5805 [161] 20.0 dBm     5825 [165] 20.0 dBm     5845 [169] disabled
  5865 [173] disabled     5885 [177] disabled

I don't have NetworkManager running. I tried with iwd enabled/disabled, same error.
tshark/airodump-ng/angryoxide don't have issues running.

Services running (I have the same behavior when iwd is disabled):

$ sudo systemctl --type=service --state=running
  UNIT                       LOAD   ACTIVE SUB     DESCRIPTION
  cronie.service             loaded active running Command Scheduler
  dbus-broker.service        loaded active running D-Bus System Message Bus
  getty@tty1.service         loaded active running Getty on tty1
  iwd.service                loaded active running Wireless service
  serial-getty@ttyS1.service loaded active running Serial Getty on ttyS1
  sshd.service               loaded active running OpenSSH Daemon
  systemd-journald.service   loaded active running Journal Service
  systemd-logind.service     loaded active running User Login Management
  systemd-networkd.service   loaded active running Network Configuration
  systemd-resolved.service   loaded active running Network Name Resolution
  systemd-timesyncd.service  loaded active running Network Time Synchronization
  systemd-udevd.service      loaded active running Rule-based Manager for Device Events and Files
  user@1000.service          loaded active running User Manager for UID 1000

Anything else I can try?

[ZerBea]

I've installed Linux kernel 6.9.10 and a can reproduce it:

$ uname -r
6.9.10-arch1-1

The driver crashes immediately after the device entered promiscuous mode and tries to inject the first frame.

To make sure it really isn't hcxdumptool it can be reproduced running aireplay-ng:

$ sudo ip link set wlp5s0f4u2 down
$ sudo iw wlp5s0f4u2 set monitor none
$ sudo ip link set wlp5s0f4u2 up
$ sudo ./aireplay-ng --test wlp5s0f4u2

BTW:
The driver crashes my entire notebook and I have to power it off (hard reset).

While the LTS kernel is still working as expected:

$ uname -r
6.6.40-1-lts

ALFA AWUS036ACM

$ lsusb
Bus 003 Device 003: ID 0e8d:7612 MediaTek Inc. MT7612U 802.11a/b/g/n/ac Wireless Adapter

$ hcxdumptool -l
  1	  4	00c0caa1fe82	aae98516fb65	*	wlp5s0f4u2      	mt76x2u	NETLINK

sudo hcxdumptool -i wlp5s0f4u2 --rcascan=active
...
48 Packet(s) captured by kernel
0 Packet(s) dropped by kernel
15 PROBERESPONSE(s) captured

$ dmesg
[   17.994124] usb 3-2: new high-speed USB device number 3 using xhci_hcd
[   18.137382] usb 3-2: New USB device found, idVendor=0e8d, idProduct=7612, bcdDevice= 1.00
[   18.137396] usb 3-2: New USB device strings: Mfr=2, Product=3, SerialNumber=4
[   18.137402] usb 3-2: Product: Wireless 
[   18.137407] usb 3-2: Manufacturer: MediaTek Inc.
[   18.137412] usb 3-2: SerialNumber: 000000000
[   18.330951] usb 3-2: reset high-speed USB device number 3 using xhci_hcd
[   18.473890] mt76x2u 3-2:1.0: ASIC revision: 76120044
[   18.547565] mt76x2u 3-2:1.0: ROM patch build: 20141115060606a
[   18.732828] mt76x2u 3-2:1.0: Firmware Version: 0.0.00
[   18.732841] mt76x2u 3-2:1.0: Build: 1
[   18.732846] mt76x2u 3-2:1.0: Build Time: 201507311614____
[   19.637232] ieee80211 phy1: Selected rate control algorithm 'minstrel_ht'
[   19.638657] usbcore: registered new interface driver mt76x2u
[   19.661432] mt76x2u 3-2:1.0 wlp5s0f4u2: renamed from wlan0
[   25.584547] block nvme0n1: No UUID available providing old NGUID
[   78.168025] mt76x2u 3-2:1.0 wlp5s0f4u2: entered promiscuous mode
[   86.300331] mt76x2u 3-2:1.0 wlp5s0f4u2: left promiscuous mode

Looks like a driver error because the LTS kernel is not affected.

[ZerBea]

Linux Kernel 5.15.150 (OpenWRT) 23.05 is also working as expected.

[ZerBea]

Anything else I can try?
I suggest to go back to a kernel that is not broken.

[morrownr]

Going back to the last kernel that is not broken is an excellent idea.

Over the last several months, there has been an extraordinary amount of new code added to the wifi stack and we are regularly seeing bug reports caused by this modernization of the stack. Adding WiFi 7 support can cause bugs in drivers that not WiFi 7. It will take time.

[olivervbk]

Thanks for the answers!
I rolled back to 6.8.2-1-aarch64-ARCH (last 6.8.x kernel I found) and it works again.

For anyone curious on how to do it on archlinux ARM:
I used https://aur.archlinux.org/packages/downgrade
with this ALA: https://alaa.ad24.cz/packages/l/linux-aarch64/
downgrade --ala-url https://alaa.ad24.cz linux-aarch64
(don't have linux-headers installed, only linux-api-headers, which seems to not have the same version as the kernel? idk)

Or just download the pkg.tar.xz file directly and pacman -Qu <pkg.tar.zx it.
downgrade asks if you'd like to add the package to the IgnorePkg list in /etc/pacman.conf so it doesn't get upgraded again.

[ZerBea]

@olivervbk
this is a helpful information for other users running into this trouble.

I started to bisect the Linux kernel, because this "kernel NULL pointer dereference at virtual address 0000000000000000" is a fatal error.
I simple command line causes that my AMD systems become unresponsive.
It is necessary to report this to https://bugzilla.kernel.org/
First I try to identify the faulty commit (git bisecet on git+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git)
If I got the faulty commit, I'll open a Linux Kernel bug report.

[ZerBea]

Just to mention it here: bisecting the Linux kernel is a slow process and it will take a lot of time (even on my AMD Ryzen 7 5700G):

$ git bisect log
git bisect start
# Status: warte auf guten und schlechten Commit
# bad: [933069701c1b507825b514317d4edd5d3fd9d417] Merge tag '6.11-rc-smb3-server-fixes' of git://git.samba.org/ksmbd
git bisect bad 933069701c1b507825b514317d4edd5d3fd9d417
# Status: warte auf gute(n) Commit(s), schlechter Commit bekannt
# good: [e8f897f4afef0031fe618a8e94127a0934896aba] Linux 6.8
git bisect good e8f897f4afef0031fe618a8e94127a0934896aba
# good: [46c6d2b186915176be5acc5d4b6f9793eb32a0c7] Merge tag 'asymmetric-keys-next-6.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd
git bisect good 46c6d2b186915176be5acc5d4b6f9793eb32a0c7
# bad: [99298eb615debd41c1fccff05b163d0a29091904] Merge tag 'm68k-for-v6.11-tag1' of git://git.kernel.org/pub/scm/linux/kernel/git/geert/linux-m68k
git bisect bad 99298eb615debd41c1fccff05b163d0a29091904
# good: [f0bae243b2bcf2b160ae547463bf542762beef8f] Merge tag 'pci-v6.10-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci
git bisect good f0bae243b2bcf2b160ae547463bf542762beef8f
# good: [f1909e859753c9bda87c6d2b82a7f832ef80aa2d] Merge tag 'drm-xe-fixes-2024-06-13' of https://gitlab.freedesktop.org/drm/xe/kernel into drm-fixes
git bisect good f1909e859753c9bda87c6d2b82a7f832ef80aa2d

@olivervbk mentioned that kernel 6.8 is not affected.
bisect confirmed this:
# good: [e8f897f4afef0031fe618a8e94127a0934896aba] Linux 6.8

[ZerBea]

Some good news.
I was able to identify the commit that cause the problem.
The bug report is here.
https://bugzilla.kernel.org/show_bug.cgi?id=219086
Now we have to wait for a fix.

If someone is interested, this is git bisect log:

$ git bisect log
git bisect start
# Status: warte auf guten und schlechten Commit
# bad: [933069701c1b507825b514317d4edd5d3fd9d417] Merge tag '6.11-rc-smb3-server-fixes' of git://git.samba.org/ksmbd
git bisect bad 933069701c1b507825b514317d4edd5d3fd9d417
# Status: warte auf gute(n) Commit(s), schlechter Commit bekannt
# good: [e8f897f4afef0031fe618a8e94127a0934896aba] Linux 6.8
git bisect good e8f897f4afef0031fe618a8e94127a0934896aba
# good: [46c6d2b186915176be5acc5d4b6f9793eb32a0c7] Merge tag 'asymmetric-keys-next-6.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd
git bisect good 46c6d2b186915176be5acc5d4b6f9793eb32a0c7
# bad: [99298eb615debd41c1fccff05b163d0a29091904] Merge tag 'm68k-for-v6.11-tag1' of git://git.kernel.org/pub/scm/linux/kernel/git/geert/linux-m68k
git bisect bad 99298eb615debd41c1fccff05b163d0a29091904
# good: [f0bae243b2bcf2b160ae547463bf542762beef8f] Merge tag 'pci-v6.10-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci
git bisect good f0bae243b2bcf2b160ae547463bf542762beef8f
# good: [f1909e859753c9bda87c6d2b82a7f832ef80aa2d] Merge tag 'drm-xe-fixes-2024-06-13' of https://gitlab.freedesktop.org/drm/xe/kernel into drm-fixes
git bisect good f1909e859753c9bda87c6d2b82a7f832ef80aa2d
# bad: [9b48104b2cde4d392ebf52fda17c3eb5282cddaa] Merge tag 'sound-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect bad 9b48104b2cde4d392ebf52fda17c3eb5282cddaa
# bad: [3c1d29e53d34537063e60f5eafe0482780a1735a] Merge tag 'sound-6.10-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect bad 3c1d29e53d34537063e60f5eafe0482780a1735a
# good: [042742a1ff1f37ca88c303a9e61afc8c56b23657] Merge tag 'sound-6.10-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect good 042742a1ff1f37ca88c303a9e61afc8c56b23657
# bad: [d4ba3313e84dfcdeb92a13434a2d02aad5e973e1] Merge tag 'loongarch-fixes-6.10-2' of git://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson
git bisect bad d4ba3313e84dfcdeb92a13434a2d02aad5e973e1
# bad: [a21b52aa6e98187a82956e296efde5f2521789bb] Merge tag 'phy-fixes-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/phy/linux-phy
git bisect bad a21b52aa6e98187a82956e296efde5f2521789bb
# bad: [2ebe8f840c7450ecbfca9d18ac92e9ce9155e269] tipc: force a dst refcount before doing decryption
git bisect bad 2ebe8f840c7450ecbfca9d18ac92e9ce9155e269
# bad: [c64da10adb57a135bf91e32202d7077931472533] Merge tag 'for-netdev' of https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf
git bisect bad c64da10adb57a135bf91e32202d7077931472533
# bad: [1afe4a64379f65e7bd0c841e6ba7adf312b4c928] Merge tag 'wireless-2024-06-14' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless
git bisect bad 1afe4a64379f65e7bd0c841e6ba7adf312b4c928
# good: [7d09e17c0415fe6d946044c7e70bce31cda952ec] wifi: mac80211: Recalc offload when monitor stop
git bisect good 7d09e17c0415fe6d946044c7e70bce31cda952ec
# good: [89aa3619d141d6cfb6040a561aebb6d99d3e2285] cipso: make cipso_v4_skbuff_delattr() fully remove the CIPSO options
git bisect good 89aa3619d141d6cfb6040a561aebb6d99d3e2285
# bad: [0d9c2beed116e623ac30810d382bd67163650f98] wifi: mac80211: fix monitor channel with chanctx emulation
git bisect bad 0d9c2beed116e623ac30810d382bd67163650f98
# good: [2663d0462eb32ae7c9b035300ab6b1523886c718] wifi: mac80211: Avoid address calculations via out of bounds array indexing
git bisect good 2663d0462eb32ae7c9b035300ab6b1523886c718
# first bad commit: [0d9c2beed116e623ac30810d382bd67163650f98] wifi: mac80211: fix monitor channel with chanctx emulation

[ZerBea]

@morrownr
You're absolutely right. There is massive work on mac80211 and the WiFi driver in progress. I've noticed that, too.
I also noticed a significant increase of issue reports related to mac80211 and WiFi drivers on your git hub presence and mine.

[ZerBea]

We now have a working patch:
https://patchwork.kernel.org/project/linux-wireless/patch/20240725184836.25d334157a8e.I02574086da2c5cf0e18264ce5807db6f14ffd9c0@changeid/

The kernel developers have done a great work!

$ uname -r
6.10.0-1-git-12246-g786c8248dbd3-dirty

$ sudo hcxdumptool -i wlp5s0f4u2 --rcascan=active
...
^C
24 Packet(s) captured by kernel
0 Packet(s) dropped by kernel
14 PROBERESPONSE(s) captured

[morrownr]

@ZerBea

Thank you for your work in identifying, bisecting and reporting several bugs lately.

[ZerBea]

@morrownr
You're welcome.
Unfortunately the regression reached the longterm kernel (6.6.41) before the patch has been available.

$ uname -r
6.6.41-1-lts

$ sudo hcxdumptool --rcascan=active
^C
0 Packet(s) captured by kernel
0 Packet(s) dropped by kernel
Warning: too less packets received (monitor mode may not work as expected)
Possible reasons:
 no transmitter in range
 frames are filtered out by BPF
 driver is broken
Warning: no PROBERESPONSES received (frame injection may not work as expected)
Possible reasons:
 no AP in range
 frames are filtered out by BPF
 driver is broken
 driver does not support frame injection

exit on sigterm

Regarding mt76, last working (monitor mode and frame injection) kernel versions are
longterm 6.6.40
stable 6.8.12

@olivervbk
tshark/airodump-ng/angryoxide don't have issues running.
You can't compare hcxdumptool/hcxlabtool/hcxtools to this tools. They may look the same but that's all.
Wireshark/tshark (my favorite reference tools) and airodump-ng are passive tools. Like all passive tools, they can't detect frame injection failures.

[ZerBea]

BTW:
Regardless of the patch, the ALFA AWUS036AXML is still not working on my AMD RYZEN systems due to a bluetooth bug that has an impact on the WiFi part of the device:
https://bugzilla.kernel.org/show_bug.cgi?id=219040