[5.9] intermittently semi-broken network after boot on RPi4/kernel 5.9

[HiassofT]

Describe the bug

Sometimes I get huge ping times on ethernet, network packets going out very slowly from RPi4 (up to several seconds between tcp packets) and (almost) non-functional NFS after boot.

Most of the time network performance is fine though (about 0.1ms ping time and linux part of NFS boot completed in about 20 seconds).

To reproduce
So far I've only seen the issue when netbooting the RPi4, I wouldn't rule out though that it can happen with SD card boot as well. I usually get the issue in about 1-3 out of 10 boots, sometimes it can need more than 10 reboots though.

Expected behaviour
Network performs normally (low ping, working NFS mounts), like with 5.4 kernel.

Actual behaviour
Kernel level DHCP configuration and root NFS mount sometimes take very long, NFS performance is sometimes so bad that the userspace system fails to start up.

System
4GB RPi4 with B1 stepping, Sep 03 boot eeprom, Sep 11 firmware, kernel 5.9-rc4 githash 4ba2756

Tested with current 32bit RaspiOS (bcm2711_defconfig) and LibreELEC master (custom kernel config).

Logs
Working network, as a reference: http://ix.io/2xzu , ping time is about 0.1ms

One try with semi-working NFS, systemd somewhat came up http://ix.io/2xzw , ping was about 300-500ms:

64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=1 ttl=64 time=0.494 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=2 ttl=64 time=0.442 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=3 ttl=64 time=0.360 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=4 ttl=64 time=0.464 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=5 ttl=64 time=0.425 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=6 ttl=64 time=0.344 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=7 ttl=64 time=0.404 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=8 ttl=64 time=0.345 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=9 ttl=64 time=0.425 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=10 ttl=64 time=0.387 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=11 ttl=64 time=0.330 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=12 ttl=64 time=0.435 ms

Another try, NFS pretty much unusable: http://ix.io/2xzx , ping up to 1024ms

64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=1 ttl=64 time=0.333 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=2 ttl=64 time=0.348 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=3 ttl=64 time=0.267 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=4 ttl=64 time=1024 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=5 ttl=64 time=0.312 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=6 ttl=64 time=1024 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=7 ttl=64 time=1024 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=8 ttl=64 time=0.293 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=9 ttl=64 time=0.334 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=10 ttl=64 time=0.317 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=11 ttl=64 time=0.342 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=12 ttl=64 time=0.336 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=13 ttl=64 time=0.322 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=14 ttl=64 time=1024 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=15 ttl=64 time=0.291 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=16 ttl=64 time=0.337 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=17 ttl=64 time=0.250 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=18 ttl=64 time=0.349 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=19 ttl=64 time=0.321 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=20 ttl=64 time=0.270 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=21 ttl=64 time=0.382 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=22 ttl=64 time=0.305 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=23 ttl=64 time=0.239 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=24 ttl=64 time=0.409 ms

Additional context

In other tests I had seen rather tunny ping times of 1024, 2048, 3072 and 4906ms.

I had seen this issue with the previous rpi-eeprom version as well, but only on kernel 5.9. I used that very often to netboot kernel 5.4 systems (mainly LibreELEC)

[pelwell]

@popcornmix Have you seen any connectivity issues with an NFS root?

[popcornmix]

No I haven't. I'm still using sdcard from bootrom, then nfs for rootfs and that seems okay in my setup.
I can leave a reboot loop going on 5.9 later, but it's not an issue I've seen on numerous boots.
I think @HiassofT has network boot from bootrom.

[HiassofT]

Yes, I'm using TFTP boot from the bootrom here.

So far I haven't seen the issue with SD card boot yet, but due to the intermittent nature I wouldn't dare to say that's related at this point (already got another red herring when looking into this issue)

[HiassofT]

I just caught a not-too-broken-network state where LibreELEC managed to copy the squashfs root to RAM. Network and NFS are slow but I've got a running userspace.

Ping me if you'd like me to do some checks (peeking with miitool etc).

[popcornmix]

I've got a reboot loop running on 5.9-rc5 with nfs booted raspbian (using sdcard for firmware). 50 reboots and counting.
I'll switch it to a LibeELEC build on 5.9 later.

Are you able to test with a different Pi4 or a different switch? Also curious if unplugging/replugging ethernet affects it.
Afraid I don't know muck about miitool diagnostics.

[HiassofT]

Replugging didn't help, tried both plugging it back into the original switch (HP 1810-24G) and into another one (Netgear GS108). dmesg showed the link down/up status change but ping was still ~1000ms

[15437.671394] bcmgenet fd580000.ethernet eth0: Link is Down                    
[15441.726886] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off                                                                      
[15538.041544] bcmgenet fd580000.ethernet eth0: Link is Down                    
[15550.207741] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx                      

hias@camel2:~$ ping rpi4b1
PING rpi4b1.lan (192.168.1.70) 56(84) bytes of data.
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=1 ttl=64 time=1081 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=2 ttl=64 time=47.6 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=3 ttl=64 time=1047 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=4 ttl=64 time=1024 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=5 ttl=64 time=1024 ms
64 bytes from rpi4b1.lan (192.168.1.70): icmp_seq=6 ttl=64 time=256 ms

I'll do test with another RPi4 and boot-tests with the Netgear switch a bit later.

[popcornmix]

I assume 5.4 has always been reliable for booting? And 5.9 always unreliable?
Had you tested any kernels in between?

[HiassofT]

yes, 5.4 has always been fine. 5.9 has these intermittent issues (i.e. mostly fine). haven't tested any kernels in between yet

[popcornmix]

Can you try adding genet.skip_umac_reset=1 to cmdline.txt?
That seems to be set differently between 5.4 and 5.9.

[HiassofT]

I checked another RPi4 (2GB) model with the Netgear switch and got slow net on the second reboot.

Hardware        : BCM2711
Revision        : b03111
Serial          : 100000006a339d6f
Model           : Raspberry Pi 4 Model B Rev 1.1

Sep 3 bootloader, only change is BOOT_UART=1 and BOOT_ORDER=0xf241, Sep 16 fw and 5.9.0-rc5

will give umac_reset a try now.

[HiassofT]

with genet.skip_umac_reset=1 dmesg is spammed with "eth0: hw csum failure" errors, dumps and backtraces http://ix.io/2xT0

[    7.222757] Sending DHCP requests .
[    7.230477] eth0: hw csum failure
[    7.237285] skb len=340 headroom=144 headlen=340 tailroom=1692
[    7.237285] mac=(130,14) net=(144,20) trans=164
[    7.237285] shinfo(txflags=0 nr_frags=0 gso(size=0 type=0 segs=0))
[    7.237285] csum(0x78fc ip_summed=2 complete_sw=0 valid=0 level=0)
[    7.237285] hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=0
[    7.265611] dev name=eth0 feat=0x0x0000010000004829
[    7.270501] skb headroom: 00000000: 2f 75 73 72 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d
[    7.278171] skb headroom: 00000010: 6f 76 65 72 6c 61 79 73 2f 62 61 73 65 2f 6c 69
[    7.285841] skb headroom: 00000020: 62 2f 66 69 72 6d 77 61 72 65 2f 62 72 63 6d 2f
[    7.293499] skb headroom: 00000030: 42 43 4d 32 30 37 30 32 41 30 2d 30 61 35 63 2d
[    7.301152] skb headroom: 00000040: 80 7f a8 01 80 80 00 00 78 fc 00 00 00 00 00 00
[    7.308802] skb headroom: 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.316455] skb headroom: 00000060: fb 49 f7 0e 81 bf d4 02 bf d7 5b fe 24 7e 31 7e
[    7.324108] skb headroom: 00000070: 07 f1 03 7f bd b7 78 3e fa 8d 3e 7e 5f 6d df 8f
[    7.331762] skb headroom: 00000080: 00 00 dc a6 32 0a 84 2f c0 4a 00 2d 38 94 08 00
[    7.339416] skb linear:   00000000: 45 c0 01 54 9e 73 00 00 40 11 55 d7 c0 a8 01 fa
[    7.347069] skb linear:   00000010: c0 a8 01 44 00 43 00 44 01 40 b3 a3 02 01 06 00
[    7.354721] skb linear:   00000020: 99 dd e9 02 00 00 00 00 00 00 00 00 c0 a8 01 44
[    7.362374] skb linear:   00000030: c0 a8 01 fa 00 00 00 00 dc a6 32 0a 84 2f 00 00
[    7.370027] skb linear:   00000040: 00 00 00 00 00 00 00 00 31 39 32 2e 31 36 38 2e
[    7.377681] skb linear:   00000050: 31 2e 32 30 00 00 00 00 00 00 00 00 00 00 00 00
[    7.385335] skb linear:   00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.392989] skb linear:   00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.400642] skb linear:   00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.408291] skb linear:   00000090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.415944] skb linear:   000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.423598] skb linear:   000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.431253] skb linear:   000000c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.438904] skb linear:   000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.446560] skb linear:   000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.454214] skb linear:   000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.461868] skb linear:   00000100: 00 00 00 00 00 00 00 00 63 82 53 63 35 01 02 36
[    7.469524] skb linear:   00000110: 04 c0 a8 01 fa 33 04 00 00 a8 c0 3a 04 00 00 54
[    7.477178] skb linear:   00000120: 60 3b 04 00 00 93 a8 01 04 ff ff ff 00 1c 04 c0
[    7.484830] skb linear:   00000130: a8 01 ff 03 04 c0 a8 01 fa 0f 03 6c 61 6e 0c 07
[    7.492483] skb linear:   00000140: 72 70 69 34 2d 32 67 2a 04 c0 a8 01 fa 06 04 c0
[    7.500137] skb linear:   00000150: a8 01 fa ff
[    7.504666] skb tailroom: 00000000: 16 f8 6b 61 00 00 00 00 00 00 00 00 00 00 00 00
[    7.512317] skb tailroom: 00000010: 00 00 00 00 02 00 00 00 00 00 00 00 2f 75 73 72
[    7.519971] skb tailroom: 00000020: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    7.527635] skb tailroom: 00000030: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    7.535304] skb tailroom: 00000040: 72 6d 77 61 72 65 2f 62 34 33 2f 75 63 6f 64 65
[    7.542974] skb tailroom: 00000050: 32 31 5f 73 73 6c 70 6e 5f 6e 6f 62 74 2e 66 77
[    7.550633] skb tailroom: 00000060: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    7.558287] skb tailroom: 00000070: 37 32 30 78 35 37 36 00 00 00 00 00 00 00 00 00
[    7.565940] skb tailroom: 00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.573594] skb tailroom: 00000090: 00 00 00 00 02 00 00 00 00 00 00 00 2f 75 73 72
[    7.581246] skb tailroom: 000000a0: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    7.588898] skb tailroom: 000000b0: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    7.596551] skb tailroom: 000000c0: 72 6d 77 61 72 65 2f 62 34 33 2f 75 63 6f 64 65
[    7.604204] skb tailroom: 000000d0: 32 30 5f 73 73 6c 70 6e 5f 6e 6f 62 74 2e 66 77
[    7.611857] skb tailroom: 000000e0: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    7.619511] skb tailroom: 000000f0: 31 34 34 30 78 35 37 36 00 00 00 00 00 00 00 00
[    7.627165] skb tailroom: 00000100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.634819] skb tailroom: 00000110: 00 00 00 00 02 00 00 00 00 00 00 00 2f 75 73 72
[    7.642473] skb tailroom: 00000120: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    7.650127] skb tailroom: 00000130: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    7.657776] skb tailroom: 00000140: 72 6d 77 61 72 65 2f 62 34 33 2f 75 63 6f 64 65
[    7.665429] skb tailroom: 00000150: 31 39 5f 73 73 6c 70 6e 5f 6e 6f 62 74 2e 66 77
[    7.673084] skb tailroom: 00000160: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    7.680739] skb tailroom: 00000170: 37 32 30 78 35 37 36 69 00 00 00 00 00 00 00 00
[    7.688390] skb tailroom: 00000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.696042] skb tailroom: 00000190: 00 00 00 00 02 00 00 00 00 00 00 00 2f 75 73 72
[    7.703693] skb tailroom: 000001a0: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    7.711348] skb tailroom: 000001b0: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    7.719012] skb tailroom: 000001c0: 72 6d 77 61 72 65 2f 62 34 33 2f 75 63 6f 64 65
[    7.726682] skb tailroom: 000001d0: 31 36 5f 73 73 6c 70 6e 5f 6e 6f 62 74 2e 66 77
[    7.734351] skb tailroom: 000001e0: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    7.742020] skb tailroom: 000001f0: 36 34 30 78 34 38 30 00 00 00 00 00 00 00 00 00
[    7.749678] skb tailroom: 00000200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.757331] skb tailroom: 00000210: 00 00 00 00 01 00 00 00 00 00 00 00 2f 75 73 72
[    7.764981] skb tailroom: 00000220: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    7.772630] skb tailroom: 00000230: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    7.780283] skb tailroom: 00000240: 72 6d 77 61 72 65 2f 62 34 33 2f 73 73 6c 70 6e
[    7.787947] skb tailroom: 00000250: 34 69 6e 69 74 76 61 6c 73 32 32 2e 66 77 00 d0
[    7.795616] skb tailroom: 00000260: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    7.803285] skb tailroom: 00000270: 31 39 32 30 78 31 30 38 30 00 00 00 00 00 00 00
[    7.810944] skb tailroom: 00000280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.818598] skb tailroom: 00000290: 00 00 00 00 02 00 00 00 00 00 00 00 2f 75 73 72
[    7.826252] skb tailroom: 000002a0: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    7.833905] skb tailroom: 000002b0: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    7.841558] skb tailroom: 000002c0: 72 6d 77 61 72 65 2f 62 34 33 2f 73 73 6c 70 6e
[    7.849207] skb tailroom: 000002d0: 34 62 73 69 6e 69 74 76 61 6c 73 32 32 2e 66 77
[    7.856859] skb tailroom: 000002e0: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    7.864512] skb tailroom: 000002f0: 31 32 38 30 78 37 32 30 00 00 00 00 00 00 00 00
[    7.872167] skb tailroom: 00000300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.879817] skb tailroom: 00000310: 00 00 00 00 02 00 00 00 00 00 00 00 2f 75 73 72
[    7.887468] skb tailroom: 00000320: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    7.895121] skb tailroom: 00000330: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    7.902786] skb tailroom: 00000340: 72 6d 77 61 72 65 2f 62 34 33 2f 73 73 6c 70 6e
[    7.910455] skb tailroom: 00000350: 33 69 6e 69 74 76 61 6c 73 32 31 2e 66 77 00 d0
[    7.918116] skb tailroom: 00000360: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    7.925785] skb tailroom: 00000370: 31 39 32 30 78 31 30 38 30 69 00 00 00 00 00 00
[    7.933445] skb tailroom: 00000380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    7.941100] skb tailroom: 00000390: 00 00 00 00 02 00 00 00 00 00 00 00 2f 75 73 72
[    7.948751] skb tailroom: 000003a0: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    7.956402] skb tailroom: 000003b0: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    7.964053] skb tailroom: 000003c0: 72 6d 77 61 72 65 2f 62 34 33 2f 73 73 6c 70 6e
[    7.971709] skb tailroom: 000003d0: 33 62 73 69 6e 69 74 76 61 6c 73 32 31 2e 66 77
[    7.979373] skb tailroom: 000003e0: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    7.987042] skb tailroom: 000003f0: 37 32 30 78 34 38 30 69 00 00 00 00 00 00 00 00
[    7.994711] skb tailroom: 00000400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    8.002370] skb tailroom: 00000410: 00 00 00 00 02 00 00 00 00 00 00 00 2f 75 73 72
[    8.010023] skb tailroom: 00000420: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    8.017672] skb tailroom: 00000430: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    8.025325] skb tailroom: 00000440: 72 6d 77 61 72 65 2f 62 34 33 2f 73 73 6c 70 6e
[    8.032980] skb tailroom: 00000450: 32 69 6e 69 74 76 61 6c 73 31 39 2e 66 77 00 d0
[    8.040635] skb tailroom: 00000460: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    8.048286] skb tailroom: 00000470: 37 32 30 78 34 38 30 00 00 00 00 00 00 00 00 00
[    8.055937] skb tailroom: 00000480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    8.063589] skb tailroom: 00000490: 00 00 00 00 02 00 00 00 00 00 00 00 2f 75 73 72
[    8.071244] skb tailroom: 000004a0: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    8.078908] skb tailroom: 000004b0: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    8.086577] skb tailroom: 000004c0: 72 6d 77 61 72 65 2f 62 34 33 2f 73 73 6c 70 6e
[    8.094246] skb tailroom: 000004d0: 32 62 73 69 6e 69 74 76 61 6c 73 31 39 2e 66 77
[    8.101905] skb tailroom: 000004e0: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    8.109558] skb tailroom: 000004f0: 31 34 34 30 78 34 38 30 00 00 00 00 00 00 00 00
[    8.117208] skb tailroom: 00000500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    8.124861] skb tailroom: 00000510: 00 00 00 00 01 00 00 00 00 00 00 00 2f 75 73 72
[    8.132516] skb tailroom: 00000520: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    8.140171] skb tailroom: 00000530: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    8.147822] skb tailroom: 00000540: 72 6d 77 61 72 65 2f 62 34 33 2f 73 73 6c 70 6e
[    8.155473] skb tailroom: 00000550: 31 69 6e 69 74 76 61 6c 73 32 37 2e 66 77 00 d0
[    8.163128] skb tailroom: 00000560: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    8.170793] skb tailroom: 00000570: 37 32 30 78 34 38 30 00 00 00 00 00 00 00 00 00
[    8.178461] skb tailroom: 00000580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    8.186131] skb tailroom: 00000590: 00 00 00 00 01 00 00 00 00 00 00 00 2f 75 73 72
[    8.193788] skb tailroom: 000005a0: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    8.201441] skb tailroom: 000005b0: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    8.209090] skb tailroom: 000005c0: 72 6d 77 61 72 65 2f 62 34 33 2f 73 73 6c 70 6e
[    8.216745] skb tailroom: 000005d0: 31 69 6e 69 74 76 61 6c 73 32 30 2e 66 77 00 d0
[    8.224401] skb tailroom: 000005e0: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    8.232055] skb tailroom: 000005f0: 31 39 32 30 78 31 30 38 30 69 00 00 00 00 00 00
[    8.239705] skb tailroom: 00000600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    8.247356] skb tailroom: 00000610: 00 00 00 00 02 00 00 00 00 00 00 00 2f 75 73 72
[    8.255010] skb tailroom: 00000620: 2f 6c 69 62 2f 6b 65 72 6e 65 6c 2d 6f 76 65 72
[    8.262674] skb tailroom: 00000630: 6c 61 79 73 2f 62 61 73 65 2f 6c 69 62 2f 66 69
[    8.270343] skb tailroom: 00000640: 72 6d 77 61 72 65 2f 62 34 33 2f 73 73 6c 70 6e
[    8.278005] skb tailroom: 00000650: 31 62 73 69 6e 69 74 76 61 6c 73 32 37 2e 66 77
[    8.285674] skb tailroom: 00000660: 00 01 00 00 22 01 00 00 00 00 00 00 00 00 00 00
[    8.293334] skb tailroom: 00000670: 31 39 32 30 78 31 30 38 30 00 00 00 00 00 00 00
[    8.300989] skb tailroom: 00000680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[    8.308640] skb tailroom: 00000690: 00 00 00 00 02 00 00 00 00 00 00 00
[    8.315253] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.9.0-rc5 #1
[    8.321425] Hardware name: BCM2711
[    8.324818] Backtrace: 
[    8.327267] [<c020d8c0>] (dump_backtrace) from [<c020dc34>] (show_stack+0x20/0x24)
[    8.334830]  r7:ffffffff r6:00000000 r5:60000113 r4:c18862a8
[    8.340488] [<c020dc14>] (show_stack) from [<c074c768>] (dump_stack+0xc4/0xf0)
[    8.347709] [<c074c6a4>] (dump_stack) from [<c0affed0>] (netdev_rx_csum_fault.part.0+0x50/0x54)
[    8.356400]  r9:00009e73 r8:c0e97d50 r7:ef3d0000 r6:00000000 r5:4418db61 r4:ef126000
[    8.364139] [<c0affe80>] (netdev_rx_csum_fault.part.0) from [<c0af2f44>] (__skb_gro_checksum_complete+0xa0/0xa4)
[    8.374305]  r5:4418db61 r4:ef126000
[    8.377878] [<c0af2ea4>] (__skb_gro_checksum_complete) from [<c0ba9e40>] (udp4_gro_receive+0x250/0x2c4)
[    8.387263]  r7:00000090 r6:ef3d3efc r5:d0d1f724 r4:ef126000
[    8.392917] [<c0ba9bf0>] (udp4_gro_receive) from [<c0bb393c>] (inet_gro_receive+0x298/0x2f8)
[    8.401348]  r7:00000090 r6:ef3d3efc r5:00000000 r4:ef126000
[    8.407001] [<c0bb36a4>] (inet_gro_receive) from [<c0afbf44>] (dev_gro_receive+0x2f0/0x5b4)
[    8.415346]  r10:00000000 r9:00000000 r8:00000000 r7:ef3d3ed8 r6:c1806648 r5:ef126000
[    8.423167]  r4:c1807b90
[    8.425695] [<c0afbc54>] (dev_gro_receive) from [<c0afcdc8>] (napi_gro_receive+0x44/0x1e4)
[    8.433953]  r10:ef3d0000 r9:ef3d0580 r8:ef3d3580 r7:00000162 r6:ef3d3ed8 r5:ef126000
[    8.441774]  r4:ef126000
[    8.444303] [<c0afcd84>] (napi_gro_receive) from [<c093b884>] (bcmgenet_rx_poll+0x1d0/0x51c)
[    8.452733]  r7:00000162 r6:01a87f80 r5:ef126000 r4:ef3d3ed8
[    8.458386] [<c093b6b4>] (bcmgenet_rx_poll) from [<c0afca1c>] (net_rx_action+0x1b0/0x518)
[    8.466556]  r10:0000012c r9:00000001 r8:c1800000 r7:00000040 r6:2e843000 r5:00000000
[    8.474377]  r4:ef3d3ed8
[    8.476906] [<c0afc86c>] (net_rx_action) from [<c0201518>] (__do_softirq+0x188/0x468)
[    8.484729]  r10:00000000 r9:00000008 r8:c1800000 r7:00000100 r6:00000003 r5:00000004
[    8.492550]  r4:c180308c
[    8.495079] [<c0201390>] (__do_softirq) from [<c022986c>] (irq_exit+0xd0/0x128)
[    8.502380]  r10:00000000 r9:c1800000 r8:ef810800 r7:00000001 r6:00000000 r5:00000000
[    8.510201]  r4:ffffe000
[    8.512730] [<c022979c>] (irq_exit) from [<c0279010>] (__handle_domain_irq+0x90/0xe0)
[    8.520551]  r5:00000000 r4:c168eec0
[    8.524120] [<c0278f80>] (__handle_domain_irq) from [<c0201354>] (gic_handle_irq+0x50/0x8c)
[    8.532463]  r9:c1800000 r8:c1801eb0 r7:f0815000 r6:f0814000 r5:f081400c r4:c18059cc
[    8.540200] [<c0201304>] (gic_handle_irq) from [<c0200abc>] (__irq_svc+0x5c/0x7c)
[    8.547674] Exception stack(0xc1801eb0 to 0xc1801ef8)
[    8.552718] 1ea0:                                     00000000 2e843000 60000093 60000093
[    8.560889] 1ec0: c1800000 00000000 c1804f74 c1804fb4 00000000 c168f268 00000000 c1801f0c
[    8.569059] 1ee0: c1805344 c1801f00 c1800000 c0209428 60000013 ffffffff
[    8.575666]  r9:c1800000 r8:00000000 r7:c1801ee4 r6:ffffffff r5:60000013 r4:c0209428
[    8.583407] [<c02093f4>] (arch_cpu_idle) from [<c0c90ef0>] (default_idle_call+0x3c/0x134)
[    8.591578] [<c0c90eb4>] (default_idle_call) from [<c02593d0>] (do_idle+0x270/0x2cc)
[    8.599313]  r5:00000000 r4:c1800000
[    8.602882] [<c0259160>] (do_idle) from [<c025972c>] (cpu_startup_entry+0x28/0x2c)
[    8.610444]  r10:c1250f40 r9:000001ca r8:effff800 r7:c1804f40 r6:00000000 r5:c1250f40
[    8.618265]  r4:000000d7
[    8.620794] [<c0259704>] (cpu_startup_entry) from [<c0c8a48c>] (rest_init+0xb4/0xbc)
[    8.628534] [<c0c8a3d8>] (rest_init) from [<c1200cc4>] (arch_call_rest_init+0x18/0x1c)
[    8.636442]  r5:c1250f40 r4:c18ea078
[    8.640013] [<c1200cac>] (arch_call_rest_init) from [<c1201520>] (start_kernel+0x7e0/0x820)
[    8.648358] [<c1200d40>] (start_kernel) from [<00000000>] (0x0)
[    8.654271]  r10:30c5387d r9:410fd083 r8:2eff3f00 r7:00000c42 r6:30c0387d r5:00000000
[    8.662092]  r4:c1200330
[    8.664716] eth0: hw csum failure
...

[popcornmix]

Hmm, this is beyond my knowledge.
drivers/net/ethernet/broadcom/genet/bcmgenet.c has changed significantly upstream for 5.9.
It may be that cf55f6c (*) is now skipping something essential.
@pelwell may know more.

(*) that was enabled by default on 5.4 and doesn't cause the csum spam. It is present on 5.9 but disabled by default.

[HiassofT]

I checked the 5.6 kernel branch on RPiOS and that showed the same behaviour as 5.9 - intermittently slow network and splats with genet.skip_umac_reset=1

I didn't have much luck with 5.5, that wouldn't mount the nfsroot. kernel dhcp config succeeded but nfs mount failed and kernel panicked

[    8.121449] Sending DHCP requests ., OK
[    8.168350] IP-Config: Got DHCP answer from 192.168.1.250, my address is 192.168.1.68
[    8.179200] IP-Config: Complete:
[    8.185448]      device=eth0, hwaddr=dc:a6:32:0a:84:2f, ipaddr=192.168.1.68, mask=255.255.255.0, gw=192.168.1.250
[    8.198807]      host=rpi4-2g, domain=lan, nis-domain=(none)
[    8.207554]      bootserver=192.168.1.250, rootserver=192.168.1.20, rootpath=
[    8.207561]      nameserver0=192.168.1.250
[    8.225018]      ntpserver0=192.168.1.250
[    8.233833] uart-pl011 fe201000.serial: no DMA platform data
[   38.871478] vcc-sd: disabling
[  104.151553] VFS: Unable to mount root fs via NFS, trying floppy.
[  104.161141] VFS: Cannot open root device "nfs" or unknown-block(2,0): error -6

I also checked with 5.4 kernel (on LibreELEC) before and that survived about 20 reboots without any issues.

I did all these tests on the same setup (2GB RPi4, Sep 16 firmware, Netgear switch).

[popcornmix]

Libreelec with 5.9 kernel and sdcard boot doing ping -c 1 192.168.4.1 && reboot in /storage/.config.autostart.sh seems stable for me - running for some dozens of reboots.

I suspect there's something pi4/switch specific, or bootrom netboot dependent that is causing this.

[HiassofT]

I think I managed to narrow it down: TFTP boot from bootloader in combination with kernel 5.6 or newer seem to be the key points to trigger the issue.

The nfsroot mount issue with kernel 5.5 seems to have been caused by the rw option in nfsroot. No idea why I had it in there, with that removed raspios survived 20 reboots with TFTP/NFS without any issue.

[    0.000000] Linux version 5.5.19-v7l+ (hias@delle) (gcc version 8.3.0 (Debian 8.3.0-2)) #1 SMP Thu Sep 17 21:21:39 CEST 2020

raspios with kernel 5.6 and TFTP/NFS boot failed at the 3rd reboot and in a later test even at the very first boot

[    0.000000] Linux version 5.6.19-v7l+ (hias@delle) (gcc version 8.3.0 (Debian 8.3.0-2)) #3 SMP Fri Sep 18 12:43:45 CEST 2020

However, with the same 5.6 kernel and firmware on SD card and only root on NFS it survived 50 reboots.

I also checked kernel 5.9.0-rc4, that survived 20 SD/NFS reboots as well.