zebra: kernel: fib_nh_match warnings

[k0ste]

Kernel: Linux linux01 5.6.10-arch1-1-nfcustom #1 SMP PREEMPT Wed, 06 May 2020 04:26:51 +0000 x86_64 GNU/Linux
Zebra:

zebra version 7.3.1
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
	'--prefix=/usr' '--sbindir=/usr/bin' '--sysconfdir=/etc/frr' '--localstatedir=/run/frr' '--enable-exampledir=/usr/share/doc/frr/examples' '--with-libpam' '--enable-snmp=agentx' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' '--enable-shell-access' '--enable-systemd' '--enable-rpki' '--enable-fpm'

systemd:

systemd 245 (245.5-2-arch)
+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

How to reproduce:

1. Setup systemd-networkd:

/etc/systemd/network/dummy0.netdev:

[NetDev]
Name=dummy0
Kind=dummy

/etc/systemd/network/dummy0.network:

[Match]
Name=dummy0

[Network]
Address=100.100.100.1/24

[RoutingPolicyRule]
From=100.100.100.1
Table=100
Priority=100

[Route]
Gateway=100.100.100.254
Destination=0.0.0.0/0
Table=100

2. frr daemons:

ospfd=yes
ldpd=yes
nhrpd=yes
pbrd=yes
bfdd=yes
vrrpd=yes
zebra=yes
ospfd_instances="1,2"
vtysh_enable=yes

3. frr conf:

frr version 7.3.1
frr defaults traditional
hostname linux01
log syslog informational
no ip forwarding
no ipv6 forwarding
!
ip route 0.0.0.0/0 100.100.100.254 5
!
line vty
!

4. systemctl reenable frr systemd-networkd

5. systemctl reboot

6. check ip routes & rules:

[root@linux01 /]# ip rule show
0:	from all lookup local
100:	from 100.100.100.1 lookup networkd
32766:	from all lookup main
32767:	from all lookup default

[root@linux01 /]# ip route show table 100
default via 100.100.100.254 dev dummy0 proto static 

7. systemctl restart systemd-networkd

8. Result:

[Fri May 15 04:05:25 2020] ------------[ cut here ]------------
[Fri May 15 04:05:25 2020] WARNING: CPU: 1 PID: 510 at include/net/nexthop.h:251 fib_nh_match+0x210/0x400
[Fri May 15 04:05:25 2020] Modules linked in: intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev mousedev aesni_intel crypto_simd dummy cryptd glue_helper ttm drm_kms_helper cec psmouse intel_agp intel_rapl_perf rc_core syscopyarea sysfillrect i2c_piix4 sysimgblt pcspkr battery e1000 fb_sys_fops vboxguest intel_gtt input_leds ac evdev mac_hid drm agpgart ip_tables x_tables xfs libcrc32c crc32c_generic sr_mod cdrom ata_generic pata_acpi ohci_pci serio_raw atkbd libps2 crc32c_intel ehci_pci ehci_hcd ohci_hcd i8042 ata_piix serio
[Fri May 15 04:05:25 2020] CPU: 1 PID: 510 Comm: systemd-network Not tainted 5.6.10-arch1-1-nfcustom #1
[Fri May 15 04:05:25 2020] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[Fri May 15 04:05:25 2020] RIP: 0010:fib_nh_match+0x210/0x400
[Fri May 15 04:05:25 2020] Code: 7c 24 18 48 8b b5 90 00 00 00 e8 cb 97 f6 ff 48 8b 7c 24 18 41 89 c4 e8 7e 93 f6 ff 45 85 e4 0f 85 43 fe ff ff e9 66 ff ff ff <0f> 0b e9 2e ff ff ff 3c 0a 0f 85 72 fe ff ff 48 8b 8d 98 00 00 00
[Fri May 15 04:05:25 2020] RSP: 0018:ffffa5d00064ba68 EFLAGS: 00010282
[Fri May 15 04:05:25 2020] RAX: 0000000000000012 RBX: ffffa5d00064bbc8 RCX: 0000000000000000
[Fri May 15 04:05:25 2020] RDX: ffffa5d00064bce8 RSI: ffff88a25a02be80 RDI: ffffa5d00064bb08
[Fri May 15 04:05:25 2020] RBP: ffff88a25a02be80 R08: 00000000000000fe R09: ffffa5d00064bce8
[Fri May 15 04:05:25 2020] R10: 0000000000000000 R11: ffff88a2586f0d40 R12: ffffa5d00064ba88
[Fri May 15 04:05:25 2020] R13: 0000000000000000 R14: ffffa5d00064bbc8 R15: ffff88a2582f2ec0
[Fri May 15 04:05:25 2020] FS:  00007f0439e7aa80(0000) GS:ffff88a25bd00000(0000) knlGS:0000000000000000
[Fri May 15 04:05:25 2020] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri May 15 04:05:25 2020] CR2: 00005565ebb83048 CR3: 000000011ab8a004 CR4: 00000000000606e0
[Fri May 15 04:05:25 2020] Call Trace:
[Fri May 15 04:05:25 2020]  ? memcg_check_events+0x40/0x220
[Fri May 15 04:05:25 2020]  fib_table_delete+0x1a7/0x310
[Fri May 15 04:05:25 2020]  inet_rtm_delroute+0x96/0x100
[Fri May 15 04:05:25 2020]  rtnetlink_rcv_msg+0x137/0x3c0
[Fri May 15 04:05:25 2020]  ? rtnl_calcit.isra.0+0x120/0x120
[Fri May 15 04:05:25 2020]  netlink_rcv_skb+0x78/0x150
[Fri May 15 04:05:25 2020]  netlink_unicast+0x19c/0x240
[Fri May 15 04:05:25 2020]  netlink_sendmsg+0x243/0x480
[Fri May 15 04:05:25 2020]  sock_sendmsg+0x5e/0x60
[Fri May 15 04:05:25 2020]  __sys_sendto+0x120/0x190
[Fri May 15 04:05:25 2020]  __x64_sys_sendto+0x25/0x30
[Fri May 15 04:05:25 2020]  do_syscall_64+0x4e/0x150
[Fri May 15 04:05:25 2020]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Fri May 15 04:05:25 2020] RIP: 0033:0x7f043ade392a
[Fri May 15 04:05:25 2020] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 76 c3 0f 1f 44 00 00 55 48 83 ec 30 44 89 4c
[Fri May 15 04:05:25 2020] RSP: 002b:00007ffe695e6018 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[Fri May 15 04:05:25 2020] RAX: ffffffffffffffda RBX: 00007ffe695e609c RCX: 00007f043ade392a
[Fri May 15 04:05:25 2020] RDX: 0000000000000034 RSI: 0000555632aab630 RDI: 0000000000000003
[Fri May 15 04:05:25 2020] RBP: 0000555632a85820 R08: 00007ffe695e6020 R09: 0000000000000010
[Fri May 15 04:05:25 2020] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555632aceb50
[Fri May 15 04:05:25 2020] R13: 00000000000001fe R14: 0000555632aabf20 R15: 0000555631c5ac98
[Fri May 15 04:05:25 2020] ---[ end trace bbe9130f8bdf79fb ]---

[k0ste]

@sworleys continue #6089

[k0ste]

Reported this?🤔

[sworleys]

@k0ste sorry just havent had the cycles to stab at it yet.

Will try to get to it sooner rather than later.

In the meantime you should be able to work around by:

no zebra nexthop kernel enable

[k0ste]

Any news?

[majst01]

Can confirm the same kernel BUG with frr-7.5 and kernel 5.4.0-54-generic #60-Ubuntu

But does not happen with frr-7.2

[62150.785942] ------------[ cut here ]------------                                                                                                                                                                
[62150.785956] WARNING: CPU: 15 PID: 2023722 at include/net/nexthop.h:283 fib_nh_match+0x206/0x430                                                                                                                 
[62150.785958] Modules linked in: ip6table_nat ip6_tables xt_statistic xt_set ipt_rpfilter xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel xt_nat ipt_REJECT nf_reject_ipv4 x
t_tcpudp ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_co
nntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c bpfilter br_netfilter bridge stp llc aufs overlay ipmi_ssif intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit x86_pkg_temp_thermal intel_powerclamp cor
etemp kvm_intel nls_iso8859_1 kvm crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd ast cryptd glue_helper drm_vram_helper rapl ttm intel_cstate joydev input_leds drm_kms_helper fb_sys_fops syscopyare
a sysfillrect sysimgblt mei_me mei ioatdma ipmi_si acpi_power_meter acpi_pad mac_hid sch_fq_codel ipmi_watchdog ipmi_devintf ipmi_msghandler drm binfmt_misc                                                       
[62150.786045]  ip_tables x_tables autofs4 hid_generic usbhid hid nvme igb i2c_algo_bit crc32_pclmul nvme_core i40e dca i2c_i801 ahci lpc_ich libahci wmi                                                          
[62150.786070] CPU: 15 PID: 2023722 Comm: calico-node Tainted: G        W         5.4.0-54-generic #60-Ubuntu                                                                                                      
[62150.786073] Hardware name: Supermicro SYS-2029BT-HNR/X11DPT-B, BIOS 3.3 02/22/2020                                                                                                                              
[62150.786082] RIP: 0010:fib_nh_match+0x206/0x430                                                                                                                                                                  
[62150.786087] Code: 49 8b b4 24 90 00 00 00 e8 17 7e f7 ff 48 8b bd 48 ff ff ff 41 89 c5 e8 d8 7b f7 ff 45 85 ed 0f 85 3c fe ff ff e9 59 ff ff ff <0f> 0b e9 28 ff ff ff 3c 0a 0f 85 68 fe ff ff 49 8b 8c 24 98 00
 00                                                                                                      
[62150.786091] RSP: 0018:ffffb7634a987a20 EFLAGS: 00010282                      
[62150.786096] RAX: 000000000000001e RBX: ffffb7634a987b80 RCX: 0000000000000000           
[62150.786099] RDX: ffffb7634a987cb0 RSI: ffff9bedc79bdc00 RDI: ffffb7634a987b80           
[62150.786103] RBP: ffffb7634a987af0 R08: 00000000000000fe R09: ffffb7634a987cb0           
[62150.786105] R10: 0000000000000000 R11: 00000000d422530e R12: ffff9bedc79bdc00           
[62150.786109] R13: 0000000000000000 R14: 0000000000000000 R15: ffff9bedcc12e5a0           
[62150.786113] FS:  00007fec537fe700(0000) GS:ffff9bf9cf9c0000(0000) knlGS:0000000000000000
[62150.786117] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033                
[62150.786120] CR2: 000000c0008d1000 CR3: 0000000c075e4005 CR4: 00000000007606e0
[62150.786123] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[62150.786127] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[62150.786129] PKRU: 55555554                   
[62150.786131] Call Trace:                      
[62150.786144]  fib_table_delete+0x19f/0x3c0    
[62150.786153]  inet_rtm_delroute+0x92/0x100    
[62150.786163]  rtnetlink_rcv_msg+0x2c0/0x380   
[62150.786171]  ? rtnl_calcit.isra.0+0x100/0x100
[62150.786177]  netlink_rcv_skb+0x50/0x120 
[62150.786184]  rtnetlink_rcv+0x15/0x20    
[62150.786189]  netlink_unicast+0x187/0x220  
[62150.786196]  netlink_sendmsg+0x222/0x3e0  
[62150.786205]  sock_sendmsg+0x65/0x70       
[62150.786211]  __sys_sendto+0x113/0x190     
[62150.786223]  ? do_futex+0x160/0x1e0       
[62150.786230]  ? __x64_sys_futex+0x13f/0x170
[62150.786239]  __x64_sys_sendto+0x29/0x30
[62150.786247]  do_syscall_64+0x57/0x190
[62150.786256]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

[napaster]

Hey. on 5.19 the same bug.

Kernel version

Linux gw.ru 5.6.19-1-nfcustom # 1 SMP PREEMPT Thu, 10 Sep 2020 13:01:18 +0000 x86_64 GNU / Linux

Systemd version

Version         : 246.6-1

FRR version

Version         : 7.4-2

Error

------------[ cut here ]------------
WARNING: CPU: 0 PID: 311 at include/net/nexthop.h:283 fib_nh_match+0x1fc/0x3d0
Modules linked in: nfsd auth_rpcgss nfsv4 dns_resolver nfs_acl nfs lockd grace sunrpc fscache raid1 md_mod sctp hid_generic usbhid hid xt_nat xt_addrtype xt_recent xt_state xt_conntrack xt_set ipt_REJECT xt_multiport iptable_filter xt_TCPMSS xt_tcpudp iptable_mangle iptable_nat tun intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm 8021q garp mrp irqbypass stp llc crct10dif_pclmul iTCO_wdt ipmi_ssif crc32_pclmul 8250_dw iTCO_vendor_support ghash_clmulni_intel bonding dcdbas aesni_intel crypto_simd cryptd glue_helper intel_cstate intel_uncore ofpart intel_rapl_perf cmdlinepart drm_vram_helper drm_ttm_helper i2c_algo_bit intel_spi_pci intel_spi ttm pcspkr ip_set_hash_net drm_kms_helper spi_nor mtd cec i2c_i801 rc_core tg3 intel_lpss_pci syscopyarea intel_lpss sysfillrect idma64 sysimgblt libphy fb_sys_fops ip_set mei_me mei uas usb_storage intel_pch_thermal ie31200_edac ipmi_si ipmi_devintf ipmi_msghandler evdev ac mac_hid
 acpi_power_meter nf_conntrack_pptp nf_reject_ipv4 nf_nat nf_conntrack_netlink nfnetlink nf_conntrack nf_defrag_ipv6 drm nf_defrag_ipv4 sch_htb agpgart ip_tables x_tables xfs libcrc32c crc32c_generic crc32c_intel xhci_pci sr_mod xhci_hcd cdrom megaraid_sas
CPU: 0 PID: 311 Comm: systemd-network Tainted: G        W         5.6.19-1-nfcustom #1
Hardware name: Dell Inc. PowerEdge R340/045M96, BIOS 2.3.5 09/27/2018
RIP: 0010:fib_nh_match+0x1fc/0x3d0
Code: 7c 24 20 48 8b b5 90 00 00 00 e8 df 85 f6 ff 48 8b 7c 24 20 41 89 c4 e8 92 81 f6 ff 45 85 e4 0f 85 57 fe ff ff e9 66 ff ff ff <0f> 0b e9 2e ff ff ff 3c 0a 0f 85 86 fe ff ff 48 8b 8d 98 00 00 00
RSP: 0018:ffffa30f0082ba60 EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffffa30f0082bbc0 RCX: 0000000000000000
RDX: ffffa30f0082bce0 RSI: ffff973adc06ee80 RDI: ffffa30f0082bb08
RBP: ffff973adc06ee80 R08: 00000000000000fe R09: ffffa30f0082bce0
R10: 0000000000000000 R11: 0000000000000000 R12: ffffa30f0082ba88
R13: 0000000000000015 R14: ffffa30f0082bbc0 R15: ffff973ae9520980
FS:  00007f04b8e97440(0000) GS:ffff973aeee00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1f62ee2370 CR3: 00000004600ae006 CR4: 00000000003606f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 fib_table_delete+0x1aa/0x300
 inet_rtm_delroute+0xa7/0x120
 rtnetlink_rcv_msg+0x137/0x3c0
 ? rtnl_calcit.isra.0+0x120/0x120
 netlink_rcv_skb+0x78/0x140
 netlink_unicast+0x245/0x350
 netlink_sendmsg+0x243/0x480
 sock_sendmsg+0x5e/0x60
 __sys_sendto+0x120/0x180
 __x64_sys_sendto+0x25/0x30
 do_syscall_64+0x49/0x90
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f04b9cc288a
Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 76 c3 0f 1f 44 00 00 55 48 83 ec 30 44 89 4c
RSP: 002b:00007ffdde952e68 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 00007ffdde952eec RCX: 00007f04b9cc288a
RDX: 0000000000000034 RSI: 000055e0afb70cf0 RDI: 0000000000000003
RBP: 000055e0afb33080 R08: 00007ffdde952e70 R09: 0000000000000010
R10: 0000000000000000 R11: 0000000000000246 R12: 000055e0afb41c70
R13: 0000000000000137 R14: 000055e0afb6c750 R15: 000055e0af829378
---[ end trace a891504c031e281b ]---
RPC: fragment too large: 50331695
RPC: fragment too large: 50331695
RPC: fragment too large: 50331695
[root@gw ~]#

[holdenk]

Ran into this on Linux ohana 5.8.0-55-generic #62-Ubuntu SMP Tue Jun 1 08:21:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux