Incorrect FRR configuration occurs if we set up import VRF routing after the FRR service is restarted in version 9.0.2/10.0.0/master

[skyblueted]

Description

Scenario: [Router A] <--iBGP, AS 64600, vrf: default--> [Router B] <--iBGP, AS65500, vrf: vrf1--> [Router C]

We want to establish two iBGP peers. Router B and Router C will establish a peer connection through VRF vrf1, and then import the routing information from Router B to Router A's default VRF.

Everything works as expected initially, until we restart the FRR service. After the restart, the running config is different from FRR configuration. If we run the 'vtysh -b' command, we will get 'BGP is already running; AS is XXXXX'

Additionally, when we remove the 'import vrf vrf1' configuration and then restart the FRR service, the FRR configuration returns to normal.

Version

FRR version: 9.0.2/10.0.0
Kernel version: 6.2.0-39-generic

How to reproduce

Using frr version 9.0.2 or 10.0.0

• Set up vrf table in linux kernel by iproute2

  ip link add vrf1 type table (table ID)
  ip link set vrf1 up
  ip link set eno5 master vrf1

• Establish two sets of iBGP peers using two different AS numbers: one in the default VRF (AS 64600) and the other in VRF1 (AS 65500)

• Restart FRR service

• vtysh -b
  It will return the result

BGP is already running; AS is 64600
line 63: Failure to communicate[13] to bgpd, line: router bgp 65500 vrf vrf1
line 64: % Unknown command[4]: bgp router-id 10.10.10.46
line 65: % Unknown command[4]: neighbor vm_fabric peer-group
line 66: % Unknown command[4]: neighbor vm_fabric remote-as 65500
line 67: % Unknown command[4]: neighbor vm_fabric description Internal VM Network
line 68: % Unknown command[4]: neighbor vm_fabric bfd
line 69: % Unknown command[4]: neighbor vm_fabric bfd profile bfd_template

Expected behavior

Restarting the FRR service should not affect the FRR configuration file

router bgp 64600
bgp router-id 10.10.10.46
neighbor fabric peer-group
neighbor fabric remote-as 64600
neighbor fabric timers connect 10
neighbor fabric capability extended-nexthop
neighbor eno5 interface peer-group fabric
!
address-family ipv4 unicast
redistribute kernel route-map route_filter
redistribute connected route-map route_filter
neighbor fabric soft-reconfiguration inbound
maximum-paths 64
maximum-paths ibgp 64
import vrf vrf1
exit-address-family
!
address-family ipv6 unicast
redistribute kernel route-map v6_filter
redistribute connected route-map v6_filter
neighbor fabric activate
neighbor fabric soft-reconfiguration inbound
maximum-paths 64
maximum-paths ibgp 64
import vrf vrf1
exit-address-family
exit
!
router bgp 65500 vrf vrf1
bgp router-id 10.10.10.46
neighbor vrf1_fabric peer-group
neighbor vrf1_fabric remote-as 65500
neighbor vrf1_fabric timers connect 10
neighbor vrf1_fabric capability extended-nexthop
neighbor br interface peer-group vrf1_fabric
!
address-family ipv4 unicast
redistribute kernel route-map route_filter
redistribute connected route-map route_filter
neighbor vrf1_fabric default-originate
neighbor vrf1_fabric soft-reconfiguration inbound
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family ipv6 unicast
redistribute kernel route-map v6_filter
redistribute connected route-map v6_filter
neighbor vrf1_fabric activate
neighbor vrf1_fabric default-originate
neighbor vrf1_fabric soft-reconfiguration inbound
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
exit

Actual behavior

The FRR configuration settings disappeared, and the AS number changed from 65500 to 64600.

router bgp 64600
bgp router-id 10.10.10.46
neighbor fabric peer-group
neighbor fabric remote-as 64600
neighbor fabric timers connect 10
neighbor fabric capability extended-nexthop
neighbor eno5 interface peer-group fabric
!
address-family ipv4 unicast
redistribute kernel route-map route_filter
redistribute connected route-map route_filter
neighbor fabric soft-reconfiguration inbound
maximum-paths 64
maximum-paths ibgp 64
import vrf vrf1
exit-address-family
!
address-family ipv6 unicast
redistribute kernel route-map v6_filter
redistribute connected route-map v6_filter
neighbor fabric activate
neighbor fabric soft-reconfiguration inbound
maximum-paths 64
maximum-paths ibgp 64
import vrf vrf1
exit-address-family
exit
!
> router bgp 64600 vrf vrf1
exit

Additional context

No response

Checklist

• I have searched the open issues for this bug.
• I have not included sensitive information in this report.

[skyblueted]

In the past two days, similar issues have been discovered (#16152 ).
However, testing with the latest master branch and following the reproduction steps still shows the same problem

root@ubuntu:/userap/frr# vtysh -b
[1126810|mgmtd] sending configuration
[1126811|zebra] sending configuration
[1126817|bgpd] sending configuration
[1126825|watchfrr] sending configuration
[1126827|staticd] sending configuration
[1126828|bfdd] sending configuration
Waiting for children to finish applying config...
[1126811|zebra] done
The route-map 'route_filter' does not exist.
The route-map 'route_filter' does not exist.
BGP is already running; AS is 64600
line 35: Failure to communicate[13] to bgpd, line: router bgp 65500 vrf vm
line 36: % Unknown command[4]: bgp router-id 10.10.10.46
line 37: % Unknown command[4]: neighbor vm_fabric peer-group
...

[skyblueted]

Steps for easier reproducibility:

Not load correctly when FRR (re)starts

vtysh <<EOF
configure
vrf vrf100
vni 10100
exit-vrf

router bgp 50
address-family ipv4 unicast
import vrf vrf100
exit-address-family
exit

router bgp 100 vrf vrf100
exit
EOF

2. Restart FRR service
3. vtysh -b

Correctly load configuration

vtysh <<EOF
configure
vrf vrf100
vni 10100
exit-vrf

router bgp 50
address-family ipv4 unicast
exit-address-family
exit

router bgp 100 vrf vrf100
exit
EOF

2. Restart FRR service
3. vtysh -b

Hi @ton31337 , I wanted to ask if it's possible to modify this issue so that one bgpd process can run multiple BGP instances segmented by VRFs and import vrf routing between different AS?

[ton31337]

@skyblueted could you check this PR (compile and test)? #16374

[skyblueted]

Hi @ton31337 , thank you for your support.
I'll check the PR in next week.

BTW, I think that is my fault. I noticed an issue with the order of execution of configurations yesterday.
When importing VRFs, placing the VRF configuration first and then executing the default VRF prevents any issues from occurring

The correct order to load configuration, it works fine.

vtysh <<EOF
configure
vrf vrf100
vni 10100
exit-vrf

router bgp 100 vrf vrf100
exit

router bgp 50
address-family ipv4 unicast
import vrf vrf100
exit-address-family
exit

EOF

[ton31337]

Yes, but it's still a bug, we shouldn't care about the order, because when you type "write mem", you get a wrong order again.

[ton31337]

Yes, but it's still a bug, we shouldn't care about the order, because when you type "write mem", you get a wrong order again.

[skyblueted]

Thanks for your explanation!
I wasn't aware that typing 'write mem' would change the configuration order again.

[skyblueted]

Hi @ton31337 , the original issue did not occur again after using the PR (#16374). 👍
However, another scenario was discovered: encountering an error message when resetting the environment
I'm not sure if this is another bug.

Steps:
Check this PR can fix auto-generated AS issue.

router bgp 50
!
address-family ipv4 unicast
import vrf vrf100
exit-address-family
exit
!
router bgp 100 vrf vrf100
exit
!
End

Reset commands:

vtysh
conf t
no router bgp 100 vrf vrf100
router bgp 50
address-family ipv4 unicast
no import vrf vrf100
no router bgp 50

It will return this error messages:
"% Cannot delete default BGP instance. Dependent VRF instances exist"

[ton31337]

Could you pull the latest (#16374) and retest again?