BGP neighborship not coming up using ebgp-multihop and ttl-security

[madhu222]

Description

BGP neighborship not coming up after configuring ebgp multihop or ttl-security hops.

Steps to Reproduce

1. Have two systems connected back to back
2. Configure layer 3 vlan , ip address on both systems
3. Configure loopback on both systems
4. Have a static route on each system to the corresponding loopback IP

(Lo-1.1.1.1)R1(br100)------100.1.1.0/24---------(br100)R2(Lo-2.2.2.2)

R1 --- ip route 2.2.2.2/32 100.1.1.2
R2 --- ip route 1.1.1.1/32 100.1.1.1

Expected behavior:
EBGP neighborship should get established between the loopbacks 1.1.1.1 and 2.2.2.2

Actual behavior:
Neighborship didn't come up. The problem is it gets struck in active state. Only after flapping the br100 interface, the neighborship comes up. I checked the neighbor ARP entries are not timed out.

Components

[bgpd]

Versions

• Kernel: 4.9.0-6-amd64 A small collection of patches that fix issues found by valgrind #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
• FRR: 6.0

Attachments

R1# clear ip bgp *
2018/11/02 21:52:23 BGP: 2.2.2.2 [FSM] BGP_Stop (Active->Idle), fd -1
2018/11/02 21:52:23 BGP: 2.2.2.2 [FSM] BGP_Stop (Active->Idle), fd -1
2018/11/02 21:52:23 BGP: 2.2.2.2 went from Active to Idle
2018/11/02 21:52:23 BGP: 2.2.2.2 went from Active to Idle
R1# 2018/11/02 21:52:24 BGP: 2.2.2.2 [FSM] Timer (start timer expire).
2018/11/02 21:52:24 BGP: 2.2.2.2 [FSM] BGP_Start (Idle->Connect), fd -1
2018/11/02 21:52:24 BGP: 2.2.2.2 [FSM] Waiting for NHT
2018/11/02 21:52:24 BGP: 2.2.2.2 went from Idle to Connect
2018/11/02 21:52:24 BGP: 2.2.2.2 [FSM] TCP_connection_open_failed (Connect->Active), fd -1
2018/11/02 21:52:24 BGP: 2.2.2.2 [FSM] Timer (start timer expire).
2018/11/02 21:52:24 BGP: 2.2.2.2 [FSM] BGP_Start (Idle->Connect), fd -1
2018/11/02 21:52:24 BGP: 2.2.2.2 [FSM] Waiting for NHT
2018/11/02 21:52:24 BGP: 2.2.2.2 went from Idle to Connect
2018/11/02 21:52:24 BGP: 2.2.2.2 [FSM] TCP_connection_open_failed (Connect->Active), fd -1
2018/11/02 21:52:24 BGP: 2.2.2.2 went from Connect to Active
2018/11/02 21:52:24 BGP: 2.2.2.2 went from Connect to Active

R1(config)# int br100
R1(config-if)# do show interface br100
Interface br100 is up, line protocol is up
  Link ups:       2    last: 2018/11/02 21:48:47.99
  Link downs:     2    last: 2018/11/02 21:48:45.76
  vrf: Default-IP-Routing-Table
  index 42 metric 0 mtu 1500 speed 0
  flags: <UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
  Type: Ethernet
  HWaddr: 14:18:77:95:03:57
  inet 100.1.1.1/24 broadcast 100.1.1.255
  inet6 fe80::1618:77ff:fe95:357/64
  Interface Type Bridge
  Bridge VLAN-aware: no
R1(config-if)# do show 2018/11/02 21:53:12 BGP: [Event] 200.1.1.2 connection rejected - not configured and not valid for dynamic
2018/11/02 21:53:12 BGP: [Event] 200.1.1.2 connection rejected - not configured and not valid for dynamic

% Command incomplete: do show
R1(config-if)# int br100
R1(config-if)# shutdown
2018/11/02 21:53:24 BGP: Rx Intf down VRF 0 IF br100
2018/11/02 21:53:24 BGP: Rx Intf address del VRF 0 IF br100 addr 100.1.1.1/24
2018/11/02 21:53:24 BGP: Rx Intf address del VRF 0 IF br100 addr fe80::1618:77ff:fe95:357/64
2018/11/02 21:53:24 BGP: Rx Intf down VRF 0 IF br100
2018/11/02 21:53:24 BGP: Rx Intf address del VRF 0 IF br100 addr 100.1.1.1/24
2018/11/02 21:53:24 BGP: Rx Intf address del VRF 0 IF br100 addr fe80::1618:77ff:fe95:357/64
2018/11/02 21:53:24 BGP: 0: Rcvd NH update 2.2.2.2/32 - metric 0/0 #nhops 0/1 flags 0xa
2018/11/02 21:53:24 BGP: NH update for 2.2.2.2/32 - flags 0x2 chgflags 0x1 - evaluate paths
2018/11/02 21:53:24 BGP: evaluate_paths: Updating peer (2.2.2.2) status with NHT
2018/11/02 21:53:24 BGP: 0: Rcvd NH update 2.2.2.2/32 - metric 0/0 #nhops 0/1 flags 0xa
2018/11/02 21:53:24 BGP: NH update for 2.2.2.2/32 - flags 0x2 chgflags 0x1 - evaluate paths
2018/11/02 21:53:24 BGP: evaluate_paths: Updating peer (2.2.2.2) status with NHT
R1(config-if)# no shutdown
2018/11/02 21:53:29 BGP: Rx Intf up VRF 0 IF br100
2018/11/02 21:53:29 BGP: Rx Intf up VRF 0 IF br100
2018/11/02 21:53:29 BGP: Rx Intf address add VRF 0 IF br100 addr 100.1.1.1/24
2018/11/02 21:53:29 BGP: Rx Intf address add VRF 0 IF br100 addr 100.1.1.1/24
2018/11/02 21:53:29 BGP: 0: Rcvd NH update 2.2.2.2/32 - metric 0/0 #nhops 1/0 flags 0xa
2018/11/02 21:53:29 BGP:     nhop via 100.1.1.2 (0 labels)
2018/11/02 21:53:29 BGP: NH update for 2.2.2.2/32 - flags 0x3 chgflags 0x1 - evaluate paths
2018/11/02 21:53:29 BGP: evaluate_paths: Updating peer (2.2.2.2) status with NHT
2018/11/02 21:53:29 BGP: 0: Rcvd NH update 2.2.2.2/32 - metric 0/0 #nhops 1/0 flags 0xa
2018/11/02 21:53:29 BGP:     nhop via 100.1.1.2 (0 labels)
2018/11/02 21:53:29 BGP: NH update for 2.2.2.2/32 - flags 0x3 chgflags 0x1 - evaluate paths
2018/11/02 21:53:29 BGP: evaluate_paths: Updating peer (2.2.2.2) status with NHT
2018/11/02 21:53:29 BGP: 2.2.2.2 [FSM] ConnectRetry_timer_expired (Active->Connect), fd -1
2018/11/02 21:53:29 BGP: 2.2.2.2 [FSM] ConnectRetry_timer_expired (Active->Connect), fd -1
2018/11/02 21:53:29 BGP: 2.2.2.2 [Event] Connect start to 2.2.2.2 fd 22
2018/11/02 21:53:29 BGP: 2.2.2.2 [FSM] Non blocking connect waiting result, fd 22
2018/11/02 21:53:29 BGP: 2.2.2.2 went from Active to Connect
2018/11/02 21:53:29 BGP: 2.2.2.2 [Event] Connect start to 2.2.2.2 fd 22
2018/11/02 21:53:29 BGP: 2.2.2.2 [FSM] Non blocking connect waiting result, fd 22
2018/11/02 21:53:29 BGP: 2.2.2.2 went from Active to Connect
2018/11/02 21:53:29 ZEBRA: Unknown netlink nlmsg_type RTM_GETNEIGH(30) vrf 0

2018/11/02 21:53:29 ZEBRA: Unknown netlink nlmsg_type RTM_GETNEIGH(30) vrf 0

R1(config-if)# 2018/11/02 21:53:31 BGP: 2.2.2.2 [FSM] TCP_connection_open (Connect->OpenSent), fd 22
2018/11/02 21:53:31 BGP: 2.2.2.2 [FSM] TCP_connection_open (Connect->OpenSent), fd 22
2018/11/02 21:53:31 BGP: 2.2.2.2 open active, local address 1.1.1.1
2018/11/02 21:53:31 BGP: 2.2.2.2 open active, local address 1.1.1.1
2018/11/02 21:53:31 BGP: 2.2.2.2 Sending hostname cap with hn = R2, dn = (null)
2018/11/02 21:53:31 BGP: 2.2.2.2 Sending hostname cap with hn = R2, dn = (null)
2018/11/02 21:53:31 BGP: 2.2.2.2 sending OPEN, version 4, my as 234567, holdtime 180, id 1.1.1.1
2018/11/02 21:53:31 BGP: 2.2.2.2 sending OPEN, version 4, my as 234567, holdtime 180, id 1.1.1.1
2018/11/02 21:53:31 BGP: 2.2.2.2 went from Connect to OpenSent
2018/11/02 21:53:31 BGP: 2.2.2.2 went from Connect to OpenSent
2018/11/02 21:53:31 BGP: 2.2.2.2 rcv OPEN, version 4, remote-as (in open) 23456, holdtime 180, id 2.2.2.2
2018/11/02 21:53:31 BGP: 2.2.2.2 rcv OPEN w/ OPTION parameter len: 47
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has MultiProtocol Extensions capability (1), length 4
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has MP_EXT CAP for afi/safi: 1/1
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has Route Refresh (Old) capability (128), length 0
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has Route Refresh capability (2), length 0
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has 4-octet AS number capability (65), length 4
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has AddPath capability (69), length 4
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has AddPath CAP for afi/safi: 1/1, receive
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 7
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has FQDN capability (73), length 5
2018/11/02 21:53:31 BGP: 2.2.2.2 received hostname R2, domainname (null)
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 4
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has Graceful Restart capability (64), length 2
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has Graceful Restart capability
2018/11/02 21:53:31 BGP: 2.2.2.2 Peer has not restarted. Restart Time : 120
2018/11/02 21:53:31 BGP: 2.2.2.2 [FSM] Receive_OPEN_message (OpenSent->OpenConfirm), fd 22
2018/11/02 21:53:31 BGP: 2.2.2.2 sending KEEPALIVE
2018/11/02 21:53:31 BGP: 2.2.2.2 went from OpenSent to OpenConfirm
2018/11/02 21:53:31 BGP: 2.2.2.2 rcv OPEN, version 4, remote-as (in open) 23456, holdtime 180, id 2.2.2.2
2018/11/02 21:53:31 BGP: 2.2.2.2 rcv OPEN w/ OPTION parameter len: 47
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has MultiProtocol Extensions capability (1), length 4
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has MP_EXT CAP for afi/safi: 1/1
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has Route Refresh (Old) capability (128), length 0
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has Route Refresh capability (2), length 0
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has 4-octet AS number capability (65), length 4
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has AddPath capability (69), length 4
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has AddPath CAP for afi/safi: 1/1, receive
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 7
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has FQDN capability (73), length 5
2018/11/02 21:53:31 BGP: 2.2.2.2 received hostname R2, domainname (null)
2018/11/02 21:53:31 BGP: 2.2.2.2 rcvd OPEN w/ optional parameter type 2 (Capability) len 4
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has Graceful Restart capability (64), length 2
2018/11/02 21:53:31 BGP: 2.2.2.2 OPEN has Graceful Restart capability
2018/11/02 21:53:31 BGP: 2.2.2.2 Peer has not restarted. Restart Time : 120
2018/11/02 21:53:31 BGP: 2.2.2.2 [FSM] Receive_OPEN_message (OpenSent->OpenConfirm), fd 22
2018/11/02 21:53:31 BGP: 2.2.2.2 sending KEEPALIVE
2018/11/02 21:53:31 BGP: 2.2.2.2 went from OpenSent to OpenConfirm
2018/11/02 21:53:31 BGP: 2.2.2.2 KEEPALIVE rcvd
2018/11/02 21:53:31 BGP: 2.2.2.2 [FSM] Receive_KEEPALIVE_message (OpenConfirm->Established), fd 22
2018/11/02 21:53:31 BGP: 2.2.2.2 went from OpenConfirm to Established
2018/11/02 21:53:31 BGP: 2.2.2.2 [FSM] Timer (routeadv timer expire)
2018/11/02 21:53:31 BGP: 2.2.2.2 KEEPALIVE rcvd
2018/11/02 21:53:31 BGP: 2.2.2.2 [FSM] Receive_KEEPALIVE_message (OpenConfirm->Established), fd 22
2018/11/02 21:53:31 BGP: 2.2.2.2 went from OpenConfirm to Established
2018/11/02 21:53:31 BGP: 2.2.2.2 [FSM] Timer (routeadv timer expire)
2018/11/02 21:53:32 BGP: Rx Intf address add VRF 0 IF br100 addr fe80::1618:77ff:fe95:357/64
2018/11/02 21:53:32 BGP: Rx Intf address add VRF 0 IF br100 addr fe80::1618:77ff:fe95:357/64
2018/11/02 21:53:32 BGP: u11:s11 announcing routes upon coalesce timer expiry
2018/11/02 21:53:32 BGP: 2.2.2.2 [FSM] Timer (routeadv timer expire)
2018/11/02 21:53:32 BGP: send End-of-RIB for IPv4 Unicast to 2.2.2.2
2018/11/02 21:53:32 BGP: u11:s11 announcing routes upon coalesce timer expiry
2018/11/02 21:53:32 BGP: 2.2.2.2 [FSM] Timer (routeadv timer expire)
2018/11/02 21:53:32 BGP: send End-of-RIB for IPv4 Unicast to 2.2.2.2
2018/11/02 21:53:32 BGP: 2.2.2.2 rcvd UPDATE wlen 0 attrlen 0 alen 0
2018/11/02 21:53:32 BGP: rcvd End-of-RIB for IPv4 Unicast from 2.2.2.2
2018/11/02 21:53:32 BGP: 2.2.2.2 rcvd UPDATE wlen 0 attrlen 0 alen 0
2018/11/02 21:53:32 BGP: rcvd End-of-RIB for IPv4 Unicast from 2.2.2.2

Thanks!

[bisdhdh]

@madhu222 .
Is this issue same as #3264 ?

Please verify and let me know.

Thanks
Biswajit.

[madhu222]

@bisdhdh I believe this is a different issue. The issue I am facing the issues where BGP messages are sent only if I ping or flap the next hop to reach the neighbor which is loopback

[vijaykug]

@madhu222 i#3264 also recovered after the next hop flap , this looks like sequencing issue .
#3264 is observed when ebgp multihop cli configured before the update source cli .
if update source cli is use before ebgp multihop then BGP session coming up fine.
can you check the CLI sequence you used ?

[bisdhdh]

@madhu222
Since your BGP session is on loopback. I hope you had used multi hop & update source lo command ?
Could you please share the configuration of the both the routers ?

[madhu222]

@bisdhdh Yes I used update source and ebgp-multihop command.

[bisdhdh]

@madhu222
I had fixed one major issue related to ebgp multihop.
Pull request -> #3387
I could not recreate the issue reported by you on the build with my fix. You may try recreating this issue with the above pull request changes, see if you can else we may close this as duplicate.

[qlyoung]

@madhu222 - can you please retry with the latest master build?

[qlyoung]

Closing due to lack of activity.