Skip to content

mgd: cleanup routes from deleted bgp peer #360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 20, 2024
Merged

mgd: cleanup routes from deleted bgp peer #360

merged 1 commit into from
Sep 20, 2024

Conversation

@taspelund
Copy link
Contributor

@taspelund taspelund commented Sep 10, 2024

Fixes: #349

@taspelund taspelund added Bug needs testing bgp Border Gateway Protocol labels Sep 10, 2024
@taspelund taspelund self-assigned this Sep 10, 2024
@taspelund taspelund force-pushed the trey/admin_shut_withdrawal branch 2 times, most recently from 4c4a202 to 18355eb Compare September 17, 2024 19:03
@elaine-oxide
Copy link

elaine-oxide commented Sep 19, 2024

I am running a4x2 with:

  • omicron bac635f68831c8a3792156fcfd5ce13bef61fa37
    • with changes required to get maghemite 18355eb19164876c79b76b15a5439fd54269ff12 and dependencies (anyhow, serde, serde_json)
    • oxnet 7dacd265f1bcd0f8b47bd4805250c4f0812da206
  • Oxide CLI version compiled from main with this commit:
$ oxide version
Oxide CLI 0.7.0+20240821.0
Built from commit: 5ffdf5a0325bb62f3235a2e3a2f0d73f8452525a 
Oxide API: 20240821.0

I ran a slightly modified version of the test flow in my comments in #349

Test run 1:

Initial state on g3 (serves requests from Oxide CLI) before bgp peer delete (already set local-pref to 50):

root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  Some(50)    64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status selected 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  Some(50)    64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status neighbors 65547
Peer Address  Peer ASN     State        State Duration  Hold   Keepalive
169.254.40.1  Some(64502)  Established  24s 240ms       6s/6s  2s/2s
169.254.20.1  Some(64500)  Established  8m 7s 774ms     6s/6s  2s/2s

Initial state reported by CLI output before bgp peer delete:

$ oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.10.1  65547      64500       Established    12m 55s 440ms
169.254.30.1  65547      64502       Established    12m 55s 440ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.40.1  65547      64502       Established    47s 853ms
169.254.20.1  65547      64500       Established    8m 31s 387ms

$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None
169.254.40.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.40.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          Some(50)    None      None     None  None        None

Destination  Nexthop  Vlan  Preference

switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None
169.254.30.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.30.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

Delete the bgp peer.

$ oxide system networking bgp peer delete --rack $rack --switch switch1 --port qsfp0 --addr 169.254.40.1

Resulting state on g3 after bgp peer delete (presence of 169.254.40.1 unexpected, but this may be due to checking too quickly):

root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  Some(50)    64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status selected 65547
BGP Routes
=============
Prefix     Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0  169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None

root@oxz_switch:~# mgadm bgp status neighbors 65547
Peer Address  Peer ASN     State        State Duration  Hold   Keepalive
169.254.20.1  Some(64500)  Established  9m 28s 811ms    6s/6s  2s/2s

Resulting state reported by CLI output after bgp peer delete:

$ oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.30.1  65547      64502       Established    14m 24s 408ms
169.254.10.1  65547      64500       Established    14m 24s 408ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.20.1  65547      64500       Established    10m 354ms

$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None
169.254.40.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None
169.254.30.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.30.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

Now add the BGP peer back without local-pref.

$ oxide system networking bgp peer set --rack $rack --switch switch1 --port qsfp0 --addr 169.254.40.1 --bgp-config as65547

Resulting state on g3 after bgp peer set (looks good, doesn't have the leftover route anymore):

root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  None        64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status selected 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  None        64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status neighbors 65547
Peer Address  Peer ASN     State        State Duration  Hold   Keepalive
169.254.20.1  Some(64500)  Established  11m 16s 3ms     6s/6s  2s/2s
169.254.40.1  Some(64502)  Established  24s 971ms       6s/6s  2s/2s

Resulting state reported by CLI output after bgp peer set:

$ oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.10.1  65547      64500       Established    15m 56s 364ms
169.254.30.1  65547      64502       Established    15m 56s 364ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.20.1  65547      64500       Established    11m 32s 307ms
169.254.40.1  65547      64502       Established    41s 275ms

$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None
169.254.40.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.40.1  as65547  [no filtering]  [no filtering]  []           0              0           false             6          0               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None
169.254.30.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.30.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

@elaine-oxide
Copy link

elaine-oxide commented Sep 19, 2024 •
edited
Loading

Continuation of previous comment.

Test run 2.

Initial state on g3 (serves requests from Oxide CLI) before bgp peer delete (already set local-pref to 50):

root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  Some(50)    64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status selected 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  Some(50)    64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status neighbors 65547
Peer Address  Peer ASN     State        State Duration  Hold   Keepalive
169.254.20.1  Some(64500)  Established  16m 27s 723ms   6s/6s  2s/2s
169.254.40.1  Some(64502)  Established  10s 845ms       6s/6s  2s/2s

Initial state reported by CLI output before bgp peer delete:

$ oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.30.1  65547      64502       Established    21m 2s 310ms
169.254.10.1  65547      64500       Established    21m 2s 310ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.20.1  65547      64500       Established    16m 38s 247ms
169.254.40.1  65547      64502       Established    21s 369ms

$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None
169.254.40.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.40.1  as65547  [no filtering]  [no filtering]  []           0              0           false             6          0               2          Some(50)    None      None     None  None        None

Destination  Nexthop  Vlan  Preference

switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None
169.254.30.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.30.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

Delete the bgp peer.

$ oxide system networking bgp peer delete --rack $rack --switch switch1 --port qsfp0 --addr 169.254.40.1

Resulting state on g3 after bgp peer delete (this time it looks ok, no leftover routes):

root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix     Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0  169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None

root@oxz_switch:~# mgadm bgp status selected 65547
BGP Routes
=============
Prefix     Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0  169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None

root@oxz_switch:~# mgadm bgp status neighbors 65547
Peer Address  Peer ASN     State        State Duration  Hold   Keepalive
169.254.20.1  Some(64500)  Established  27m 27s 705ms   6s/6s  2s/2s

Resulting state reported by CLI output after bgp peer delete:

$ oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.30.1  65547      64502       Established    32m 32s 60ms
169.254.10.1  65547      64500       Established    32m 32s 60ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.20.1  65547      64500       Established    28m 7s 985ms

$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None
169.254.40.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None
169.254.30.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.30.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

Now add the BGP peer back without local-pref.

$ oxide system networking bgp peer set --rack $rack --switch switch1 --port qsfp0 --addr 169.254.40.1 --bgp-config as65547

Resulting state on g3 after bgp peer set (looks good):

root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  None        64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status selected 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  None        64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status neighbors 65547
Peer Address  Peer ASN     State        State Duration  Hold   Keepalive
169.254.20.1  Some(64500)  Established  29m 59s 552ms   6s/6s  2s/2s
169.254.40.1  Some(64502)  Established  14s 656ms       6s/6s  2s/2s

Resulting state reported by CLI output after bgp peer set:

$ oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.10.1  65547      64500       Established    34m 40s 248ms
169.254.30.1  65547      64502       Established    34m 40s 248ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.20.1  65547      64500       Established    30m 16s 174ms
169.254.40.1  65547      64502       Established    31s 278ms

$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None
169.254.40.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.40.1  as65547  [no filtering]  [no filtering]  []           0              0           false             6          0               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None
169.254.30.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.30.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

@taspelund taspelund marked this pull request as ready for review September 19, 2024 16:25
@taspelund
Copy link
Contributor Author

taspelund commented Sep 19, 2024

Thanks for running this through your testing @elaine-oxide! If you're satisfied with the results of test #2, then so am I.

@rcgoodfellow this is ready for review

@elaine-oxide
Copy link

elaine-oxide commented Sep 19, 2024 •
edited
Loading

Continuation of my previous comment.

Test run 3

Note: the purpose of this third run is for sanity, to make sure we can ignore the fact that I saw leftover routes in Test run 1.

  • In Test run 1, there was a moderate delay between running the Oxide CLI oxide system networking bgp peer command and then the mgadm bgp status imported 65547 command.
  • In Test run 2, there was a longer delay before running the second command.
  • In Test run 3, there was a very short (shorter than for Test run 1) delay before running the second command.
    Results of Test run 3 show that leftover routes disappear within a few seconds, on second (or third) invocation of the second command, once the new information has finished propagating.

So, we can safely ignore the leftover routes observed in Test run 1, since if I had run the second command a second (or third) time, I would get the same results as in Test run 3.

Initial state on g3 (serves requests from Oxide CLI) before bgp peer delete (already set local-pref to 50):

root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  Some(50)    64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status selected 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  Some(50)    64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status neighbors 65547
Peer Address  Peer ASN     State        State Duration     Hold   Keepalive
169.254.40.1  Some(64502)  Established  8s 574ms           6s/6s  2s/2s
169.254.20.1  Some(64500)  Established  11h 29m 14s 412ms  6s/6s  2s/2s

Initial state reported by CLI output before bgp peer delete:

$ oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.10.1  65547      64500       Established    11h 34m 12s 97ms
169.254.30.1  65547      64502       Established    11h 34m 12s 97ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.20.1  65547      64500       Established    11h 29m 47s 391ms
169.254.40.1  65547      64502       Established    41s 553ms

$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None
169.254.40.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.40.1  as65547  [no filtering]  [no filtering]  []           0              0           false             6          0               2          Some(50)    None      None     None  None        None

Destination  Nexthop  Vlan  Preference

switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None
169.254.30.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.30.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

Delete the bgp peer.

$ oxide system networking bgp peer delete --rack $rack --switch switch1 --port qsfp0 --addr 169.254.40.1

Resulting state on g3 after bgp peer delete (ran these commands very soon after the above command, the new information is propagated after a second or two):

# Too soon
root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  Some(50)    64502      172.20.2.186  None  [64502, 64520]  None

# Ok now
root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix     Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0  169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None

root@oxz_switch:~# mgadm bgp status selected 65547
BGP Routes
=============
Prefix     Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0  169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None

root@oxz_switch:~# mgadm bgp status neighbors 65547
Peer Address  Peer ASN     State        State Duration     Hold   Keepalive
169.254.20.1  Some(64500)  Established  11h 30m 55s 450ms  6s/6s  2s/2s

Resulting state reported by CLI output after bgp peer delete:

$ oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.10.1  65547      64500       Established    11h 36m 9s 408ms
169.254.30.1  65547      64502       Established    11h 36m 9s 408ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.20.1  65547      64500       Established    11h 31m 44s 704ms

$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None
169.254.40.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None
169.254.30.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.30.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

Now add the BGP peer back without local-pref.

$ oxide system networking bgp peer set --rack $rack --switch switch1 --port qsfp0 --addr 169.254.40.1 --bgp-config as65547

Resulting state on g3 after bgp peer set (ran these commands very soon after the above command, the new information is propagated after a second or two):

# Too soon
root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix     Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0  169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None

# Still too soon
root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix     Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0  169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None

# Ok now
root@oxz_switch:~# mgadm bgp status imported 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  None        64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status selected 65547
BGP Routes
=============
Prefix       Nexthop       Local Pref  Origin AS  Peer ID       MED   AS Path         Stale
0.0.0.0/0    169.254.20.1  None        64500      172.20.2.191  None  [64500, 64510]  None
240.0.0.0/4  169.254.40.1  None        64502      172.20.2.186  None  [64502, 64520]  None

root@oxz_switch:~# mgadm bgp status neighbors 65547
Peer Address  Peer ASN     State        State Duration  Hold   Keepalive
169.254.40.1  Some(64502)  Established  10s 412ms       6s/6s  2s/2s
169.254.20.1  Some(64500)  Established  11h 33m 929ms   6s/6s  2s/2s

Resulting state reported by CLI output after bgp peer set:

$ oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.30.1  65547      64502       Established    11h 38m 23s 324ms
169.254.10.1  65547      64500       Established    11h 38m 23s 324ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.20.1  65547      64500       Established    11h 33m 58s 618ms
169.254.40.1  65547      64502       Established    1m 8s 101ms

$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None
169.254.40.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.40.1  as65547  [no filtering]  [no filtering]  []           0              0           false             6          0               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None
169.254.30.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None
169.254.30.1  as65547  [no filtering]  [no filtering]  []           3              0           false             6          3               2          None        None      None     None  None        None

Destination  Nexthop  Vlan  Preference

rcgoodfellow
rcgoodfellow reviewed Sep 19, 2024
View reviewed changes
@taspelund
mgd: cleanup routes from deleted bgp peer
549492a 
Fixes: #349

Signed-off-by: Trey Aspelund <trey@oxidecomputer.com>
@taspelund taspelund force-pushed the trey/admin_shut_withdrawal branch from 18355eb to 549492a Compare September 20, 2024 20:37
rcgoodfellow
rcgoodfellow approved these changes Sep 20, 2024
View reviewed changes
Copy link
Collaborator

@rcgoodfellow rcgoodfellow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@taspelund taspelund merged commit 375f7c9 into main Sep 20, 2024
@taspelund taspelund deleted the trey/admin_shut_withdrawal branch September 20, 2024 21:12
This was referenced Sep 27, 2024
Draft
Merged
Closed
ahl added a commit to oxidecomputer/omicron that referenced this pull request Sep 27, 2024
@ahl
hyper v1: work around the current maghemite / omicron lag (#6706)
75214e1 
This is kind of gross.

Omicron currently is out of sync with maghemite, see #6693 and the two
maghemite pushes that require synchronization:
oxidecomputer/maghemite#359 and
oxidecomputer/maghemite#360. I'd like to make
forward progress with the hyper v1 migration, and that's blocked on
updating the omicron dependency of maghemite which pulls in old hyper
(reqwest, progenitor, etc). This gets pull into other repos via
omicron-common.

It's worth noting that this circular arrangement seems lousy. The
"vassal crates" (crucible, propolis, maghemite, (and dendrite to a
lesser degree)) depend on omicron-common, but omicron-common pulls in
maghemite's `mg-admin-client` which in turn pulls in... lots...
including progenitor, hyper, reqwest, etc.

My goal here is to have a temporary dependency on a branch of maghemite
(oxidecomputer/maghemite#378); once #6693 is
integrated, we can then pin the dependency on maghemite's HEAD rev.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rcgoodfellow rcgoodfellow rcgoodfellow approved these changes

Assignees

@taspelund taspelund

Labels

bgp Border Gateway Protocol Bug

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Administrative removal of peers does not remove prefixes in RIB imported by that peer

4 participants

@taspelund @elaine-oxide @rcgoodfellow