Separate installations on the same network can interfere with each other

[iliana]

Here is a CI failure: https://github.com/oxidecomputer/omicron/runs/7887995648

This failure occurred shortly after we added sock to Buildomat to work alongside buskin for lab jobs. This job ran on sock, and ddmd picked up a prefix during startup:

SledAgent (RSS): Received prefixes from ddmd
    prefixes = {"fe80::8:20ff:fe9e:7b26": [Ipv6Prefix { addr: fd00:1122:3344:1::, mask: 64 }, Ipv6Prefix { addr: fdb0:18c0:4d0d:9fb2::, mask: 64 }, Ipv6Prefix { addr: fd00:1122:3344:101::, mask: 64 }]}

This prefix was advertised by ddmd on buskin, during https://github.com/oxidecomputer/omicron/runs/7887685952:

SledAgent: Sending prefix to ddmd for advertisement
    prefix = Ipv6Prefix { addr: fdb0:18c0:4d0d:9fb2::, mask: 64 }

The result is that the job on sock created a plan for two sled agents, I think, then failed to start sled agent because it was already running on buskin?

There is a timestamp discrepancy in the logs; either sock's time is one hour in the future, or... well, I'm not sure what else it'd be. Some sort of lab network state being cached somewhere?

@jclulow is changing the lab network so that sock and buskin are on two separate VLANs, which seems like the correct configuration for CI, so not labeling this as a test flake. Instead I'd like to know if we want to prevent this kind of thing from happening elsewhere, e.g. someone testing two separate control plane installations at once on their home network without intending for them to talk to each other, or if that's just a bad idea.

[iliana]

(It's also possible sock's clock is right, and that another job outside of anything that landed on main interfered... I'm not 100% sure how IPv6 prefix selection works but if it's based on the machine's MAC address, I can see how this could reasonably happen.)

[jclulow]

With my apologies for not having through this through, both lab machines now have their own wholly separate VLAN.

[davepacheco]

Was this a bug or just a CI deployment issue?

[jclulow]

Was this a bug or just a CI deployment issue?

There was definitely a CI environment issue, which has been corrected. It feels though like there is another issue on some level though, i.e., assume someone sticks a malicious host on the network, how do we intend to ignore the irrelevant annoucements?

[rmustacc]

Was this a bug or just a CI deployment issue?

There was definitely a CI environment issue, which has been corrected. It feels though like there is another issue on some level though, i.e., assume someone sticks a malicious host on the network, how do we intend to ignore the irrelevant annoucements?

In general, if you're on the bootstrap network, you're allowed to find them and talk to other sled agents. The main router will restrict this to only being for the bootstrap network which also is not allowed to leave the rack (in theory). The mainstay networks are supposed to eventually be signed. It's not clear to me though where assumptions were (that is sled agent, surrounding job, something else, etc.) that ended up using it.