Description
Currently we constrain multiple NICs for an instance in various ways. For example, they must all be in the same VPC. An additional constraint that's not yet enforced is that an instance may not have multiple NICs in the same VPC Subnet. The main reason for this is that it will likely result in problems for the guest networking stack. Should they create multiple such NICs, the routing tables in the guest would be confusing and probably wrong, since there's no obvious way for the guest kernel to pick one NIC over another for traffic destined for an address on that IP subnetwork.
This should be pretty straightforward, adding an additional constraint-checking subquery to the query in omicron_nexus::db::queries::network_interface::InsertQuery, and an error-handling branch in the corresponding network_interface::InsertError type.
EDIT: It's also possible this would be better implemented as a unique index, e.g., on (instance_id, subnet_id). I don't know if we'd use that index for searching, however, which means the space cost may not be worth it.