Use Equinix Metal LB service to provision the Kubernetes API VIP

[displague]

User Story

As a [developer/user/operator] I would like to have CAPP provision an IP via Metal LBaaS to create a Kubernetes API IP that will balance traffic between the control plane nodes. This would be a simpler to manage process than the existing K8s controller managing via KubeVIP or Metal API calls that juggle a Metal EIP.

Detailed Description

TODO: [A clear and concise description of what you want to happen.]

Anything else you would like to add:

TODO: [Miscellaneous information that will assist in solving the issue.]

• https://deploy.equinix.com/developers/docs/metal/networking/load-balancers/

/kind feature

[cprivitere]

Current Status:

• Add "EMLB" VIPManager type
• Create emlb based template that creates a cluster with an EMLB-based VIP
• Add LBaaS SDK to codebase (since it's not publically available yet)
• Add emlb internal package
• Can create new load balancer
• Can create new origins
• Can create new pools
• Can create new listener ports
• Can add origins to pools
• Can add pool to listener port
• Creates new load balancer, pool, port, and origin with new cluster using EMLB VIPManager type.
• Deletes load balancer, pool, port, origin when deleting cluster.
• Test coverage for new functions
• Update e2e test to check emlb template
• Update packetmachine and packetcluster to do the same things for kube-vip as they do for cpem, as it needs those as well.
• Update packetmachine to handle worker node load balancer annotations. Exclude worker nodes from EMLB pool deletion processing.
• Update e2e cleanup to clean up old emlbs too.
• Update README to have an example in using EMLB