WIP: Add new external-dns addon for exposing ingress & LoadBalancer external IPs to the host

[segevfiner]

An initial version of an addon for
external-dns. Based on ExternalDNS for CoreDNS with minikube. Cobbled together from that tutorial, upstream helm charts, and the way minikube/kubeadm deploy etcd & CoreDNS. Those manifests probably
need some reviewing and tweaking (See TODO below)

Add an addon for installing external DNS. See https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/coredns.md which uses CoreDNS as the DNS server.

Unlike ingress-dns, which is minikube specific:

1. This is sometimes used in production, so will give similar behavior in development to production.
2. Will support LoadBalancer services.

Some notes about configuring the host that also apply for ingress-dns:

1. For Linux distros using systemd-resolved, you can use systemd.network units for configuring DNS domain specific DNS servers. Using the Domains & DNS keys in a new network unit matching the required interface. I used it before, but don't remember the details.
2. For Windows, there is NRPT which should allow setting this.

At this point it is still WIP/incomplete, I will need some help from someone to finish this according to the TODO list below and any other feedback.

TODO

• Which images to use for etcd and CoreDNS? The k8s.io ones minikube uses? or upstream ones (From docs or helm charts)? Using minikube's images means it can use the preloaded ones as long as it is kept up to date, at least for the latest Kubernetes version, though updating those is likely to be forgotten...
• readinessProbe/livenessProbe, what times, thresholds should be used, are the scripts, endpoints correct, etc. (I kinda copied them from somewhere and they are not necessarily optimal)
  • The readinessProbe for etcd breaks it because the headless service doesn't register it until it is ready but it queries for it to become ready, At first I tried just disabling it, but that also didn't always work, not sure why, so I added publishNotReadyAddresses to the service, which might be correct anyhow, though not sure if I should then add a separate service that does not publish unready addresses, though it is a single node deployment so might not matter much either way.
• Do any pods here need a priorityClass?
• Do any pods here need tolerations?
• Set resources correctly.
• Use hostPath instead of PVC for etcd? or maybe make it ephemeral and not use a volume at all?
• labels for the objects, etc. Are there any labels add-ons normally use? Any label schema this should use instead of just app?
• I used the headless service to access etcd, not sure that's the best/correct way, some etcd deployments seem to also add ClusterIP service and use that instead, might not matter much for a single node.
• The CoreDNS server currently forwards DNS to the in cluster DNS which will also answer for cluster.local, which is not desired on the host, had to explicitly except it. Not sure if that's the best config. I can't just dnsPolicy: Default like the in-cluster CoreDNS because the CoreDNS server needs to resolve the etcd server address.
• The CoreDNS server is not currently exposed to the host. I think we should use hostPort to expose it. This also requires hostIP due to having another DNS server listening on localhost on the minikube VM, this needs to be set automatically but I'm not sure how to add such a template parameter, as in: where to extract it from inside minikube. Just using hostNetwork will be problematic with accessing etcd.
• Add README.md explaining how to configure host, etc. Like ingress-dns has. Maybe also an example, though upstream examples using the appropriate domain should work too.
• The domain is currently hard-coded to test, we might want to make that configurable.
• Will likely be effected by whatever issue plagues ingress & ingress-dns on Docker driver, so might need the same provisions and documentation as them.

fixes #8980

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-ci-robot]

Welcome @segevfiner!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @segevfiner. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[minikube-bot]

Can one of the admins verify this patch?

[segevfiner]

/assign @josedonizetti

[medyagh]

hi @segevfiner this seems to be an interesting addon, do you mind pulling upstream and putting an example output in the description how to use this addon?

[medyagh]

if you please put steps in the description how to use this addon with example output this would be a good addon, if you can also document if it works with all drivers or only VM drivers?

[segevfiner]

To make this operational, we still need to somehow set the hostIP for exposing the DNS server to the host.

The CoreDNS server is not currently exposed to the host. I think we should use hostPort to expose it. This also requires hostIP due to having another DNS server listening on localhost on the minikube VM, this needs to be set automatically but I'm not sure how to add such a template parameter, as in: where to extract it from inside minikube. Just using hostNetwork will be problematic with accessing etcd.

But I wasn't sure how to get the addon infrastructure of minikube render that into a templated deployment spec.

The rest of the stuff in the TODO list I left in the PR description are about polishing this as far as I remember.

For trying it out:

kubectl create deployment nginx --image nginx
kubectl expose deployment nginx --port 80 --type LoadBalancer
kubectl annotate service nginx "external-dns.alpha.kubernetes.io/hostname=nginx.test"

And you should be able to query the created DNS server for it, but we need to fix the expose to host thing so you can do so from the host.

See for my initial POC of this #8980 (comment) for more information.

I only tested this with VM drivers, it likely suffers from whatever issue that prevent the ingress addon from working, that I don't fully understand, on the docker driver, as this too tries to expose a port via the host machine. Though it tries to do so differently using hostPort/hostIP so it can still communicate with the deployed etcdz rather than hostNetwork , once we get the hostPort/hostIP thing sorted, we can try to see if that works or not.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: segevfiner
To complete the pull request process, please assign josedonizetti after the PR has been reviewed.
You can assign the PR to them by writing /assign @josedonizetti in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[medyagh]

hi @segevfiner thanks for this PR, do you mind adding a tutorial of using this new addon with examples
the documentation for addons can be added to https://minikube.sigs.k8s.io/docs/handbook/addons/

[segevfiner]

I will once we can get it fully working, I need help with the last issue I described here: #9000 (comment)

The rest of the things mentioned in the PR description are about polishing it, stuff that I wasn't sure about and only someone more familiar with this project might be able to help with.

[bl-ue]

@segevfiner Keep up the great work! 👍 This is a fantastic idea. It's also PR 9,000 :)

[tstromberg]

is this ready to merge?

[segevfiner]

See the PR description for the list of remaining tasks. The major ones being figuring out how to handle hostIP so the dns server is exposed, minikube needs to set the hostIP in the dns server pod template, and I couldn't figure out where to get it from, and documentation (Which I'm not sure where to write, it is similar to the existing ingress-dns in terms of host setup but I can add some more details on to of what's already there)

[codecov-io]

Codecov Report

Merging #9000 (46e50b2) into master (a282354) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #9000   +/-   ##
=======================================
  Coverage   29.05%   29.05%           
=======================================
  Files         171      171           
  Lines       10443    10443           
=======================================
  Hits         3034     3034           
  Misses       6987     6987           
  Partials      422      422           

[k8s-ci-robot]

@segevfiner: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spowelljr]

@segevfiner Is this still something you're working on?

[spowelljr]

@segevfiner I'm going to close this PR due to long duration without activity, but feel free to reopen if you have time to work on this again.

[bl-ue]

(off-topic: @spowelljr you must have had a reminder or something, didn't you? Because you closed this PR almost exactly 1 week after you last commented, off by only +3 minutes!)

[spowelljr]

@bl-ue That's a really good observation! But actually there's no reminder, every Wednesday 11-12 PST minikube maintainers do through PRs and issues to comment/label/close. The 3 minute difference is somewhat of a coincidence, but would have been within an hour if that makes sense.

[bl-ue]

Yeah that does make sense. Thank you!