Minishift DNS Issues When Host Has Search Domain Set

[InfoSec812]

General information

• Minishift version: v1.18.0+756b128
• OS: Linux
• Hypervisor: VirtualBox

Steps to reproduce

1. On host machine, set a search domain in resolv.conf
2. Start Minishift instance with fresh config (Either CentOS or RHEL)
3. Try to do an S2I build which pulls from GitHub
4. Watch build logs and see that github.com fails to connect/resolve properly

Expected

S2I build SHOULD succeed

Actual

S2I build fails

Workaround

On the host machine, remove the search domain from resolv.conf

[anjannath]

@InfoSec812 I am not able to reproduce this issue, could you please paste your resolv.conf file.
You could also try to use a local dns, that might solve the issue. https://docs.openshift.org/latest/minishift/using/experimental-features.html#local-dns-server

$ oc logs ruby-ex-1-build
---> Installing application source ...
---> Building your Ruby application from source ...
---> Running 'bundle install --retry 2 --deployment --without development:test' ...
Fetching gem metadata from https://rubygems.org/..............
Fetching version metadata from https://rubygems.org/.
Installing puma 3.10.0 with native extensions
Installing rack 2.0.3
Using bundler 1.13.7
Bundle complete! 2 Gemfile dependencies, 3 gems now installed.
Gems in the groups development and test were not installed.
Bundled gems are installed into ./bundle.
---> Cleaning up unused ruby gems ...
Running `bundle clean   --verbose` with bundler 1.13.7
Found no changes, using resolution from the lockfile
Pushing image 172.30.1.1:5000/myproject/ruby-ex:latest ...
Pushed 0/10 layers, 2% complete
Pushed 1/10 layers, 22% complete
Pushed 2/10 layers, 30% complete
Pushed 3/10 layers, 37% complete
Pushed 4/10 layers, 47% complete
Pushed 5/10 layers, 55% complete
Pushed 6/10 layers, 64% complete
Pushed 7/10 layers, 72% complete
Pushed 8/10 layers, 83% complete
Pushed 9/10 layers, 96% complete
Pushed 10/10 layers, 100% complete
Push successful

$ cat /etc/resolv.conf 
; generated by /usr/sbin/dhclient-script
domain anjan.com
search anjan.com localhost example.com
nameserver 1.1.1.1
nameserver 202.83.21.25

[InfoSec812]

Here's what my resolv.conf looked like when I was experiencing issues:

search ad.zanclus.com
nameserver 8.8.8.8
nameserver 8.8.4.4

[InfoSec812]

I noticed that when the containers start up the resolv.conf files have some interesting options:

nameserver 172.30.0.1
search labs-ci-cddeveloper.svc.cluster.local svc.cluster.local cluster.local ad.zanclus.com
options ndots:5

I wonder if options ndots:5 has something to do with it? Could it be that trying to resolve github.com.ad.zanclus.com exceeds the limits?

Just thinking out loud.

[anjannath]

@InfoSec812 You can use the flag --network-nameserver with minishift start to use a different dns server from the vm.

[gbraad]

This is how DNS works. The search add the local domain to traverse until it finds a result. It seems Kubernetes handles the request and therefore I guess a NXDOMAIN is given This is likely a configuration issue, related to `oc cluster up` But I am wondering why you need to set the `search` option. Note: this is also standard behaviour on Hyper-V with the default switch, but resolving does succeed as far as I remember.

…

On Sat, Jul 7, 2018 at 2:55 PM, Anjan Nath ***@***.***> wrote: @InfoSec812 <https://github.com/InfoSec812> You can use the flag --network-nameserver with minishift start to use a different dns server from the vm. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2560 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAHZrWSRDCHFu7ZgTyY55TWtsR8kGgLks5uEFt6gaJpZM4VD37A> .

-- Gerard Braad | http://gbraad.nl [ Doing Open Source Matters ]

[InfoSec812]

@anjannath I have been using the --network-nameserver option. It does not resolve the issue.

@gbraad I'm quite familiar how DNS works as I used to work for DNS.com and have implemented both authoritative and recursive DNS services. The resolution SHOULD be as follows when attempting to resolve github.com with a search parameter as shown in my example:

1. Attempt to resolve github.com.ad.zanclus.com.
2. Attempt to resolve github.com.zanclus.com.
3. Attempt to resolve github.com.com.
4. Attempt to resolve github.com.

It appears that only the first step is actually happening.

[gbraad]

Right, why I believethis is likely a configuration issue (as it shou;dn't stop after the first lookup) Can this be reproduced with a production install of Openshift, or using `oc cluster up` ?

[gbraad]

Note: the options ndots:5 is part of Kubernetes since about 2015 => kubernetes/kubernetes@23caf44#diff-0db82891d463ba14dd59da9c77f4776eR66 (ref: kubernetes/kubernetes#10266)

[InfoSec812]

@gbraad I am having this problem on oc cluster up as well (e.g. Without Minishift). I do not have sufficient hardware at home to create a "true" OpenShift cluster. The machine I am running on is a SuperMicro Dual Xeon with 16 cores and 64GB of RAM. It is running Debian, Kernel 4.13, Docker 18.05.0-ce, and OpenShift client 3.9.0 downloaded from GitHub.

[gbraad]

That is what I guessed. I will try to reproduce it on a OpenShift minimal cluster setup, but that will likely happen after the weekend. I suspect this to be something specific to oc cluster up.

[gbraad]

BTW, that is enough to run a small VM setup (and perhaps even an all-in-one) of OpenShift on. I do this with even an i7 octo, 32GB as that is all that my budget allowed ;-)

[InfoSec812]

Looks like the issue is related to openshift/origin#17316

[InfoSec812]

Since that issue covers all of the other projects (minishift et. al.), I will close this issue and track the work there.

[gbraad]

I would prefer to keep this open

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.