How to chage a IP range

[niwashing]

IP range 10.1.1.0/24, 10.152.183.0/24 are userd for cluster or pods by default.
How can I change a cluster IP and IP range that is assigned to nodes?

[ktsakalozos]

Hi @niwashing

These two IP ranges are configured in a couple places:

10.152.183.0/24:

• /var/snap/microk8s/current/args/kube-apiserver: --service-cluster-ip-range=10.152.183.0/24
• /var/snap/microk8s/current/args/kubelet: --non-masquerade-cidr=10.152.183.0/24
• /var/snap/microk8s/current/args/kube-proxy: --cluster-cidr=10.152.183.0/24

10.1.1.0/24:

• /var/snap/microk8s/current/args/cni-network/cni.conf "subnet": "10.1.1.0/24",
• /var/snap/microk8s/current/args/kubelet:--pod-cidr=10.1.1.0/24

I hope I am not missing anything. Remember to stop/start MicroK8s after you update any of those arguments.

Thank you for using MicroK8s

[niwashing]

@ktsakalozos

Thank you! I could change ip range, but microk8s.enable dns still throw error as follows:

$ microk8s.enable dns
Enabling DNS
Applying manifest
serviceaccount/kube-dns unchanged
configmap/kube-dns unchanged
deployment.extensions/kube-dns configured
The Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.152.183.10": provided IP is not in the valid range. The range of valid IPs is 172.22.183.0/24
Failed to enable dns

Although I found /snap/microk8s/354/actions/dns.yaml and /snap/microk8s/354/actions/enable.dns.sh still include a default IP range, they cannot be changed due to snap permission.

Do I have to build microk8s from source by using snapcraft?

[ktsakalozos]

You have the option to recompile MicroK8s and produce your own .snap file.

I suspect that after microk8s.enable dns you can also microk8s.kubectl edit the part of the dns manifest that is failing.

[niwashing]

I suspect that after microk8s.enable dns you can also microk8s.kubectl edit the part of the dns >manifest that is failing.

Sorry, I'm not very familier with kubernetes,
but kubectl edit is not available because kube-dns has not been deployed yet due to ip range error?

I would appreciate if you could show me entier commands.

[ktsakalozos]

Sure, here is what I have:

> microk8s.kubectl get all --all-namespaces
NAMESPACE     NAME                            READY   STATUS    RESTARTS   AGE
kube-system   pod/kube-dns-6ccd496668-qx5m4   3/3     Running   0          41s

NAMESPACE     NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes   ClusterIP   10.152.183.1    <none>        443/TCP         72s
kube-system   service/kube-dns     ClusterIP   10.152.183.10   <none>        53/UDP,53/TCP   41s

NAMESPACE     NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/kube-dns   1/1     1            1           41s

NAMESPACE     NAME                                  DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/kube-dns-6ccd496668   1         1         1       42s

At this point I have kube-dns running, it should be failing in your case. I suspect you can go and edit the kube-dns service clusterIP with:

microk8s.kubectl edit -n kube-system service/kube-dns

If this is does not work you will need to download and edit this file https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/dns.yaml and then microk8s.kubectl apply -f ./dns.yaml

[fquinner]

I can confirm I just hit this too because i didn't enable dns before adding pods so there was an IP conflict. I can also confirm that the installation of the dns.yaml definition was required since dns never started.

[AdamIsrael]

What happens when the microk8s snap refreshes? Since the files to be edited are in /var/snap/microk8s/current/, I suspect the changes will revert back to the defaults.

What about storing the subnet to use in /var/snap/microk8s/common, and modifying the configuration to get the value from that file?

[ktsakalozos]

@AdamIsrael I do not think this is what happens during a refresh. ${SNAP_DATA} is backed up and could be reverted (contrary to ${SNAP_COMMON}) but the contents are preserved. If you configure your daemons in a specific way we respect your configuration, we do not overwrite the configuration with the defaults. Have a look here https://forum.snapcraft.io/t/proper-way-to-simulate-a-snap-refresh-release/5565 on how you can simulate a refresh and check this yourself.

[AdamIsrael]

@ktsakalozos That's good to know, thanks!

I still think having the subnet defined in a single location would be better. Also perhaps a command for changing it to a specific or random subnet?

The specific use-case I have is that I may install microk8s to lxd, and thus have multiple microk8s on a host. I can create new networks in lxd for each, but I then need for each microk8s to use the appropriate network.

[evilhamsterman]

I agree with @AdamIsrael, I started playing with K8s and decided that since we were and Ubuntu shop microk8s would be a great way to get started. However we also happen to use the subnet 10.1.1.0/24 and so the cbr0 interface caused issues for me accessing portions of our network.

[camille-rodriguez]

Hello everyone, @ktsakalozos,
I followed these steps to deploy microk8s on a different range (simply 10.152.182.0/24). However, I run into a similar issue when trying to enable istio. It tries to re-deploy kube-dns (even if it is already deployed) and gets stuck because of the clusterIP spec again.

$ microk8s.kubectl get all --all-namespaces
NAMESPACE            NAME                                        READY   STATUS             RESTARTS   AGE
container-registry   pod/registry-d7d7c8bc9-g86qw                0/1     Pending            0          15h
kube-system          pod/coredns-9b8997588-v2hs6                 0/1     Running            3          16h
kube-system          pod/hostpath-provisioner-7b9cb5cdb4-c2z2l   0/1     CrashLoopBackOff   22         15h
kube-system          pod/kube-dns-579bd8fb8d-gh2m6               0/3     InvalidImageName   0          15h

NAMESPACE            NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
container-registry   service/registry     NodePort    10.152.182.65   <none>        5000:32000/TCP   15h
default              service/kubernetes   ClusterIP   10.152.182.1    <none>        443/TCP          17h
kube-system          service/kube-dns     ClusterIP   10.152.182.10   <none>        53/UDP,53/TCP    15h

NAMESPACE            NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
container-registry   deployment.apps/registry               0/1     1            0           15h
kube-system          deployment.apps/coredns                0/1     1            0           16h
kube-system          deployment.apps/hostpath-provisioner   0/1     1            0           15h
kube-system          deployment.apps/kube-dns               0/1     1            0           15h

NAMESPACE            NAME                                              DESIRED   CURRENT   READY   AGE
container-registry   replicaset.apps/registry-d7d7c8bc9                1         1         0       15h
kube-system          replicaset.apps/coredns-9b8997588                 1         1         0       16h
kube-system          replicaset.apps/hostpath-provisioner-7b9cb5cdb4   1         1         0       15h
kube-system          replicaset.apps/kube-dns-579bd8fb8d               1         1         0       15h
 
$ microk8s.enable istio
Enabling Istio
Enabling DNS
Applying manifest
serviceaccount/coredns unchanged
configmap/coredns unchanged
deployment.apps/coredns unchanged
clusterrole.rbac.authorization.k8s.io/coredns unchanged
clusterrolebinding.rbac.authorization.k8s.io/coredns unchanged
The Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.152.183.10": field is immutable
Failed to enable dns
Failed to enable istio

Could you recommend what I should do with this, or which files I should download and edit? Thank you!

[sadoMasupilami]

Hello,
this problem is still relevant now with flannel.
we do a lot of work from home (using VPN) and many of our internal services run in 10.1.0.0/16

is there any way to change the adresses with flannel?
ui tried to change /var/snap/microk8s/current/args/flannel-network-mgr-config and after a reset and reboot the new range is used. But sometimes it just stops working as flannel daemon can not start with:

Jul 02 09:50:48 xxx microk8s.daemon-flanneld[1747]: error #0: dial tcp: lookup none on 127.0.0.53:53: server misbehaving
Jul 02 09:50:53 xxx microk8s.daemon-flanneld[2174]: Error: dial tcp: lookup none on 127.0.0.53:53: server misbehaving

we want to replace minikube with microk8s for local development on linux. So it would be really important to us.

Thanks for any good tips

[strigona-worksight]

Microk8s 1.16+

Modify:

/var/snap/microk8s/current/args/flannel-network-mgr-config

Change: "10.1.0.0/16" to: "10.8.0.0/16" (or any other range)

Then restart microk8s:

microk8s.stop

microk8s.start

[uGiFarukh]

Right now, what is the best option to achieve this local IP range change for the latest 1.19 microk8s without breaking any addons or functionality? And is it possible to use custom ip ranges outside of the standard ip ranges used in local? for example: 1.1.1.1? Or will this conflict with the internet?

[uGiFarukh]

@ktsakalozos any idea on how to achieve this properly without breaking anything?