Description
What happened?
First of all, I really don't know too much about the internals, so this might as well(and I hope) be an configuration error on my side.
I am testing a 3 Node Cluster with IPv6 only networking.
I am running 3 virtual machines, with 2 interfaces each, one interface is directly attached to the host interface, and the other interface is connected to a bridge. So ideally all in-cluster communication happens around that bridge.
I have a ULA IPv6 Space(assume fd00:0000:0000:1000::/56) that is configured as follows:
fd00:0000:0000:1000::/59 - nodes
fd00:0000:0000:1020::/59 - Virtual IPs(also API Server)
fd00:0000:0000:1040::/108 - Service CIDR
fd00:0000:0000:10c0::/58 - Pod CIDR
Since everything is connected using said bridge, I configured for each node the following IP addresses and routes on the interface that is connected to the bridge:
node1:
address:
fd00:0000:0000:1000::1/64
routes:
fd00:0000:0000:1000::/59
fd00:0000:0000:1020::/59
node2:
address:
fd00:0000:0000:1001::1/64
routes:
fd00:0000:0000:1000::/59
fd00:0000:0000:1020::/59
node3:
address:
fd00:0000:0000:1002::1/64
routes:
fd00:0000:0000:1000::/59
fd00:0000:0000:1020::/59
Since everything happens on that bridge I thought I might as well try out the pod-to-pod and pod-to-service communication without an overlay, hence I set --advertise-cluster-ip=true and --enable-overlay=false, thinking kube-router would pass around the routes using iBGP.
What did you expect to happen?
I expected the networking to work without an overlay. It does work when I use the overlay.
How can we reproduce the behavior you experienced?
Steps to reproduce the behavior:
Basically a setup like I had above, this is my deployment for kube-router:
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-router-cfg
namespace: kube-system
labels:
tier: node
k8s-app: kube-router
data:
cni-conf.json: |
{
"cniVersion":"0.3.0",
"name":"mynet",
"plugins":[
{
"name":"kubernetes",
"type":"bridge",
"bridge":"kube-bridge",
"isDefaultGateway":true,
"hairpinMode":true,
"ipam":{
"type":"host-local"
}
}
]
}
kubeconfig: |
apiVersion: v1
kind: Config
clusterCIDR: "fd00:0000:0000:10c0::/58"
clusters:
- name: cluster
cluster:
certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
server: https://[fd00:0000:0000:1020::1]:6443
users:
- name: kube-router
user:
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
contexts:
- context:
cluster: cluster
user: kube-router
name: kube-router-context
current-context: kube-router-context
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
k8s-app: kube-router
tier: node
name: kube-router
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: kube-router
tier: node
template:
metadata:
labels:
k8s-app: kube-router
tier: node
spec:
priorityClassName: system-node-critical
serviceAccountName: kube-router
containers:
- name: kube-router
image: docker.io/cloudnativelabs/kube-router
imagePullPolicy: Always
args:
- "--run-router=true"
- "--run-firewall=true"
- "--run-service-proxy=true"
- "--enable-ipv4=false"
- "--enable-ipv6=true"
- "--service-cluster-ip-range=fd00:0000:0000:1040::/108"
- "--bgp-graceful-restart=true"
- "--kubeconfig=/var/lib/kube-router/kubeconfig"
- "--router-id=generate"
- "--advertise-cluster-ip=true"
- "--enable-overlay=false"
...
System Information (please complete the following information)
- Kube-Router Version (
kube-router --version): Running kube-router version v2.5.0, built on 2025-02-14T20:21:02+0000, go1.23.6 - Kube-Router Parameters: See deployment
- Kubernetes Version (
kubectl version) : v1.32.0 - Cloud Type: on premise
- Kubernetes Deployment Type: Talos
- Kube-Router Deployment Type: DaemonSet
- Cluster Size: 3 Nodes