[BUG] host Pod Failed to Start Due to CoreDNS DNS Timeout after reboot

[elvizlai]

问题描述/What happened:

[root@cloud-mgr ~]# kubectl logs -f default-host-2mgtp
Defaulted container "host" out of: host, ovn-controller, sdnagent
[debug 260603 14:42:28 db.newTableSpec(tablespec.go:77)] table hostdmesg_log_tbl maxDuration 720 hour maxSegements 6
[info 260603 14:42:28 procutils.WaitZombieLoop(zombie_others.go:36)] My pid is not 1 and no need to wait zombies
[info 260603 14:42:28 options.parseOptions(options.go:354)] Use configuration file: /etc/yunion/host.conf
[warning 260603 14:42:28 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1215)] Cannot find argument enable-rbac
[warning 260603 14:42:28 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1215)] Cannot find argument health-driver
[warning 260603 14:42:28 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1215)] Cannot find argument disk-is-ssd
[warning 260603 14:42:28 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1215)] Cannot find argument enable-qmp-monitor
[warning 260603 14:42:28 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1215)] Cannot find argument enable-health-checker
[warning 260603 14:42:28 structarg.(*ArgumentParser).parseJSONKeyValue(structarg.go:1215)] Cannot find argument start-host-ignore-sys-error
[info 260603 14:42:28 options.parseOptions(options.go:377)] Set log level to "info"
[info 2026-06-03 14:42:28 options.parseOptions(options.go:354)] Use configuration file: /etc/yunion/common/common.conf
[info 2026-06-03 14:42:28 options.parseOptions(options.go:377)] Set log level to "info"
[info 2026-06-03 14:42:28 hostman.(*SHostService).InitService(host_services.go:69)] exec socket path: /var/run/onecloud/exec.sock
[info 2026-06-03 14:42:28 app.InitApp(app.go:32)] RequestWorkerCount: 8
[info 2026-06-03 14:42:28 appsrv.NewApplication(appsrv.go:122)] App hostId: 38UXWy0UyWyJFSpoXpbVECBtSoU= (host,anode-04.icc.local,172.20.1.118)
2026/06/03 14:42:28 Allow hosts []
[info 2026-06-03 14:42:28 appsrv.(*Application).SetDefaultTimeout(appsrv.go:138)] adjust application default timeout to 60.000000 seconds
[error 2026-06-03 14:42:28 auth.(*authManager).startRefreshRevokeTokens(auth.go:193)] refreshRevokeTokens: No valid admin token credential
Post "https://default-keystone:30357/v3/auth/tokens": dial tcp: lookup default-keystone on 10.96.0.10:53: read udp 10.96.0.10:55199->10.96.0.10:53: i/o timeout
[error 2026-06-03 14:43:18 auth.(*authManager).authAdmin(auth.go:269)] Admin auth failed: {"error":{"class":"DNSError","code":499,"details":"Post \"https://default-keystone:30357/v3/auth/tokens\": dial tcp: lookup default-keystone on 10.96.0.10:53: read udp 10.96.0.10:55199->10.96.0.10:53: i/o timeout","request":{"body":"{\"auth\":{\"context\":{\"source\":\"srv\"},\"identity\":{\"methods\":[\"password\"],\"password\":{\"user\":{\"domain\":...ult\"},\"name\":\"system\"}}}}","headers":{"Content-Length":"238","Content-Type":"application/json","User-Agent":"yunioncloud-go/201708","X-Yunion-Parent-Id":"","X-Yunion-Peer-Service-Name":"host","X-Yunion-Remote-Addr":"default-keystone:30357","X-Yunion-Span-Id":"0","X-Yunion-Span-Name":"","X-Yunion-Strace-Debug":"true","X-Yunion-Strace-Id":"9edad442"},"method":"POST","url":"https://default-keystone:30357/v3/auth/tokens"}}}
goroutine 1 [running]:
runtime/debug.Stack()
	/usr/lib/go/src/runtime/debug/stack.go:24 +0x64
runtime/debug.PrintStack()
	/usr/lib/go/src/runtime/debug/stack.go:16 +0x1c
yunion.io/x/log.Fatalf({0x2fd3be1, 0x19}, {0x4001047c08, 0x1, 0x1})
	/root/go/src/yunion.io/x/onecloud/vendor/yunion.io/x/log/log.go:138 +0x30
yunion.io/x/onecloud/pkg/mcclient/auth.AsyncInit(0x4001752fc0, 0x30?, 0x35?, {0x40018743e0?, 0x2140c?}, {0x4001874420?, 0x20?}, 0x0)
	/root/go/src/yunion.io/x/onecloud/pkg/mcclient/auth/auth.go:448 +0xe0
yunion.io/x/onecloud/pkg/mcclient/auth.Init(...)
	/root/go/src/yunion.io/x/onecloud/pkg/mcclient/auth/auth.go:455
yunion.io/x/onecloud/pkg/cloudcommon/app.InitAuth(0x575bc70, 0x3128b68)
	/root/go/src/yunion.io/x/onecloud/pkg/cloudcommon/app/auth.go:75 +0x320
yunion.io/x/onecloud/pkg/hostman.(*SHostService).RunService(0x4000a998b0?)
	/root/go/src/yunion.io/x/onecloud/pkg/hostman/host_services.go:88 +0xf4
yunion.io/x/onecloud/pkg/cloudcommon/service.(*SServiceBase).StartService(0x4000a639f8)
	/root/go/src/yunion.io/x/onecloud/pkg/cloudcommon/service/services.go:58 +0xdc
yunion.io/x/onecloud/pkg/hostman.StartService(...)
	/root/go/src/yunion.io/x/onecloud/pkg/hostman/host_services.go:214
main.main()
	/root/go/src/yunion.io/x/onecloud/cmd/host/main.go:30 +0xf0
[fatal 2026-06-03 14:43:18 auth.AsyncInit(auth.go:448)] Auth manager init err: authAdmin: {"error":{"class":"DNSError","code":499,"details":"Post \"https://default-keystone:30357/v3/auth/tokens\": dial tcp: lookup default-keystone on 10.96.0.10:53: read udp 10.96.0.10:55199->10.96.0.10:53: i/o timeout","request":{"body":"{\"auth\":{\"context\":{\"source\":\"srv\"},\"identity\":{\"methods\":[\"password\"],\"password\":{\"user\":{\"domain\":...ult\"},\"name\":\"system\"}}}}","headers":{"Content-Length":"238","Content-Type":"application/json","User-Agent":"yunioncloud-go/201708","X-Yunion-Parent-Id":"","X-Yunion-Peer-Service-Name":"host","X-Yunion-Remote-Addr":"default-keystone:30357","X-Yunion-Span-Id":"0","X-Yunion-Span-Name":"","X-Yunion-Strace-Debug":"true","X-Yunion-Strace-Id":"9edad442"},"method":"POST","url":"https://default-keystone:30357/v3/auth/tokens"}}}

环境/Environment:

• OS (e.g. cat /etc/os-release):
  NAME="openEuler"
  VERSION="22.03 (LTS-SP3)"
  ID="openEuler"
  VERSION_ID="22.03"
  PRETTY_NAME="openEuler 22.03 (LTS-SP3)"
  ANSI_COLOR="0;31"

• Kernel (e.g. uname -a):
  Linux eda-cloud-mgr.icc.local 5.10.0-182.0.0.95.oe2203sp3.x86_64 scheduler: fix specified network not enough still pass. #1 SMP Sat Dec 30 13:10:36 CST 2023 x86_64 x86_64 x86_64 GNU/Linux

• Host: (e.g. dmidecode | egrep -i 'manufacturer|product' |sort -u)
  Manufacturer: QEMU
  Product Name: Standard PC (i440FX + PIIX, 1996)

• Service Version (e.g. kubectl exec -n onecloud $(kubectl get pods -n onecloud | grep climc | awk '{print $1}') -- climc version-list):

其中，历史遗留问题已处理：
https://www.cloudpods.org/v3.11/docs/onpremise/operations/platform-issues/dns

也有查看这个文档：
https://www.cloudpods.org/v3.11/docs/onpremise/operations/k8s/dnserror

其中：（openvswitch 重启前 status 还有，重启后好像不用了？）
[root@anode-03 ~]# systemctl status openvswitch
○ openvswitch.service - Open vSwitch
     Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; disabled; vendor preset: disabled)
     Active: inactive (dead)
[root@anode-03 ~]#
[root@anode-03 ~]# ipvsadm -Ln | grep -A 3 10.96.0.10
TCP  10.96.0.10:53 rr
  -> 10.40.136.120:53             Masq    1      0          0
TCP  10.96.0.10:9153 rr
  -> 10.40.136.120:9153           Masq    1      0          0
TCP  10.97.86.26:8000 rr
  -> 172.20.2.100:8000            Masq    1      0          0
--
UDP  10.96.0.10:53 rr
  -> 10.40.136.120:53             Masq    1      0          434

[root@anode-03 ~]# ip route | grep 10.40.136
10.40.136.64/26 via 172.20.1.200 dev tunl0 proto bird onlink


kubectl -n kube-system edit daemonset calico-node
        - name: IP_AUTODETECTION_METHOD
          value: can-reach=172.20.1.200

[swordqiu]

@elvizlai 重启后openvswitch未自动启动是正常的，因为3.x版本的openvswitch是host正常启动后才会被host拉起。现在host卡在keystone认证，还未到拉起openvswitch的步骤。根本问题还是容器网络连通性，host无法通过容器网络访问coredns服务（10.96.0.10:53），无法解析keystone的地址。如果是之前一直正常，突然出现这个问题。主要排查思路是排查是否应用了新的iptables规则，改变了网络ACL等。

[elvizlai]

集群里有n台机器，只有重启的遇到了这个问题。（试了2台，都如此）其他未重启的正常，暂时没敢动其他机器了。

有什么比较有效的排查思路吗？

[elvizlai]

补充一下版本信息（从之前版本升级上来的）

climc version-list
Get "172.20.2.100/version": unsupported protocol scheme ""
Get "https://172.20.1.200:30443/version": dial tcp 172.20.1.200:30443: connect: connection refused
+---------------+-------------------------------------------------------------------------------------------------------------------+
|     Field     |                                                       Value                                                       |
+---------------+-------------------------------------------------------------------------------------------------------------------+
| ansible       | release/3.11.13(c6d5c9ea026021209)                                                                                |
| cloudevent    | release/3.11.13(c6d5c9ea026021209)                                                                                |
| cloudid       | release/3.11.13(c6d5c9ea026021209)                                                                                |
| cloudmon      | release/3.11.13(c6d5c9ea026021209)                                                                                |
| cloudproxy    | release/3.11.13(c6d5c9ea026021209)                                                                                |
| compute_v2    | eda-v3.11.13(0e74c671f26041803)                                                                                   |
| devtool       | release/3.11.13(c6d5c9ea026021209)                                                                                |
| dns           | {"error":{"class":"ClientError","code":499,"details":"Get \"172.20.2.100/version\": unsupported protocol scheme \ |
|               | "\"","request":{"headers":{"User-Agent":"yunioncloud-go/201708","X-Auth-Token":"*"},"method":"GET","url":"172.20. |
|               | 2.100/version"}}}                                                                                                 |
| identity      | release/3.11.13(c6d5c9ea026021209)                                                                                |
| image         | release/3.11.13(c6d5c9ea026021209)                                                                                |
| k8s           | heads/v3.11.13-20260211.1(7a30833a26021208)                                                                       |
| log           | release/3.11.13(c6d5c9ea026021209)                                                                                |
| monitor       | release/3.11.13(c6d5c9ea026021209)                                                                                |
| notify        | release/3.11.13(c6d5c9ea026021209)                                                                                |
| scheduledtask | release/3.11.13(c6d5c9ea026021209)                                                                                |
| scheduler     | eda-v3.11.13(0e74c671f26041803)                                                                                   |
| vpcagent      | release/3.11.13(c6d5c9ea026021209)                                                                                |
| webconsole    | release/3.11.13(c6d5c9ea026021209)                                                                                |
| yunionconf    | release/3.11.13(c6d5c9ea026021209)                                                                                |
+---------------+-------------------------------------------------------------------------------------------------------------------+

[elvizlai]

节点应该还是保持原有的 ip-in-ip 模式，没有随着版本升级变更为 VXLAN

[root@anode-04 ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp125s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:4f:a6:9e:d3:c3 brd ff:ff:ff:ff:ff:ff
3: enp125s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:4f:a6:9e:d3:c4 brd ff:ff:ff:ff:ff:ff
4: enp125s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:4f:a6:9e:d3:c5 brd ff:ff:ff:ff:ff:ff
5: enp125s0f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:4f:a6:9e:d3:c6 brd ff:ff:ff:ff:ff:ff
6: enp1s0f0np0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
    link/ether 20:fa:db:b1:a7:98 brd ff:ff:ff:ff:ff:ff
7: enp1s0f1np1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 20:fa:db:b1:a7:99 brd ff:ff:ff:ff:ff:ff
8: enp2s0f0np0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
    link/ether 20:fa:db:b1:a7:98 brd ff:ff:ff:ff:ff:ff permaddr 20:fa:db:a6:e9:de
9: enp2s0f1np1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 20:fa:db:b1:a7:99 brd ff:ff:ff:ff:ff:ff permaddr 20:fa:db:a6:e9:df
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 20:fa:db:b1:a7:99 brd ff:ff:ff:ff:ff:ff
11: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 20:fa:db:b1:a7:98 brd ff:ff:ff:ff:ff:ff
12: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
    link/ether d6:c3:f4:52:5d:df brd ff:ff:ff:ff:ff:ff
13: calife946093f30@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-e6214586-306b-c88a-954c-d85cf96c80f3
14: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0

[elvizlai]

[root@cloud-mgr ~]# kubectl -n kube-system get svc kube-dns -o wide
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE    SELECTOR
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   568d   k8s-app=kube-dns

[root@cloud-mgr ~]# kubectl -n kube-system get endpoints kube-dns -o wide
NAME       ENDPOINTS                                           AGE
kube-dns   10.40.136.92:53,10.40.136.92:53,10.40.136.92:9153   568d


正常节点，有返回
[root@anode-02 ~]# dig @10.96.0.10 kubernetes.default.svc.cluster.local +short
10.96.0.1

[root@anode-02 ~]# dig @10.40.136.92 kubernetes.default.svc.cluster.local +short
10.96.0.1

重启后的异常节点，以上两个命令都是 timeout

在 anode04 启动一个 pod 用来测试
kubectl run dns-test-anode04 \
  -n default \
  --image=docker.io/rancher/mirrored-library-busybox:1.36.1 \
  --restart=Never \
  --overrides='{"spec":{"nodeName":"anode-04.icc.local"}}' \
  -- sleep 3600

kubectl -n default exec -it dns-test-anode04 -- nslookup kubernetes.default.svc.cluster.local 10.40.136.92

# 没有回包？
[root@cloud-mgr ~]# tcpdump -ni any 'host 10.40.64.26 and host 10.40.136.92 and udp port 53'
tcpdump: data link type LINUX_SLL2
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

10:17:49.584134 tunl0 In  IP 10.40.64.26.59130 > 10.40.136.92.domain: 13330+ A? kubernetes.default.svc.cluster.local. (54)
10:17:49.584145 tunl0 In  IP 10.40.64.26.59130 > 10.40.136.92.domain: 13830+ AAAA? kubernetes.default.svc.cluster.local. (54)
10:17:52.086776 tunl0 In  IP 10.40.64.26.59130 > 10.40.136.92.domain: 13330+ A? kubernetes.default.svc.cluster.local. (54)
10:17:52.086788 tunl0 In  IP 10.40.64.26.59130 > 10.40.136.92.domain: 13830+ AAAA? kubernetes.default.svc.cluster.local. (54)

[elvizlai]

Fix with this command

ethtool -K bond1 tx off tso off gso off gro off
ethtool -K enp1s0f0np0 tx off tso off gso off gro off
ethtool -K enp2s0f0np0 tx off tso off gso off gro off

But the config in /etc/yunion/host.conf already has set

ethtool_enable_gso: false

which looks not working for new version.

[swordqiu]

@elvizlai 从3.11开始，ethtool_enable_gso默认为true，你是从哪个版本升级到3.11.13？

[elvizlai]

@elvizlai 从3.11开始，ethtool_enable_gso默认为true，你是从哪个版本升级到3.11.13？

cloud-mgr.icc.local Ready control-plane,master 568d v1.28.5+k3s1

古老的版本，中间经历过多次升级。node 节点也基本没有重启过，恰好这次硬件维护，才发现问题。

看了下 anode02（本次对照组里正常的节点），uptime 为 169 days。也就是这个之前的那个版本重启是正常的也不能这么说，这次是机房断电重启。v3.11.9 可能还是正常的，仅供参考。