Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v0.10.0 k3s-agent will not start on a rasbian-based (kernel 4.19.75-v7l+) raspberry pi4 node #939

Closed
billimek opened this issue Oct 23, 2019 · 7 comments
Closed

v0.10.0 k3s-agent will not start on a rasbian-based (kernel 4.19.75-v7l+) raspberry pi4 node #939

billimek opened this issue Oct 23, 2019 · 7 comments
Milestone
v1.x - Backlog

Comments

@billimek
Copy link

billimek commented Oct 23, 2019 •
edited
Loading

Version: v0.10.0 (f9888ca)

Describe the bug
k3s-agent will not start on a rasbian-based (kernel 4.19.75-v7l+) raspberry pi4 node

To Reproduce
Attempt to join a rpi4 node to a k3s server, using the following command,

curl -sfL https://get.k3s.io | K3S_URL=https://k3s-0:6443 K3S_TOKEN=$NODE_TOKEN INSTALL_K3S_VERSION='v0.10.0' sh -s - --node-taint arm=true:NoExecute --data-dir /mnt/usb/var/lib/rancher

Expected behavior
The node will properly join

Actual behavior
The node fails to join and the k3s-agent service fails to start

Additional context

  • The same symptoms occurred with both an upgrade attempt as well as a fresh clean install
  • Joining amd64-based nodes to the same k3s master node don't exhibit the same behavior and work properly
  • The same behavior occurs with both an agent upgrade attempt or a complete fresh install of the agent
  • This issue was not observed on the same node running k3s v0.9.1

Here is the (long) syslog output from the k3s-agent crash event that occurs over and over:

Oct 23 11:43:49 pi4-a systemd[1]: Started Lightweight Kubernetes.
Oct 23 11:43:50 pi4-a k3s[11584]: time="2019-10-23T11:43:50.489503342Z" level=info msg="Starting k3s agent v0.10.0 (f9888ca3)"
Oct 23 11:43:50 pi4-a k3s[11584]: time="2019-10-23T11:43:50.490421504Z" level=info msg="Running load balancer 127.0.0.1:40501 -> [10.2.0.30:6443 k3s-0:6443]"
Oct 23 11:43:50 pi4-a k3s[11584]: time="2019-10-23T11:43:50.840536333Z" level=info msg="Logging containerd to /mnt/usb/var/lib/rancher/agent/containerd/containerd.log"
Oct 23 11:43:50 pi4-a k3s[11584]: time="2019-10-23T11:43:50.841279851Z" level=info msg="Running containerd -c /mnt/usb/var/lib/rancher/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /mnt/usb/var/lib/rancher/agent/containerd"
Oct 23 11:43:50 pi4-a k3s[11584]: time="2019-10-23T11:43:50.844693508Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused\""
Oct 23 11:43:51 pi4-a k3s[11584]: time="2019-10-23T11:43:51.848635904Z" level=info msg="module br_netfilter was already loaded"
Oct 23 11:43:51 pi4-a k3s[11584]: time="2019-10-23T11:43:51.848840917Z" level=info msg="module overlay was already loaded"
Oct 23 11:43:51 pi4-a k3s[11584]: time="2019-10-23T11:43:51.848891231Z" level=info msg="module nf_conntrack was already loaded"
Oct 23 11:43:51 pi4-a k3s[11584]: time="2019-10-23T11:43:51.874483375Z" level=info msg="Connecting to proxy" url="wss://10.2.0.30:6443/v1-k3s/connect"
Oct 23 11:43:51 pi4-a k3s[11584]: time="2019-10-23T11:43:51.890344426Z" level=warning msg="Disabling CPU quotas due to missing cpu.cfs_period_us"
Oct 23 11:43:51 pi4-a k3s[11584]: time="2019-10-23T11:43:51.890766490Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/mnt/usb/var/lib/rancher/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/3f43b16ca97dbb7ba58868cdb2137a72ad7215762a2852ed944237bf45d44f07/bin --cni-conf-dir=/mnt/usb/var/lib/rancher/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --cpu-cfs-quota=false --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=pi4-a --kubeconfig=/mnt/usb/var/lib/rancher/agent/kubelet.kubeconfig --kubelet-cgroups=/systemd/system.slice --node-labels= --read-only-port=0 --register-with-taints=arm=true:NoExecute --resolv-conf=/etc/resolv.conf --runtime-cgroups=/systemd/system.slice --serialize-image-pulls=false --tls-cert-file=/mnt/usb/var/lib/rancher/agent/serving-kubelet.crt --tls-private-key-file=/mnt/usb/var/lib/rancher/agent/serving-kubelet.key"
Oct 23 11:43:51 pi4-a k3s[11584]: W1023 11:43:51.891468   11584 server.go:208] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
Oct 23 11:43:51 pi4-a k3s[11584]: time="2019-10-23T11:43:51.906559654Z" level=info msg="waiting for node pi4-a: nodes \"pi4-a\" not found"
Oct 23 11:43:51 pi4-a systemd[1]: Started Kubernetes systemd probe.
Oct 23 11:43:51 pi4-a k3s[11584]: I1023 11:43:51.936465   11584 server.go:406] Version: v1.16.2-k3s.1
Oct 23 11:43:51 pi4-a systemd[1]: run-r2a96a18e910d4f50aadf1ddba59a7657.scope: Succeeded.
Oct 23 11:43:51 pi4-a k3s[11584]: E1023 11:43:51.965964   11584 node.go:124] Failed to retrieve node info: nodes "pi4-a" not found
Oct 23 11:43:52 pi4-a k3s[11584]: E1023 11:43:52.020130   11584 machine.go:288] failed to get cache information for node 0: open /sys/devices/system/cpu/cpu0/cache: no such file or directory
Oct 23 11:43:52 pi4-a k3s[11584]: fatal: morestack on g0
Oct 23 11:43:52 pi4-a k3s[11584]: SIGSEGV: segmentation violation
Oct 23 11:43:52 pi4-a k3s[11584]: PC=0x71118 m=12 sigcode=1
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 0 [idle]:
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.abort()
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/asm_arm.s:801 +0x4
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.badmorestackg0()
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/proc.go:438 +0x28
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.mstart()
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/proc.go:1146
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 346 [running, locked to thread]:
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.systemstack_switch()
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/asm_arm.s:298 +0x4 fp=0x61baecc sp=0x61baec8 pc=0x6f344
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.throw(0x2ebca09, 0x2d)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/panic.go:767 +0x40 fp=0x61baee0 sp=0x61baecc pc=0x4133c
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.exitsyscall()
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/proc.go:2957 +0x294 fp=0x61baef8 sp=0x61baee0 pc=0x4a2e0
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.cgocallbackg(0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/cgocall.go:188 +0x74 fp=0x61baf28 sp=0x61baef8 pc=0x12ad4
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.cgocallback_gofunc(0x0, 0x800, 0xa395c, 0x7ffff800)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/asm_arm.s:746 +0x74 fp=0x61baf34 sp=0x61baf28 pc=0x7107c
Oct 23 11:43:52 pi4-a k3s[11584]: runtime: unexpected return pc for syscall.Syscall called from 0x0
Oct 23 11:43:52 pi4-a k3s[11584]: stack: frame={sp:0x61baf34, fp:0x61baf38} stack=[0x61b8000,0x61bc000)
Oct 23 11:43:52 pi4-a k3s[11584]: 061baeb4:  001133c4 <fmt.(*pp).handleMethods+320>  00000000  00000000  00000000
Oct 23 11:43:52 pi4-a k3s[11584]: 061baec4:  05cea090  0004133c <runtime.throw+64>  0004a2e0 <runtime.exitsyscall+660>  061baed4
Oct 23 11:43:52 pi4-a k3s[11584]: 061baed4:  0006dcc4 <runtime.throw.func1+0>  02ebca09  0000002d  00012ad4 <runtime.cgocallbackg+116>
Oct 23 11:43:52 pi4-a k3s[11584]: 061baee4:  02ebca09  0000002d  00010fb4 <runtime/internal/atomic.(*spinlock).lock+36>  00011098 <runtime/internal/atomic.goCas64+88>
Oct 23 11:43:52 pi4-a k3s[11584]: 061baef4:  05413a88  0007107c <runtime.cgocallback_gofunc+116>  00000001  00011128 <runtime/internal/atomic.goCas64+232>
Oct 23 11:43:52 pi4-a k3s[11584]: 061baf04:  0008c6a8 <syscall.Syscall+8>  00000000  00000000  00000000
Oct 23 11:43:52 pi4-a k3s[11584]: 061baf14:  00000000  00000000  00000000  00000000
Oct 23 11:43:52 pi4-a k3s[11584]: 061baf24:  0616eee0  0008c6a8 <syscall.Syscall+8>  00000000  05861300
Oct 23 11:43:52 pi4-a k3s[11584]: 061baf34: <00000000 >00000000  00000800  000a395c <internal/poll.(*FD).Write+648>
Oct 23 11:43:52 pi4-a k3s[11584]: 061baf44:  7ffff800  00000004  0585418c  000a3958 <internal/poll.(*FD).Write+644>
Oct 23 11:43:52 pi4-a k3s[11584]: 061baf54:  05854180  00000000  061baf00  000aa254 <os.(*File).Write+92>
Oct 23 11:43:52 pi4-a k3s[11584]: 061baf64:  05854180  06272100  0000009d  000000ff
Oct 23 11:43:52 pi4-a k3s[11584]: 061baf74:  0000009d  00000000  00000000  00000000
Oct 23 11:43:52 pi4-a k3s[11584]: 061baf84:  0000a4f8  00000000  00000004  000c0000 <strings.(*byteStringReplacer).WriteString+544>
Oct 23 11:43:52 pi4-a k3s[11584]: 061baf94:  061baf60  000a3754 <internal/poll.(*FD).Write+128>  00000000  00000000
Oct 23 11:43:52 pi4-a k3s[11584]: 061bafa4:  00000000  05854180  05854194  005cd130 <k8s.io/klog.(*loggingT).output+1388>
Oct 23 11:43:52 pi4-a k3s[11584]: 061bafb4:  00010fec <runtime/internal/atomic.Xadd+40>
Oct 23 11:43:52 pi4-a k3s[11584]: syscall.Syscall(0x800, 0xa395c, 0x7ffff800, 0x4, 0x585418c, 0xa3958, 0x5854180)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/syscall/asm_linux_arm.s:14 +0x8 fp=0x61baf38 sp=0x61baf34 pc=0x8c6a8
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/rancher/k3s/pkg/daemons/agent.kubelet
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/daemons/agent/agent.go:143 +0x610
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 1 [sleep]:
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.goparkunlock(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/proc.go:310
Oct 23 11:43:52 pi4-a k3s[11584]: time.Sleep(0x77359400, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/time.go:105 +0x158
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/k3s/pkg/agent/flannel.Run(0x36d26e0, 0x6104760, 0x5ca01a0, 0x66ec790, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/agent/flannel/setup.go:100 +0x9c
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/k3s/pkg/agent.run(0x36d26e0, 0x6104760, 0x580a08a, 0x6a, 0x0, 0x0, 0x62e10c0, 0x17, 0x0, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/agent/run.go:68 +0x218
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/k3s/pkg/agent.Run(0x36d26e0, 0x6104760, 0x580a08a, 0x6a, 0x0, 0x0, 0x62e10c0, 0x17, 0x0, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/agent/run.go:130 +0x2a4
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/k3s/pkg/cli/agent.Run(0x5fd6370, 0x51bc7d8, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/cli/agent/agent.go:81 +0x234
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/urfave/cli.HandleAction(0x2923c68, 0x2fd531c, 0x5fd6370, 0x5fd6370, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/urfave/cli@v1.21.0/app.go:514 +0xac
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/urfave/cli.Command.Run(0x2e13ab9, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2e2ae1c, 0xe, 0x62e0fa0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/urfave/cli@v1.21.0/command.go:171 +0x370
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/urfave/cli.(*App).Run(0x5b6c9a0, 0x58541c0, 0x7, 0x8, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/urfave/cli@v1.21.0/app.go:265 +0x510
Oct 23 11:43:52 pi4-a k3s[11584]: main.main()
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/cmd/server/main.go:46 +0x2ec
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 6 [syscall]:
Oct 23 11:43:52 pi4-a k3s[11584]: os/signal.signal_recv(0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/sigqueue.go:147 +0x130
Oct 23 11:43:52 pi4-a k3s[11584]: os/signal.loop()
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/signal/signal_unix.go:23 +0x14
Oct 23 11:43:52 pi4-a k3s[11584]: created by os/signal.init.0
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/signal/signal_unix.go:29 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 7 [chan receive]:
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/klog.(*loggingT).flushDaemon(0x53fee00)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/k8s.io/klog@v0.4.0/klog.go:1018 +0x70
Oct 23 11:43:52 pi4-a k3s[11584]: created by k8s.io/klog.init.0
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/k8s.io/klog@v0.4.0/klog.go:404 +0x68
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 338 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46d2c, 0x72, 0xffffffff)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x6633504, 0x72, 0x600, 0x6c4, 0xffffffff)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*FD).Read(0x66334f0, 0x658ee00, 0x6c4, 0x6c4, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_unix.go:169 +0x178
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*netFD).Read(0x66334f0, 0x658ee00, 0x6c4, 0x6c4, 0x8, 0x1be28, 0x1c694)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/fd_unix.go:202 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*conn).Read(0x678c798, 0x658ee00, 0x6c4, 0x6c4, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/net.go:184 +0x58
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*atLeastReader).Read(0x6406120, 0x658ee00, 0x6c4, 0x6c4, 0x18d60, 0x2f913c, 0x368e6a0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:780 +0x78
Oct 23 11:43:52 pi4-a k3s[11584]: bytes.(*Buffer).ReadFrom(0x6641160, 0x3688b98, 0x6406120, 0x5925c, 0x2f7c00, 0x2a1c5d0, 0x2d8c238)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bytes/buffer.go:204 +0xa4
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readFromUntil(0x6641000, 0x368e6a0, 0x678c798, 0x5, 0x678c798, 0x6406110)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:802 +0xc4
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readRecordOrCCS(0x6641000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:609 +0x104
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readRecord(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:577
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).Read(0x6641000, 0x59be000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:1255 +0x13c
Oct 23 11:43:52 pi4-a k3s[11584]: net/http.(*persistConn).Read(0x6634960, 0x59be000, 0x1000, 0x1000, 0x6006b00, 0x406eb4, 0x6006b00)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1747 +0x164
Oct 23 11:43:52 pi4-a k3s[11584]: bufio.(*Reader).fill(0x6408660)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bufio/bufio.go:100 +0x108
Oct 23 11:43:52 pi4-a k3s[11584]: bufio.(*Reader).Peek(0x6408660, 0x1, 0x0, 0x0, 0x1, 0x5e00800, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bufio/bufio.go:138 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: net/http.(*persistConn).readLoop(0x6634960)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1900 +0x178
Oct 23 11:43:52 pi4-a k3s[11584]: created by net/http.(*Transport).dialConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1569 +0x8e8
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 349 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c4690c, 0x72, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x58df464, 0x72, 0x0, 0x0, 0x400000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.spliceDrain(0x1f, 0x58df450, 0x400000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:92 +0x144
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.Splice(0x6633680, 0x58df450, 0xfffffb7e, 0x3fffffff, 0x482, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:41 +0x154
Oct 23 11:43:52 pi4-a k3s[11584]: net.splice(0x6633680, 0x368e6a0, 0x6412d78, 0x3fff, 0x3, 0x693, 0x19ff0, 0x5842000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/splice_linux.go:39 +0xa8
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).readFrom(0x6412d70, 0x368e6a0, 0x6412d78, 0x7be01, 0x0, 0x5976664, 0x7e430)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock_posix.go:48 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).ReadFrom(0x6412d70, 0x368e6a0, 0x6412d78, 0xa6aa2d90, 0x6412d70, 0x368e601, 0x2)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock.go:103 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: io.copyBuffer(0x368e6b8, 0x6412d70, 0x368e6a0, 0x6412d78, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:388 +0x2fc
Oct 23 11:43:52 pi4-a k3s[11584]: io.Copy(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:364
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.proxyCopy(0x5d34380, 0x36f10d8, 0x6412d70, 0x36f10d8, 0x6412d78)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:439 +0xf4
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*DialProxy).HandleConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:387 +0x2dc
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 24 [chan receive]:
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.(*DialProxy).HandleConn(0x6102a50, 0x36f10d8, 0x6426000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:389 +0x32c
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.(*Proxy).serveConn(0x6104820, 0x36f10d8, 0x6426000, 0x60fa8d8, 0x1, 0x1, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:239 +0xf4
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*Proxy).serveListener
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:221 +0x50
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 322 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46ca8, 0x72, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x656d194, 0x72, 0x0, 0x0, 0x400000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.spliceDrain(0x13, 0x656d180, 0x400000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:92 +0x144
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.Splice(0x656d1d0, 0x656d180, 0xfffffc6d, 0x3fffffff, 0x393, 0x0, 0x3030201, 0x0, 0x0, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:41 +0x154
Oct 23 11:43:52 pi4-a k3s[11584]: net.splice(0x656d1d0, 0x368e6a0, 0x66812f8, 0x3fff, 0x3, 0x693, 0x19ff0, 0x5842000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/splice_linux.go:39 +0xa8
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).readFrom(0x6195f40, 0x368e6a0, 0x66812f8, 0x0, 0x0, 0x0, 0x7e430)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock_posix.go:48 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).ReadFrom(0x6195f40, 0x368e6a0, 0x66812f8, 0xa6aa2d90, 0x6195f40, 0x368e601, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock.go:103 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: io.copyBuffer(0x368e6b8, 0x6195f40, 0x368e6a0, 0x66812f8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:388 +0x2fc
Oct 23 11:43:52 pi4-a k3s[11584]: io.Copy(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:364
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.proxyCopy(0x5d1b580, 0x36f10d8, 0x6195f40, 0x36f10d8, 0x66812f8)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:439 +0xf4
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*DialProxy).HandleConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:388 +0x318
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 26 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46eb8, 0x72, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x6584064, 0x72, 0x0, 0x0, 0x400000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.spliceDrain(0xc, 0x6584050, 0x400000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:92 +0x144
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.Splice(0x6584000, 0x6584050, 0xfffff78d, 0x3fffffff, 0x873, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:41 +0x154
Oct 23 11:43:52 pi4-a k3s[11584]: net.splice(0x6584000, 0x368e6a0, 0x6426038, 0x3fff, 0x3, 0x693, 0x19ff0, 0x5842000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/splice_linux.go:39 +0xa8
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).readFrom(0x6426000, 0x368e6a0, 0x6426038, 0x7be01, 0x0, 0x5976e44, 0x7e430)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock_posix.go:48 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).ReadFrom(0x6426000, 0x368e6a0, 0x6426038, 0xa6aa2d90, 0x6426000, 0x368e601, 0x2)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock.go:103 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: io.copyBuffer(0x368e6b8, 0x6426000, 0x368e6a0, 0x6426038, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:388 +0x2fc
Oct 23 11:43:52 pi4-a k3s[11584]: io.Copy(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:364
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.proxyCopy(0x5d36080, 0x36f10d8, 0x6426000, 0x36f10d8, 0x6426038)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:439 +0xf4
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*DialProxy).HandleConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:387 +0x2dc
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 161 [chan receive]:
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/wrangler/pkg/signals.SetupSignalHandler.func1(0x60c8580, 0x60fa888)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/wrangler@v0.2.0/pkg/signals/signal.go:37 +0x24
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/rancher/wrangler/pkg/signals.SetupSignalHandler
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/wrangler@v0.2.0/pkg/signals/signal.go:36 +0xb8
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 162 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46f3c, 0x72, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x5f08f14, 0x72, 0x0, 0x0, 0x2e17d68)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*FD).Accept(0x5f08f00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a8
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*netFD).accept(0x5f08f00, 0x7e11d600, 0x3, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/fd_unix.go:238 +0x20
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPListener).accept(0x5bab490, 0x5867fa4, 0x5867fa8, 0x1c)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock_posix.go:139 +0x20
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPListener).Accept(0x5bab490, 0x2fd4fac, 0x6104820, 0x36f10d8, 0x6412d70)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock.go:261 +0x3c
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.(*Proxy).serveListener(0x6104820, 0x60c8700, 0x36c57a0, 0x5bab490, 0x60fa8d8, 0x1, 0x1)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:216 +0x74
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*Proxy).Start
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:203 +0x14c
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 163 [chan receive]:
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.(*Proxy).awaitFirstError(0x6104820, 0x60c8700)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:210 +0x34
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*Proxy).Start
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:205 +0x2ec
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 28 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46e34, 0x72, 0xffffffff)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x656c7e4, 0x72, 0x600, 0x6c4, 0xffffffff)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*FD).Read(0x656c7d0, 0x658e000, 0x6c4, 0x6c4, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_unix.go:169 +0x178
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*netFD).Read(0x656c7d0, 0x658e000, 0x6c4, 0x6c4, 0x8, 0x1be28, 0x1c694)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/fd_unix.go:202 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*conn).Read(0x6413780, 0x658e000, 0x6c4, 0x6c4, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/net.go:184 +0x58
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*atLeastReader).Read(0x65a0b80, 0x658e000, 0x6c4, 0x6c4, 0x18d60, 0x2f913c, 0x368e6a0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:780 +0x78
Oct 23 11:43:52 pi4-a k3s[11584]: bytes.(*Buffer).ReadFrom(0x58ca160, 0x3688b98, 0x65a0b80, 0x1c694, 0x2f7c00, 0x2a1c5d0, 0x2d8c238)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bytes/buffer.go:204 +0xa4
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readFromUntil(0x58ca000, 0x368e6a0, 0x6413780, 0x5, 0x6413780, 0x43)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:802 +0xc4
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readRecordOrCCS(0x58ca000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:609 +0x104
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readRecord(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:577
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).Read(0x58ca000, 0x5936000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:1255 +0x13c
Oct 23 11:43:52 pi4-a k3s[11584]: net/http.(*persistConn).Read(0x6525400, 0x5936000, 0x1000, 0x1000, 0x5ae3c00, 0x406eb4, 0x5ae3c00)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1747 +0x164
Oct 23 11:43:52 pi4-a k3s[11584]: bufio.(*Reader).fill(0x6581020)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bufio/bufio.go:100 +0x108
Oct 23 11:43:52 pi4-a k3s[11584]: bufio.(*Reader).Peek(0x6581020, 0x1, 0x0, 0x0, 0x1, 0x65ece00, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bufio/bufio.go:138 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: net/http.(*persistConn).readLoop(0x6525400)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1900 +0x178
Oct 23 11:43:52 pi4-a k3s[11584]: created by net/http.(*Transport).dialConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1569 +0x8e8
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 29 [select]:
Oct 23 11:43:52 pi4-a k3s[11584]: net/http.(*persistConn).writeLoop(0x6525400)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:2199 +0xc0
Oct 23 11:43:52 pi4-a k3s[11584]: created by net/http.(*Transport).dialConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1570 +0x904
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 27 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46db0, 0x72, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x6584014, 0x72, 0x0, 0x0, 0x400000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.spliceDrain(0xa, 0x6584000, 0x400000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:92 +0x144
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.Splice(0x6584050, 0x6584000, 0xfffffdb5, 0x3fffffff, 0x24b, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:41 +0x154
Oct 23 11:43:52 pi4-a k3s[11584]: net.splice(0x6584050, 0x368e6a0, 0x6426000, 0x3fff, 0x3, 0x693, 0x19ff0, 0x5842000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/splice_linux.go:39 +0xa8
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).readFrom(0x6426038, 0x368e6a0, 0x6426000, 0x1, 0x14528, 0x619c030, 0x7e430)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock_posix.go:48 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).ReadFrom(0x6426038, 0x368e6a0, 0x6426000, 0xa6aa2d90, 0x6426038, 0x368e601, 0x619c030)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock.go:103 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: io.copyBuffer(0x368e6b8, 0x6426038, 0x368e6a0, 0x6426000, 0x0, 0x0, 0x0, 0x35d, 0x0, 0x368e658, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:388 +0x2fc
Oct 23 11:43:52 pi4-a k3s[11584]: io.Copy(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:364
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.proxyCopy(0x5d36080, 0x36f10d8, 0x6426038, 0x36f10d8, 0x6426000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:439 +0xf4
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*DialProxy).HandleConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:388 +0x318
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 340 [select]:
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/remotedialer.connectToProxy(0x36d26e0, 0x63bfbe0, 0x6671770, 0x23, 0x63bfa00, 0x2fd5310, 0x5b56d00, 0x65d3fc0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/remotedialer@v0.2.0/client.go:52 +0x394
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/remotedialer.ClientConnect(0x36d26e0, 0x63bfbe0, 0x6671770, 0x23, 0x63bfa00, 0x5b56d00, 0x2fd5310, 0x5a0bfc0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/remotedialer@v0.2.0/client.go:15 +0x54
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/k3s/pkg/agent/tunnel.connect.func1(0x36d26e0, 0x63bfbe0, 0x6671770, 0x23, 0x63bfa00, 0x5b56d00, 0x6664ce0, 0x6664f70)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/agent/tunnel/tunnel.go:197 +0x84
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/rancher/k3s/pkg/agent/tunnel.connect
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/agent/tunnel/tunnel.go:195 +0x1c8
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 348 [chan receive]:
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.(*DialProxy).HandleConn(0x6102a50, 0x36f10d8, 0x6412d70)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:389 +0x32c
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.(*Proxy).serveConn(0x6104820, 0x36f10d8, 0x6412d70, 0x60fa8d8, 0x1, 0x1, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:239 +0xf4
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*Proxy).serveListener
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:221 +0x50
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 256 [chan receive]:
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.(*DialProxy).HandleConn(0x6102a50, 0x36f10d8, 0x66812f8)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:389 +0x32c
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.(*Proxy).serveConn(0x6104820, 0x36f10d8, 0x66812f8, 0x60fa8d8, 0x1, 0x1, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:239 +0xf4
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*Proxy).serveListener
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:221 +0x50
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 300 [syscall]:
Oct 23 11:43:52 pi4-a k3s[11584]: syscall.Syscall6(0x118, 0x1, 0x2d52, 0x5879e78, 0x1000004, 0x0, 0x0, 0x0, 0x8, 0x25b)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/syscall/asm_linux_arm.s:45 +0x8
Oct 23 11:43:52 pi4-a k3s[11584]: os.(*Process).blockUntilWaitable(0x6679170, 0x1c694, 0x1, 0x60)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/wait_waitid.go:31 +0x64
Oct 23 11:43:52 pi4-a k3s[11584]: os.(*Process).wait(0x6679170, 0x2fd96f0, 0x2fd96f4, 0x2fd96ec)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/exec_unix.go:22 +0x2c
Oct 23 11:43:52 pi4-a k3s[11584]: os.(*Process).Wait(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/exec.go:125
Oct 23 11:43:52 pi4-a k3s[11584]: os/exec.(*Cmd).Wait(0x5cca210, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/exec/exec.go:501 +0x50
Oct 23 11:43:52 pi4-a k3s[11584]: os/exec.(*Cmd).Run(0x5cca210, 0x5ac7320, 0x656d098)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/exec/exec.go:341 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/k3s/pkg/agent/containerd.Run.func1(0x656d090, 0x9, 0x9, 0x368a218, 0x5b56e80, 0x368a218, 0x5b56e80)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/agent/containerd/containerd.go:72 +0x164
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/rancher/k3s/pkg/agent/containerd.Run
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/agent/containerd/containerd.go:64 +0x164
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 254 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46c24, 0x72, 0xffffffff)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x5bfd814, 0x72, 0x8001, 0x8000, 0xffffffff)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*FD).Read(0x5bfd800, 0x6722000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_unix.go:169 +0x178
Oct 23 11:43:52 pi4-a k3s[11584]: os.(*File).read(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/file_unix.go:259
Oct 23 11:43:52 pi4-a k3s[11584]: os.(*File).Read(0x66812b8, 0x6722000, 0x8000, 0x8000, 0x63, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/file.go:116 +0x5c
Oct 23 11:43:52 pi4-a k3s[11584]: io.copyBuffer(0x368a218, 0x5b56e80, 0x368ea78, 0x66812b8, 0x6722000, 0x8000, 0x8000, 0x0, 0x656d098, 0x2a08868, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:402 +0xd8
Oct 23 11:43:52 pi4-a k3s[11584]: io.Copy(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:364
Oct 23 11:43:52 pi4-a k3s[11584]: os/exec.(*Cmd).writerDescriptor.func1(0x9, 0x9)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/exec/exec.go:311 +0x50
Oct 23 11:43:52 pi4-a k3s[11584]: os/exec.(*Cmd).Start.func1(0x5cca210, 0x670e4a0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/exec/exec.go:435 +0x1c
Oct 23 11:43:52 pi4-a k3s[11584]: created by os/exec.(*Cmd).Start
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/os/exec/exec.go:434 +0x46c
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 255 [chan receive]:
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/natefinch/lumberjack.(*Logger).millRun(0x5b56e80)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/natefinch/lumberjack@v2.0.0+incompatible/lumberjack.go:379 +0x48
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/natefinch/lumberjack.(*Logger).mill.func1
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/natefinch/lumberjack@v2.0.0+incompatible/lumberjack.go:390 +0x64
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 241 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46ba0, 0x72, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x656d1e4, 0x72, 0x0, 0x0, 0x400000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.spliceDrain(0x15, 0x656d1d0, 0x400000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:92 +0x144
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.Splice(0x656d180, 0x656d1d0, 0xfffff92c, 0x3fffffff, 0x6d4, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:41 +0x154
Oct 23 11:43:52 pi4-a k3s[11584]: net.splice(0x656d180, 0x368e6a0, 0x6195f40, 0x3fff, 0x3, 0x693, 0x19ff0, 0x5842000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/splice_linux.go:39 +0xa8
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).readFrom(0x66812f8, 0x368e6a0, 0x6195f40, 0x7be01, 0x0, 0x67103c4, 0x7e430)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock_posix.go:48 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).ReadFrom(0x66812f8, 0x368e6a0, 0x6195f40, 0xa6aa2d90, 0x66812f8, 0x368e601, 0x2)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock.go:103 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: io.copyBuffer(0x368e6b8, 0x66812f8, 0x368e6a0, 0x6195f40, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:388 +0x2fc
Oct 23 11:43:52 pi4-a k3s[11584]: io.Copy(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:364
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.proxyCopy(0x5d1b580, 0x36f10d8, 0x66812f8, 0x36f10d8, 0x6195f40)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:439 +0xf4
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*DialProxy).HandleConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:387 +0x2dc
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 339 [select]:
Oct 23 11:43:52 pi4-a k3s[11584]: net/http.(*persistConn).writeLoop(0x6634960)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:2199 +0xc0
Oct 23 11:43:52 pi4-a k3s[11584]: created by net/http.(*Transport).dialConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1570 +0x904
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 341 [sleep]:
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.goparkunlock(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/proc.go:310
Oct 23 11:43:52 pi4-a k3s[11584]: time.Sleep(0x2a05f200, 0x1)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/time.go:105 +0x158
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/k3s/pkg/agent/tunnel.Setup.func1(0x66348c0, 0x6794110, 0x66ec790, 0x63bf9e0, 0x36d26e0, 0x6104760, 0x5ca01a0, 0x5a628f0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/agent/tunnel/tunnel.go:94 +0x28
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/rancher/k3s/pkg/agent/tunnel.Setup
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/agent/tunnel/tunnel.go:91 +0x2f4
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 345 [select]:
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2fd65d8, 0x2a05f200, 0x1, 0x0, 0x0, 0x6664c01, 0x5891080)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes/staging/src/k8s.io/apimachinery@v1.16.2-k3s.1/pkg/util/wait/wait.go:164 +0x168
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/apimachinery/pkg/util/wait.Until(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes/staging/src/k8s.io/apimachinery@v1.16.2-k3s.1/pkg/util/wait/wait.go:88
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/apimachinery/pkg/util/wait.Forever(0x2fd65d8, 0x2a05f200, 0x1)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes/staging/src/k8s.io/apimachinery@v1.16.2-k3s.1/pkg/util/wait/wait.go:79 +0x4c
Oct 23 11:43:52 pi4-a k3s[11584]: created by k8s.io/component-base/logs.InitLogs
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes/staging/src/k8s.io/component-base@v1.16.2-k3s.1/logs/logs.go:58 +0x74
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 343 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46b1c, 0x72, 0xffffffff)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x63d82e4, 0x72, 0x600, 0x6c4, 0xffffffff)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*FD).Read(0x63d82d0, 0x63b2a00, 0x6c4, 0x6c4, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_unix.go:169 +0x178
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*netFD).Read(0x63d82d0, 0x63b2a00, 0x6c4, 0x6c4, 0x585421c, 0x1be28, 0x1c694)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/fd_unix.go:202 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*conn).Read(0x6195ba0, 0x63b2a00, 0x6c4, 0x6c4, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/net.go:184 +0x58
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*atLeastReader).Read(0x6464cf0, 0x63b2a00, 0x6c4, 0x6c4, 0x18d60, 0x2f913c, 0x368e6a0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:780 +0x78
Oct 23 11:43:52 pi4-a k3s[11584]: bytes.(*Buffer).ReadFrom(0x6172d60, 0x3688b98, 0x6464cf0, 0x0, 0x2f7c00, 0x2a1c5d0, 0x2d8c238)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bytes/buffer.go:204 +0xa4
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readFromUntil(0x6172c00, 0x368e6a0, 0x6195ba0, 0x5, 0x6195ba0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:802 +0xc4
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readRecordOrCCS(0x6172c00, 0x0, 0x0, 0x3030203)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:609 +0x104
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readRecord(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:577
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).Read(0x6172c00, 0x582c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:1255 +0x13c
Oct 23 11:43:52 pi4-a k3s[11584]: bufio.(*Reader).fill(0x61da1e0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bufio/bufio.go:100 +0x108
Oct 23 11:43:52 pi4-a k3s[11584]: bufio.(*Reader).Peek(0x61da1e0, 0x2, 0x18, 0x5a0be9c, 0x3641bee, 0x0, 0xa6e109be)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bufio/bufio.go:138 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/gorilla/websocket.(*Conn).read(0x5a52180, 0x2, 0x2af9990, 0x18, 0x1be00, 0x1c600, 0x4c)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:357 +0x28
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/gorilla/websocket.(*Conn).advanceFrame(0x5a52180, 0x1cb8068, 0xc, 0x2fd53e0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:787 +0x60
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/gorilla/websocket.(*Conn).NextReader(0x5a52180, 0x36d26e0, 0x63a6120, 0x23, 0x63bfa00, 0x5b56d00)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:947 +0x80
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/remotedialer.(*wsConn).NextReader(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/remotedialer@v0.2.0/wsconn.go:32
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/remotedialer.(*Session).Serve(0x63d8a50, 0x36d26e0, 0x63a6120, 0x1f3818c, 0x6664ce0, 0x6664f70)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/remotedialer@v0.2.0/session.go:108 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/remotedialer.connectToProxy.func1(0x63d8a50, 0x36d26e0, 0x63a6120, 0x6195b68, 0x5b576c0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/remotedialer@v0.2.0/client.go:48 +0x3c
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/rancher/remotedialer.connectToProxy
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/remotedialer@v0.2.0/client.go:47 +0x308
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 344 [select]:
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/remotedialer.(*Session).startPings.func1(0x63d8a50, 0x36d26e0, 0x63a6180)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/remotedialer@v0.2.0/session.go:78 +0x16c
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/rancher/remotedialer.(*Session).startPings
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/remotedialer@v0.2.0/session.go:71 +0x94
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 347 [sleep]:
Oct 23 11:43:52 pi4-a k3s[11584]: runtime.goparkunlock(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/proc.go:310
Oct 23 11:43:52 pi4-a k3s[11584]: time.Sleep(0x3ea5229b, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/time.go:105 +0x158
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x77359400, 0x0, 0x0, 0x40000000, 0x9999999a, 0x3fc99999, 0x4, 0x0, 0x0, 0x6155c5c, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes/staging/src/k8s.io/apimachinery@v1.16.2-k3s.1/pkg/util/wait/wait.go:298 +0x34
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/kubernetes/pkg/util/node.GetNodeIP(0x371efa0, 0x5abcc80, 0x63fc6b3, 0x5, 0x10, 0x6482240, 0x3690ba8)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes@v1.16.2-k3s.1/pkg/util/node/node.go:121 +0xac
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/kubernetes/cmd/kube-proxy/app.newProxyServer(0x5bd1500, 0x0, 0x0, 0x0, 0xc, 0xffffffff, 0x1fc7d70)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes@v1.16.2-k3s.1/cmd/kube-proxy/app/server_others.go:144 +0x14f4
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/kubernetes/cmd/kube-proxy/app.NewProxyServer(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes@v1.16.2-k3s.1/cmd/kube-proxy/app/server_others.go:60
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/kubernetes/cmd/kube-proxy/app.(*Options).Run(0x5cb64b0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes@v1.16.2-k3s.1/cmd/kube-proxy/app/server.go:296 +0x88
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/kubernetes/cmd/kube-proxy/app.NewProxyCommand.func1(0x5bbeb40, 0x64221c0, 0x0, 0x4)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes@v1.16.2-k3s.1/cmd/kube-proxy/app/server.go:442 +0x144
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/spf13/cobra.(*Command).execute(0x5bbeb40, 0x63a6f80, 0x4, 0x4, 0x5bbeb40, 0x63a6f80)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x1f4
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/spf13/cobra.(*Command).ExecuteC(0x5bbeb40, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x230
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/spf13/cobra.(*Command).Execute(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/rancher/k3s/pkg/daemons/agent.kubeProxy.func1(0x5bbeb40)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/daemons/agent/agent.go:46 +0x1c
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/rancher/k3s/pkg/daemons/agent.kubeProxy
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/src/github.com/rancher/k3s/pkg/daemons/agent/agent.go:45 +0x21c
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 311 [chan receive]:
Oct 23 11:43:52 pi4-a k3s[11584]: k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0x5d0b830)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes/staging/src/k8s.io/apimachinery@v1.16.2-k3s.1/pkg/watch/mux.go:207 +0x50
Oct 23 11:43:52 pi4-a k3s[11584]: created by k8s.io/apimachinery/pkg/watch.NewBroadcaster
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/rancher/kubernetes/staging/src/k8s.io/apimachinery@v1.16.2-k3s.1/pkg/watch/mux.go:75 +0xa8
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 315 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46a98, 0x72, 0xffffffff)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x64286f4, 0x72, 0x600, 0x6c4, 0xffffffff)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*FD).Read(0x64286e0, 0x6320000, 0x6c4, 0x6c4, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_unix.go:169 +0x178
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*netFD).Read(0x64286e0, 0x6320000, 0x6c4, 0x6c4, 0x8, 0x1be28, 0x1c694)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/fd_unix.go:202 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*conn).Read(0x63e28d0, 0x6320000, 0x6c4, 0x6c4, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/net.go:184 +0x58
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*atLeastReader).Read(0x63d1940, 0x6320000, 0x6c4, 0x6c4, 0x18d60, 0x2f913c, 0x368e6a0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:780 +0x78
Oct 23 11:43:52 pi4-a k3s[11584]: bytes.(*Buffer).ReadFrom(0x6641560, 0x3688b98, 0x63d1940, 0x1c694, 0x2f7c00, 0x2a1c5d0, 0x2d8c238)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bytes/buffer.go:204 +0xa4
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readFromUntil(0x6641400, 0x368e6a0, 0x63e28d0, 0x5, 0x63e28d0, 0x600b8)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:802 +0xc4
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readRecordOrCCS(0x6641400, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:609 +0x104
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).readRecord(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:577
Oct 23 11:43:52 pi4-a k3s[11584]: crypto/tls.(*Conn).Read(0x6641400, 0x5bbd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/crypto/tls/conn.go:1255 +0x13c
Oct 23 11:43:52 pi4-a k3s[11584]: net/http.(*persistConn).Read(0x5abcdc0, 0x5bbd000, 0x1000, 0x1000, 0x610e200, 0x406eb4, 0x610e200)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1747 +0x164
Oct 23 11:43:52 pi4-a k3s[11584]: bufio.(*Reader).fill(0x5a77a40)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bufio/bufio.go:100 +0x108
Oct 23 11:43:52 pi4-a k3s[11584]: bufio.(*Reader).Peek(0x5a77a40, 0x1, 0x0, 0x0, 0x1, 0x610e100, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/bufio/bufio.go:138 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: net/http.(*persistConn).readLoop(0x5abcdc0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1900 +0x178
Oct 23 11:43:52 pi4-a k3s[11584]: created by net/http.(*Transport).dialConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1569 +0x8e8
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 350 [IO wait]:
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.runtime_pollWait(0xa6c46888, 0x72, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/runtime/netpoll.go:184 +0x44
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).wait(0x6633694, 0x72, 0x0, 0x0, 0x400000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.(*pollDesc).waitRead(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.spliceDrain(0x1d, 0x6633680, 0x400000, 0x0, 0x0, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:92 +0x144
Oct 23 11:43:52 pi4-a k3s[11584]: internal/poll.Splice(0x58df450, 0x6633680, 0xfffffa8b, 0x3fffffff, 0x575, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/internal/poll/splice_linux.go:41 +0x154
Oct 23 11:43:52 pi4-a k3s[11584]: net.splice(0x58df450, 0x368e6a0, 0x6412d70, 0x3fff, 0x3, 0x693, 0x19ff0, 0x5842000)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/splice_linux.go:39 +0xa8
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).readFrom(0x6412d78, 0x368e6a0, 0x6412d70, 0x0, 0x0, 0x0, 0x7e430)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock_posix.go:48 +0x30
Oct 23 11:43:52 pi4-a k3s[11584]: net.(*TCPConn).ReadFrom(0x6412d78, 0x368e6a0, 0x6412d70, 0xa6aa2d90, 0x6412d78, 0x368e601, 0x0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/tcpsock.go:103 +0x38
Oct 23 11:43:52 pi4-a k3s[11584]: io.copyBuffer(0x368e6b8, 0x6412d78, 0x368e6a0, 0x6412d70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:388 +0x2fc
Oct 23 11:43:52 pi4-a k3s[11584]: io.Copy(...)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/io/io.go:364
Oct 23 11:43:52 pi4-a k3s[11584]: github.com/google/tcpproxy.proxyCopy(0x5d34380, 0x36f10d8, 0x6412d78, 0x36f10d8, 0x6412d70)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:439 +0xf4
Oct 23 11:43:52 pi4-a k3s[11584]: created by github.com/google/tcpproxy.(*DialProxy).HandleConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/go/pkg/mod/github.com/google/tcpproxy@v0.0.0-20180808230851-dfa16c61dad2/tcpproxy.go:388 +0x318
Oct 23 11:43:52 pi4-a k3s[11584]: goroutine 316 [select]:
Oct 23 11:43:52 pi4-a k3s[11584]: net/http.(*persistConn).writeLoop(0x5abcdc0)
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:2199 +0xc0
Oct 23 11:43:52 pi4-a k3s[11584]: created by net/http.(*Transport).dialConn
Oct 23 11:43:52 pi4-a k3s[11584]: #011/usr/local/go/src/net/http/transport.go:1570 +0x904
Oct 23 11:43:52 pi4-a k3s[11584]: trap    0xe
Oct 23 11:43:52 pi4-a k3s[11584]: error   0x207
Oct 23 11:43:52 pi4-a k3s[11584]: oldmask 0x0
Oct 23 11:43:52 pi4-a k3s[11584]: r0      0x0
Oct 23 11:43:52 pi4-a k3s[11584]: r1      0xfffff001
Oct 23 11:43:52 pi4-a k3s[11584]: r2      0x17
Oct 23 11:43:52 pi4-a k3s[11584]: r3      0x6f3d4
Oct 23 11:43:52 pi4-a k3s[11584]: r4      0x64582a0
Oct 23 11:43:52 pi4-a k3s[11584]: r5      0x61baed4
Oct 23 11:43:52 pi4-a k3s[11584]: r6      0x0
Oct 23 11:43:52 pi4-a k3s[11584]: r7      0x4
Oct 23 11:43:52 pi4-a k3s[11584]: r8      0x589a5a0
Oct 23 11:43:52 pi4-a k3s[11584]: r9      0x1
Oct 23 11:43:52 pi4-a k3s[11584]: r10     0x64582a0
Oct 23 11:43:52 pi4-a k3s[11584]: fp      0x51b381c
Oct 23 11:43:52 pi4-a k3s[11584]: ip      0x272eb50
Oct 23 11:43:52 pi4-a k3s[11584]: sp      0x61bb094
Oct 23 11:43:52 pi4-a k3s[11584]: lr      0x43fe8
Oct 23 11:43:52 pi4-a k3s[11584]: pc      0x71118
Oct 23 11:43:52 pi4-a k3s[11584]: cpsr    0x10
Oct 23 11:43:52 pi4-a k3s[11584]: fault   0x0
Oct 23 11:43:52 pi4-a systemd[1]: k3s-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 23 11:43:52 pi4-a systemd[1]: k3s-agent.service: Failed with result 'exit-code'.
@billimek billimek mentioned this issue Oct 23, 2019
Closed
@billimek billimek changed the title k3s-agent will not start on a rasbian-based (kernel 4.19.75-v7l+) raspberry pi4 node k3s-agent will not start on a rasbian-based (kernel 4.19.75-v7l+) raspberry pi4 node Oct 23, 2019
@billimek billimek changed the title k3s-agent will not start on a rasbian-based (kernel 4.19.75-v7l+) raspberry pi4 node v0.10.0 k3s-agent will not start on a rasbian-based (kernel 4.19.75-v7l+) raspberry pi4 node Oct 23, 2019
@davidnuzik
Copy link
Contributor

davidnuzik commented Oct 23, 2019 •
edited
Loading

I am having issues getting k3s v0.10.0 to run at all on any raspi3b or raspi4 with debian buster with latest kernel (same but 4.19.75-v7+ in my case).

Terminal:

[INFO]  Finding latest release
[INFO]  Using v0.10.0 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v0.10.0/sha256sum-arm.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v0.10.0/k3s-armhf
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xe" for details.

systemctl status:

● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
   Active: activating (start) since Wed 2019-10-23 10:48:24 MST; 4s ago
     Docs: https://k3s.io
  Process: 2037 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 2038 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 2039 (k3s-server)
    Tasks: 22
   Memory: 85.5M
   CGroup: /system.slice/k3s.service
           └─2039 /usr/local/bin/k3s server KillMode=process

Oct 23 10:48:28 davidspi3 k3s[2039]: I1023 10:48:28.167255    2039 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following o
Oct 23 10:48:28 davidspi3 k3s[2039]: time="2019-10-23T10:48:28.234918319-07:00" level=info msg="Running kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/
Oct 23 10:48:28 davidspi3 k3s[2039]: time="2019-10-23T10:48:28.241799615-07:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --b
Oct 23 10:48:28 davidspi3 k3s[2039]: I1023 10:48:28.271350    2039 controllermanager.go:161] Version: v1.16.2-k3s.1
Oct 23 10:48:28 davidspi3 k3s[2039]: I1023 10:48:28.275434    2039 deprecated_insecure_serving.go:53] Serving insecurely on [::]:10252
Oct 23 10:48:28 davidspi3 k3s[2039]: I1023 10:48:28.294652    2039 server.go:143] Version: v1.16.2-k3s.1
Oct 23 10:48:28 davidspi3 k3s[2039]: I1023 10:48:28.295170    2039 defaults.go:91] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is manda
Oct 23 10:48:28 davidspi3 k3s[2039]: W1023 10:48:28.301640    2039 authorization.go:47] Authorization is disabled
Oct 23 10:48:28 davidspi3 k3s[2039]: W1023 10:48:28.301728    2039 authentication.go:79] Authentication is disabled
Oct 23 10:48:28 davidspi3 k3s[2039]: I1023 10:48:28.301785    2039 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251

I didn't see any other useful information via journalctl

On my Pi4, this did run for a while and crashed with the same error that was reported in the issue body.

@davidnuzik davidnuzik added this to the v1.x - Backlog milestone Oct 23, 2019
@erikwilson erikwilson mentioned this issue Oct 23, 2019
Closed
@erikwilson
Copy link
Contributor

Another stack trace:

INFO[2019-10-23T21:02:37.861588545+01:00] waiting for node raspberrypi: nodes "raspberrypi" not found
INFO[2019-10-23T21:02:37.925966087+01:00] Waiting for master node raspberrypi startup: node "raspberrypi" not found
I1023 21:02:37.931314   23631 server.go:406] Version: v1.16.2-k3s.1
E1023 21:02:38.024209   23631 machine.go:288] failed to get cache information for node 0: open /sys/devices/system/cpu/cpu0/cache: no such file or directory
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x266c61c]

goroutine 12773 [running]:
github.com/google/cadvisor/utils/sysfs.(*realSysFs).GetSystemUUID(0x53f0578, 0x53f0578, 0x171dc900, 0x474, 0xaa544b0)
	/root/go/pkg/mod/github.com/google/cadvisor@v0.34.0/utils/sysfs/sysfs.go:242 +0xbc
github.com/google/cadvisor/utils/sysinfo.GetSystemUUID(...)
	/root/go/pkg/mod/github.com/google/cadvisor@v0.34.0/utils/sysinfo/sysinfo.go:202
github.com/google/cadvisor/machine.Info(0x36e82f8, 0x53f0578, 0x36d69b8, 0x1722cd80, 0x53dd801, 0x1722cdc0, 0x0, 0x0)
	/root/go/pkg/mod/github.com/google/cadvisor@v0.34.0/machine/info.go:138 +0x3ac
github.com/google/cadvisor/manager.New(0xaa8ad80, 0x36e82f8, 0x53f0578, 0x7e11d600, 0x3, 0x1, 0x1722c540, 0x53dcec8, 0x1722c520, 0x3, ...)
	/root/go/pkg/mod/github.com/google/cadvisor@v0.34.0/manager/manager.go:188 +0x378
k8s.io/kubernetes/pkg/kubelet/cadvisor.New(0x3673730, 0x16f7ff50, 0x2e16f2b, 0x10, 0x1722c520, 0x3, 0x4, 0x0, 0x0, 0x0, ...)
	/root/go/pkg/mod/github.com/rancher/kubernetes@v1.16.2-k3s.1/pkg/kubelet/cadvisor/cadvisor_linux.go:97 +0x238
k8s.io/kubernetes/cmd/kubelet/app.run(0x11359880, 0x16ab6f00, 0x0, 0xc, 0xe965da4)
	/root/go/pkg/mod/github.com/rancher/kubernetes@v1.16.2-k3s.1/cmd/kubelet/app/server.go:626 +0xedc
k8s.io/kubernetes/cmd/kubelet/app.Run(0x11359880, 0x16ab6f00, 0x0, 0x158486c0, 0x0)
	/root/go/pkg/mod/github.com/rancher/kubernetes@v1.16.2-k3s.1/cmd/kubelet/app/server.go:410 +0xb0
k8s.io/kubernetes/cmd/kubelet/app.NewKubeletCommand.func1(0x14d02780, 0x1640d900, 0x1c, 0x20)
	/root/go/pkg/mod/github.com/rancher/kubernetes@v1.16.2-k3s.1/cmd/kubelet/app/server.go:266 +0x4ac
github.com/spf13/cobra.(*Command).execute(0x14d02780, 0x1640d900, 0x1c, 0x20, 0x14d02780, 0x1640d900)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x1f4
github.com/spf13/cobra.(*Command).ExecuteC(0x14d02780, 0x4, 0x2e1f083, 0x12)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x230
github.com/spf13/cobra.(*Command).Execute(...)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/rancher/k3s/pkg/daemons/agent.kubelet.func1(0x1640d900, 0x1c, 0x20, 0x14d02780)
	/root/go/src/github.com/rancher/k3s/pkg/daemons/agent/agent.go:145 +0x94
created by github.com/rancher/k3s/pkg/daemons/agent.kubelet
	/root/go/src/github.com/rancher/k3s/pkg/daemons/agent/agent.go:143 +0x610

Not sure why, but reverting a specific commit from the containerd bump seems to fix the issue: k3s-io/containerd@db5da10

^ cc @AkihiroSuda

@AkihiroSuda
Copy link
Contributor

Really related?

@erikwilson
Copy link
Contributor

erikwilson commented Oct 24, 2019 via email

@AkihiroSuda
Copy link
Contributor

Are you using native snapshotter?

@erikwilson
Copy link
Contributor

Sorry for the noise @AkihiroSuda, it appears the pi is using the overlay snapshotter, and that native snapshotter code is not being called here. Testing with go v1.13.2 seems to work fine tho.

This was referenced Oct 25, 2019
Merged
Merged
Merged
Closed
@billimek
Copy link
Author

Confirmed working properly now with k3s v0.10.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.x - Backlog
Development

No branches or pull requests
4 participants
@erikwilson @billimek @AkihiroSuda @davidnuzik