backup pod: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined

[hongchaodeng]

https://jenkins-etcd.prod.coreos.systems/job/etcd-operator-flaketest/1445/consoleText

restore_test.go:93: fail to make a backup: backup pod (test-etcd-x7p8h-backup-sidecar-wfsd9): backupnow (10.216.5.14:19999) failed: Get http://10.216.5.14:19999/v1/backupnow: dial tcp 10.216.5.14:19999: getsockopt: connection refused

[xiang90]

I cannot open the log. Can you check if the url is correct?

[hongchaodeng]

@xiang90
The jenkins only store most recent 100 tests. We build 100 tests a day. So it's gone.

[xiang90]

@hongchaodeng Can we fix it fixed? Or we probably do this as etcd issues (always pre process log, copy related log to github issue when possible).

[hongchaodeng]

Can we fix it fixed?

Didn't catch that.

Or we probably do this as etcd issues (always pre process log, copy related log to github issue when possible).

I agree we need some workflow for saving logs. Currently the logs isn't easy to copy and save around. We need to have some solutions for this.

[xiang90]

I mean can we get this fixed? Like can we increase 100 to 1000 temporarily?

[hongchaodeng]

@xiang90
We didn't have any retry on this. This is rare case so we don't bother to make it robust.
When we try to make a backup in that test, the backup pod is ensured to be up. And we talk directly to pod ip.

I checked again the backup pod's log:

17:07:25.000
panic: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
17:07:25.000
17:07:25.000
goroutine 1 [running]:
17:07:25.000
panic(0x1503380, 0xc4204cf040)
17:07:25.000
/usr/local/go/src/runtime/panic.go:500 +0x1a1
17:07:25.000
github.com/coreos/etcd-operator/pkg/util/k8sutil.MustCreateClient(0x0, 0x0, 0x0, 0x0, 0xc42042f300, 0x0)
17:07:25.000
/home/jenkins/workspace/build-etcd-operator-flaketest-1386/gopath/src/github.com/coreos/etcd-operator/pkg/util/k8sutil/k8sutil.go:303 +0x108
17:07:25.000
main.main()
17:07:25.000
/home/jenkins/workspace/build-etcd-operator-flaketest-1386/gopath/src/github.com/coreos/etcd-operator/cmd/backup/main.go:72 +0x188
17:07:44.000
time="2017-02-02T01:07:44Z" level=info msg="listening on 0.0.0.0:19999"
17:14:46.000
shutting down, got signal: Terminated

This is totally wrong.. It's a pod, but it couldn't find in-cluster configs.

[xiang90]

panic: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined

Is this our problem? Or kubernetes? Shall we report to upstream if this is a k8s issue?

[hongchaodeng]

@xiang90
k8s problem.

We create in-cluster client using client-go. It failed to. But I don't think this is client problem because the logs showed that KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT isn't defined. They should be set in env by default.

[xiang90]

@hongchaodeng Can you create an issue on upstream? Thanks!

[xiang90]

kubernetes/kubernetes#40973