Failed to start controlplane in travis CI

[magicsong]

Sometimes it throws errors like "timeout" in travis and sometimes it succeed. However, always good in my computer.

[Samze]

We've been seeing this too:

      failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start

We found this was occurring when our CI machines were not powerful enough and we were running tests in parallel.

Irregardless this is not directly a kubebuilder issue but a controller-runtime envtest problem. Envtest starts the controlplane and this error is being thrown here. However, it also looks like you can change the default start/stop timeout by setting an environment variable.

[magicsong]

@Samze thank you, you are right👍. After i set the timeout longer, it has not been failed

[merlintang]

@magicsong I run into this again even if I have set the timeout to be 600seconds, it fails in the travis ci. export KUBEBUILDER_CONTROLPLANE_START_TIMEOUT=600s

[merlintang]

@Samze what is the better time to set ?

[DirectXMan12]

@merlintang you may have to check to see if your control plane is failing to start for some other reason. Perhaps you don't have the kube-apiserver or etcd binaries available?

[merlintang]

but this is not related my build environment. it fail in the travis. thanks for your kind attention. 2019/05/29 23:57:10 failed to start the controlplane. retried 5 times FAIL github.com/kubeflow/xgboost-operator/pkg/apis/xgboostjob/v1alpha1 0.039s

…

On Wed, May 29, 2019 at 4:52 PM Solly Ross ***@***.***> wrote: @merlintang <https://github.com/merlintang> you may have to check to see if your control plane is failing to start for some other reason. Perhaps you don't have the kube-apiserver or etcd binaries available? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#628?email_source=notifications&email_token=AAK5R6PPGIJHDA6KF2ED23LPX4JNTA5CNFSM4G5ITGC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWQ6LDY#issuecomment-497149327>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAK5R6O2IGDY6KTDJN6TWGLPX4JNTANCNFSM4G5ITGCQ> .

[DirectXMan12]

Right, so, your build environment is properly set up in Travis? What version of controller-runtime are you using? What version of the kubebuilder assets (the kube-apiserver and etcd)?

[merlintang]

yes, I test on the Travis env. the kubebuilder version is Version: version.Version{KubeBuilderVersion:"1.0.8", KubernetesVendor:"1.13.1", GitCommit:"1adf50ed107f5042d7472ba5ab50d5e1d357169d", BuildDate:"2019-01-25T23:14:29Z", GoOs:"unknown", GoArch:"unknown"} below are the related information: k8s.io/api v0.0.0-20181026184759-d1dc89ebaebe k8s.io/apiextensions-apiserver v0.0.0-20181026191334-ba848ee89ca3 // indirect k8s.io/apimachinery v0.0.0-20181022183627-f71dbbc36e12 k8s.io/apiserver v0.0.0-20181026185746-f1e867e1a455 // indirect k8s.io/client-go v0.0.0-20181004124242-1638f8970cef k8s.io/code-generator v0.0.0-20180823001027-3dcf91f64f63 k8s.io/gengo v0.0.0-20190327210449-e17681d19d3a // indirect k8s.io/klog v0.3.0 // indirect k8s.io/kubernetes v1.12.2 k8s.io/utils v0.0.0-20190308190857-21c4ce38f2a7 // indirect sigs.k8s.io/controller-runtime v0.1.9 sigs.k8s.io/controller-tools v0.1.8 sigs.k8s.io/testing_frameworks v0.1.1 // indirect thanks for your kind help.

…

On Wed, May 29, 2019 at 5:45 PM Solly Ross ***@***.***> wrote: Right, so, your build environment is properly set up in Travis? What version of controller-runtime are you using? What version of the kubebuilder assets (the kube-apiserver and etcd)? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#628?email_source=notifications&email_token=AAK5R6OEJ5IONXKMZCL4KMDPX4PT5A5CNFSM4G5ITGC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWRAWPA#issuecomment-497158972>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAK5R6MY3XKIIV3L5AM6SR3PX4PT5ANCNFSM4G5ITGCQ> .

[DirectXMan12]

Go modules aren't supported with v1.0.8. I doubt that's the issue, but I figured I'd point it out.

I think at some point there was a change to a flag on the API server. Think it was around 1.13 (we have v1.10 listed as the test version in CR v0.1.9).

@mengqiy PTAL

[mengqiy]

@merlintang It seems you are using the version that contains the fix related to the kube-apiserver.

I suspect it may be caused by an issue that is not directly related to kubebuilder.
kubernetes-sigs/controller-runtime#459 should help surface the issue from the testing control plane.

[merlintang]

@mengqiy <https://github.com/mengqiy> can you give hints how to solve this?

…

On Thu, May 30, 2019 at 3:45 PM Mengqi Yu ***@***.***> wrote: @merlintang <https://github.com/merlintang> It seems you are using the version that contains the fix related to the kube-apiserver. I suspect it may be caused by an issue that is not directly related to kubebuilder. kubernetes-sigs/controller-runtime#459 <kubernetes-sigs/controller-runtime#459> should help surface the issue from the testing control plane. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#628?email_source=notifications&email_token=AAK5R6IZBIAM2RT3PNLZ77DPYBKKHA5CNFSM4G5ITGC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWTW5BI#issuecomment-497512069>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAK5R6P4CWATDLAOGC2ORHLPYBKKHANCNFSM4G5ITGCQ> .

[mengqiy]

@merlintang You will need to pick up kubernetes-sigs/controller-runtime#459 in your controller-runtime dependency. How you do it depends on if you are using dep or go modules.
Then you can use normal approach to run test e.g. go test path/to/your/package. kube-apiserver and etcd logs will give you a idea if control plane is working properly.

[merlintang]

@mengqiy I have cherry pick to apply your pr, then, we can observe the error:
2019/06/03 15:23:19 failed to start the controlplane. retried 5 times
even if have change the KUBEBUILDER_CONTROLPLANE_START_TIMEOUT.

Can you point to me where I can look at the logs for further checking ?