cmk cluster-init failed to re-run with new added node

[jackiehjm]

Description

cmk cluster-init failed with new added node

ERROR:root:Exception when creating secret: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 11 Apr 2019 05:03:41 GMT', 'Content-Length': '218', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"cmk-webhook-certs\" already exists","reason":"AlreadyExists","details":{"name":"cmk-webhook-certs","kind":"secrets"},"code":409}


ERROR:root:Aborting webhook deployment ...

steps to reproduce

1. create cmk-cluster-init-pod with the following args:

  - args:
      # Change this value to pass different options to cluster-init.
      - "/cmk/cmk.py cluster-init --host-list=node-0 --saname=cmk-serviceaccount"

2. check that the cmk cluster-init is successful

$ kubectl get pods -o wide
NAME                                             READY   STATUS      RESTARTS   AGE    IP             NODE           NOMINATED NODE
cmk-cluster-init-pod                             0/1     Completed   0          128m   172.16.0.8     node-0         <none>
cmk-init-install-discover-pod-node-0             0/2     Completed   0          128m   172.16.0.9     node-0         <none>
cmk-reconcile-nodereport-ds-node-0-g5ww7         2/2     Running     0          127m   172.16.0.10    node-0         <none>
cmk-webhook-deployment-5b7895df7f-zzwgc          1/1     Running     0          127m   172.16.0.11    node-0         <none>

3. add a new node node-1 and create cmk-cluster-init-pod-node-1 with the following args:

  - args:
      # Change this value to pass different options to cluster-init.
      - "/cmk/cmk.py cluster-init --host-list=node-1 --saname=cmk-serviceaccount"

cmk-cluster-init-pod-node-1 failed with errors:

$ kubectl get pods cmk-cluster-init-pod-node-1 -o wide
NAME                                             READY   STATUS      RESTARTS   AGE    IP             NODE           NOMINATED NODE
cmk-cluster-init-pod-node-1                      0/1     Error       0          18m    172.16.1.98    node-1         <none>

$ kubectl describe logs cmk-cluster-init-pod-node-1
error: the server doesn't have a resource type "logs"
$ kubectl logs cmk-cluster-init-pod-node-1
INFO:root:Used ServiceAccount: cmk-serviceaccount
INFO:root:Creating cmk pod for ['init', 'install', 'discover'] commands ...
INFO:root:Waiting for cmk pod running ['init', 'install', 'discover'] cmds to enter Succeeded state.
INFO:root:Creating cmk pod for ['reconcile', 'nodereport'] commands ...
INFO:root:Waiting for cmk pod running ['reconcile', 'nodereport'] cmds to enter Running state.
ERROR:root:Exception when creating secret: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 11 Apr 2019 05:03:41 GMT', 'Content-Length': '218', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"cmk-webhook-certs\" already exists","reason":"AlreadyExists","details":{"name":"cmk-webhook-certs","kind":"secrets"},"code":409}


ERROR:root:Aborting webhook deployment ...

[lmdaly]

@jackiehjm Are you removing CMK before re-running with a new node added? Or running cluster-init again on an already running CMK Cluster?