Consul fails to start #10
Comments
|
Does it never stabilize? It looks like its starting to work, just hasn't joined all the servers yet. Also, PVCs might be a real issue for sure. I didn't realize actually that StatefulSets with PVCs can be started before the PVC is available (spoiled in the environment we run in I guess). Do you know what that looks like? Is the directory just not available yet? That might be something we have to build into an init container or something (to wait for it to be ready). |
|
@mitchellh no, it never does, funnily it used to work with the previous release of rook (v0.7). In my understanding the StatefulSet starts and attempts to bind the PVCs, until that is done, the pod should report an unbound pvc issue - should be easily accessbile via |
|
But during that time, the containers are started? Sorry, easiest way to figure this out would be if you did more digging or I can get a reproduction. For the latter, is there an easy way for me to get a similar environment up and running? |
|
They appear to be - the logs are there. I'm happy to do more digging, however in order for you to get a repro, I'd have to provide you with terraform files, helm value files & k8s manifests to get a copy of my env going OR I'll simply share a kubeconfig file so that you can poke around and leave it running for the night |
|
@mitchellh I've cloned the repo and made some alterations: server-statefulset.yaml Results
helm status consul
LAST DEPLOYED: Wed Sep 26 23:34:25 2018
NAMESPACE: service-discovery
STATUS: DEPLOYED
RESOURCES:
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
consul-kktl5 1/1 Running 0 2m
consul-tbjmt 1/1 Running 0 2m
consul-x5cqq 1/1 Running 0 2m
consul-server-0 1/1 Running 0 2m
consul-server-1 1/1 Running 0 2m
consul-server-2 1/1 Running 0 2m
==> v1/ConfigMap
NAME DATA AGE
consul-client-config 1 2m
consul-server-config 1 2m
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul-dns ClusterIP 10.3.228.30 <none> 53/TCP,53/UDP 2m
consul-server ClusterIP None <none> 8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP 2m
consul-ui ClusterIP 10.3.98.102 <none> 80/TCP 2m
==> v1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
consul 3 3 3 3 3 <none> 2m
==> v1/StatefulSet
NAME DESIRED CURRENT AGE
consul-server 3 3 2m
==> v1beta1/PodDisruptionBudget
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
consul-server N/A 0 0 2mCompare to previous resultsIf you compare the previous results, the NAME READY STATUS RESTARTS AGE
consul-56lvs 0/1 Running 0 36s
consul-jttwp 0/1 Running 0 36s
consul-qpgdn 0/1 Running 0 36s
consul-server-0 0/1 ContainerCreating 0 36s
consul-server-1 0/1 ContainerCreating 0 36s
consul-server-2 0/1 Running 0 36sHypothesis (FailureTreshold?)This is what the docs say: The server-statefulset.yaml failure treshold is: Meaning that kubernetes started to fail the checks and gave up before the pvc(s) were bound to the pods? |
|
@mmisztal1980 I made the adjustments that you mentioned, but am still seeing pod failing health checks and servers sitting in a pending state. Consul logs seem to indicate that the stateful set isn't binding the PVC. I am also using Rook/Ceph for storage. pod descriptionI checked the claims (kubectl get pvc) but don't see any claims made my Consul showing up in there. |
|
Ugh it looks like I was on an outdated version of the repo. After updating the repo and reinstalling it is working now. |
|
Added a PR where the probe settings are configurable via the chart values. That should help in our case. |
|
@mitchellh would the above PR be satisfactory? Tweaking the probe settings seems to have fixed the issue for myself and @jmreicha |
In my cluster I have rook running. When provisioning the consul cluster, my helm values file looks like this:
In order to start the chart, I use the following cmdline:
A quick verification of pvc indicate that they have bound successfully, please note that I've observed that it takes up to 20s to bind the pvc(s) running under rook
However the consul-server pods have failed to start:
An examination of the pod indicates that the readiness probe has failed:
kubectl -n service-discovery describe pod consul-server-2 Name: consul-server-2 Namespace: service-discovery Node: k8s-node-1.cloud-technologies.net/142.93.131.205 Start Time: Wed, 26 Sep 2018 07:47:42 +0200 Labels: app=consul chart=consul-0.1.0 component=server controller-revision-hash=consul-server-66479c5df5 hasDNS=true release=consul statefulset.kubernetes.io/pod-name=consul-server-2 Annotations: consul.hashicorp.com/connect-inject=false Status: Running IP: 10.2.1.13 Controlled By: StatefulSet/consul-server Containers: consul: Container ID: docker://09641e9f57faf0304b9e74818ee99f2a7dd23a4f1bc44fa6e426ad4be2d72578 Image: consul:1.2.3 Image ID: docker-pullable://consul@sha256:ea66d17d8c8c1f1afb2138528d62a917093fcd2e3b3a7b216a52c253189ea980 Ports: 8500/TCP, 8301/TCP, 8302/TCP, 8300/TCP, 8600/TCP, 8600/UDP Command: /bin/sh -ec CONSUL_FULLNAME="consul" exec /bin/consul agent \ -advertise="${POD_IP}" \ -bind=0.0.0.0 \ -bootstrap-expect=3 \ -client=0.0.0.0 \ -config-dir=/consul/config \ -datacenter=dc1 \ -data-dir=/consul/data \ -domain=consul \ -hcl="connect { enabled = true }" \ -ui \ -retry-join=${CONSUL_FULLNAME}-server-0.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \ -retry-join=${CONSUL_FULLNAME}-server-1.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \ -retry-join=${CONSUL_FULLNAME}-server-2.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \ -server State: Running Started: Wed, 26 Sep 2018 07:48:17 +0200 Ready: True Restart Count: 0 Readiness: exec [/bin/sh -ec curl http://127.0.0.1:8500/v1/status/leader 2>/dev/null | \ grep -E '".+"' ] delay=5s timeout=5s period=3s #success=1 #failure=2 Environment: POD_IP: (v1:status.podIP) NAMESPACE: service-discovery (v1:metadata.namespace) Mounts: /consul/config from config (rw) /consul/data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-js48j (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-consul-server-2 ReadOnly: false config: Type: ConfigMap (a volume populated by a ConfigMap) Name: consul-server-config Optional: false default-token-js48j: Type: Secret (a volume populated by a Secret) SecretName: default-token-js48j Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 10m default-scheduler Successfully assigned consul-server-2 to k8s-node-1.cloud-technologies.net Normal SuccessfulMountVolume 10m kubelet, k8s-node-1.cloud-technologies.net MountVolume.SetUp succeeded for volume "config" Normal SuccessfulMountVolume 10m kubelet, k8s-node-1.cloud-technologies.net MountVolume.SetUp succeeded for volume "default-token-js48j" Warning FailedMount 10m kubelet, k8s-node-1.cloud-technologies.net MountVolume.SetUp failed for volume "pvc-85ea1341-c14f-11e8-8c17-c6926976ea12" : invalid character '-' after top-level value Normal SuccessfulMountVolume 10m kubelet, k8s-node-1.cloud-technologies.net MountVolume.SetUp succeeded for volume "pvc-85ea1341-c14f-11e8-8c17-c6926976ea12" Normal Pulled 10m kubelet, k8s-node-1.cloud-technologies.net Container image "consul:1.2.3" already present on machine Normal Created 10m kubelet, k8s-node-1.cloud-technologies.net Created container Normal Started 9m kubelet, k8s-node-1.cloud-technologies.net Started container Warning Unhealthy 6m (x12 over 9m) kubelet, k8s-node-1.cloud-technologies.net Readiness probe failed: mmisztal@tsunami ~/Projects/@cloud-technologies/ops-k8s-services-inf/src/consul master ● An examination of the server logs indicates that it has failed to form the cluster:
Any hints what may be wrong?
I've noticed that the probe's
initialDelaySecondsdefault value is 5, so I'm guessing it may have failed before the pvcs have been bound? Perhaps it'd make sense to have this value configurable?The text was updated successfully, but these errors were encountered: