[incubator/vault] Readiness and Liveness Probe Failed leading to CrashLoopBackOff

[julekrut]

Is this a request for help?: No

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Version of Helm and Kubernetes:
Helm: v2.12.3
K8s: v1.11.6

Which chart:
incubator/vault

What happened:
Liveness probe failed
Readiness probe failed

What you expected to happen:
Vault pods up and running

How to reproduce it (as minimally and precisely as possible):
I follow the incubator/vault manual

helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator

helm install incubator/vault --set vault.dev=false --set vault.config.storage.consul.address="myconsul-svc-name:8500",vault.config.storage.consul.path="vault"

Anything else we need to know:
None

[edwardbartholomew]

@julekrut I received same error you did when trying this chart and then noticed below in README:

Please note that a backend service for Vault (for example, Consul) must be deployed beforehand and configured with the `vault.config` option.

Do you already have consul deployed [as myconsul-svc-name], otherwise you need to use helm to deploy consul before deploying Vault chart.

I used https://github.com/hashicorp/consul-helm to deploy Consul and then substituted resulting service (ie, running-hummingbird-consul-server:8500) for the vault.config.storage.consul.address parameter.

[julekrut]

@edwardbartholomew Yes I have. I already have consul running in place. Alternatively, since I use AWS instead of consul I've tried to use AWS S3 as vault storage back end, but it gives the same error.

  config:
    storage:
      s3:
        access_key: "AWS-ACCESS-KEY"
        secret_key: "AWS-SECRET-KEY"
        bucket: "AWS-BUCKET"
        region: "eu-central-1"

[kilhage]

I have the same issue, I tried to use the mysql and consul backend...

[edwardbartholomew]

Does kubectl logs <pod> reveal any error messages? I've been using both HashiCorp's official chart (https://github.com/hashicorp/consul-helm) and this repo's stable/consul without issue with incubator/vault on GKE.

[julekrut]

@edwardbartholomew : On vault pod, it just show some warning.

  Normal   Scheduled  15m                 default-scheduler                                 Successfully assigned default/vault-vault-7656cbdf8c-9pp5x to ip-x-3-0-248.xx-xx-x-1.compute.internal
  Normal   Pulled     14m (x2 over 15m)   kubelet, ip-x-3-0-248.xx-xx-x-1.compute.internal  Container image "vault:1.0.1" already present on machine
  Normal   Created    14m (x2 over 15m)   kubelet, ip-x-3-0-248.xx-xx-x-1.compute.internal  Created container
  Normal   Started    14m (x2 over 15m)   kubelet, ip-x-3-0-248.xx-xx-x-1.compute.internal  Started container
  Warning  Unhealthy  13m (x11 over 14m)  kubelet, ip-x-3-0-248.xx-xx-x-1.compute.internal  Readiness probe failed: Get http://x0.x4.0.6:8200/v1/sys/health?standbycode=204&uninitcode=204&: dial tcp x0.x4.0.6:8200: connect: connection refused
  Normal   Killing    13m (x2 over 14m)   kubelet, ip-x-3-0-248.xx-xx-x-1.compute.internal  Killing container with id docker://vault:Container failed liveness probe.. Container will be killed and recreated.
  Warning  Unhealthy  xm (x16 over 15m)  kubelet, ip-x-3-0-248.xx-xx-x-1.compute.internal  Liveness probe failed: dial tcp x0.x4.0.6:8200: connect: connection refused
  Warning  BackOff    1s (x37 over 9m)    kubelet, ip-x-3-0-248.xx-xx-x-1.compute.internal  Back-off restarting failed container

[kilhage]

I finally resolved my issue. I had to set hostNetwork: true under spec.template.spec in deployment.yml in order for the dns lookup to work properly when using the mysql storage.

This may also be the case for S3 and all other storage backends that are using external services.

Consul was not working since I didn't manage to get consul up and running properly due to network configuration issues in AWS.

When using AWS and service.type=LoadBalancer and a SSL certificate (configured through annotations) I also have to change spec.ports.name: https in service.yml, otherwise HTTPS will not work properly.

Can you please make these two parameters configurable so we don't have to manually edit the files in order to get this working?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[mkozjak]

Related to #9462.

[zdevi]

Hello, this issue is very anoying, is it going to be be merged anytime soon ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.