[incubator/vault] Use httpGet instead of TCP Socket for liveness check

[jbialy]

What this PR does / why we need it:

Currently, the liveness probe uses tcpSocket when determining if the pod is alive or not. This works OK when Vault is listening on HTTP. Although, when TLS is enabled, the liveness probe continues to work but a warning is logged for every check interval:

2018-06-12 14:25:47.867270 I | http: TLS handshake error from 100.127.0.0:16184: EOF 
...

This PR changes the liveness probe to use httpGet where the protocol scheme can be configured to HTTPS depending on the values specified in values.yaml. This avoids filling the logs with the above warning.

The probe checks Vault's /v1/sys/health endpoint in accordance with:

https://www.vaultproject.io/api/system/health.html#standbyok

Checklist

• DCO signed
• Chart Version bumped
• Variables are documented in the README.md

[jbialy]

/assign @linki

[jbialy]

/assign @seanknox

[jbialy]

👀 PTAL

[jbialy]

/assign

[jbialy]

There seem to be incubator/vault PRs that are 2 months old, are these still being looked at / considered?

[cpuspellcaster]

My team is encountering a slightly different issue that would have the same fix: On EKS, with the tcpSocket configuration, the liveness probe configures a AWS ELB health check to perform a TCP ping on the API endpoint. However, the health checks are failing even the service is healthy (when checked via port-forward). (The nearest explanation that I can find is the TCP ping times out because the socket connection is kept while the service waits for a HTTP request.)

Regardless, switching to httpGet would resolve our issue.

[jbialy]

Bump. Rebased, PTAL 👀 🙇

[jbialy]

/assign @viglesiasce

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[mattfarina]

cc @jpds

[mattfarina]

/ok-to-test

[mattfarina]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jbialy, mattfarina

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mattfarina]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mkozjak]

This one is not right. You shouldn't kill a pod when you receive an error like 501 or 503, since it means Vault is in a sealed state.
I'm specifically getting a 503, since it's a fresh installation, obviously.

https://www.vaultproject.io/api/system/health.html

So with 0.15.0 I cannot unseal Vault since it gets restarted, ending up in a CrashLoopBackOff.

Check describe output:

Warning  Unhealthy  3m (x7 over 4m)  kubelet, 159.69.17.129  \
Liveness probe failed: HTTP probe failed with statuscode: 503

Warning  Unhealthy  3m (x7 over 4m)  kubelet, 159.69.17.129  \
Readiness probe failed: HTTP probe failed with statuscode: 503

[jbialy]

Ah, good catch! I didn't foresee this. One way to fix it would be to change the sealedcode and uninitcode to return 200 response code.

[mkozjak]

@jbialy In upstream? Hmm, doubt it that they'll do it easily. Reverting back to 0.14.9, locally.
I guess you should revert this bugfix, too, as it'll only work in dev env.

[jbialy]

sealedcode and uninitcode are params that can be passed to the /health endpoint and can be used to overwrite the response from Vault, so it should do the job, the problem is that the review process for charts takes a while. I had this PR open for about four months.