Skip to content
This repository has been archived by the owner on Aug 25, 2021. It is now read-only.

Enable TLS #313

Merged
merged 26 commits into from
Jan 10, 2020
Merged

Enable TLS #313

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
452eb4b
Initial TLS implementation
Mar 18, 2019
224c8ce
Improvements to the TLS implementation
ishustava Dec 6, 2019
b81632a
Update CHANGELOG
ishustava Dec 13, 2019
9a71a8e
tls-init-cleanup job fixes
ishustava Dec 14, 2019
46e8ab9
Client and server cluster roles don't need secret permissions
ishustava Dec 16, 2019
e57223f
Enable TLS for Consul Connect
ishustava Dec 16, 2019
2470fc3
Improvements from code review
ishustava Dec 18, 2019
6ff8db1
Enable TLS for sync-catalog
ishustava Dec 18, 2019
ef97b83
Enable TLS for server-acl-init
ishustava Dec 20, 2019
eb70d6e
Update CHANGELOG and set httpsOnly to true by default
ishustava Dec 20, 2019
458ce81
Update consul-k8s image
ishustava Dec 23, 2019
19d325d
Enable TLS for the Mesh gateway deployment
ishustava Dec 23, 2019
68c12b7
Support TLS for the snapshot agent deployment
ishustava Dec 23, 2019
c1a341e
Update CHANGELOG.md
ishustava Dec 23, 2019
d5ab090
Make sure helm test passes if TLS is enabled
ishustava Dec 24, 2019
64c57a3
tls-init service account, cluster role, and clusterrole binding shoul…
ishustava Jan 2, 2020
224b7d1
Support incremental rollout of TLS
ishustava Jan 3, 2020
b339938
Update CHANGELOG and enterprise licence TLS tests
ishustava Jan 8, 2020
81efca1
Update with changes from code review
ishustava Jan 9, 2020
c9700b8
Update Changelog and fix a few typos
ishustava Jan 9, 2020
08b4a24
Fix whitespace; combine/remove redundant tests
ishustava Jan 10, 2020
3d08818
GRPC address should not have 'http'
ishustava Jan 10, 2020
357c48e
Delete incorrect comments
ishustava Jan 10, 2020
3870aeb
Merge branch 'master' into enable-tls
ishustava Jan 10, 2020
bb34bc7
Fix syntax error
ishustava Jan 10, 2020
278da9a
Fix a bug in the enterprise license job
ishustava Jan 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,24 @@
## Unreleased

IMPROVEMENTS:

* Optionally allow enabling TLS for Consul communication [[GH-313](https://github.com/hashicorp/consul-helm/pull/313)].
If `global.tls.enabled` is set to `true`, the Helm chart will generate a CA and necessary certificates and
enable TLS for servers, clients, Connect injector, Mesh gateways, catalog sync, ACL bootstrapping, and snapshot agents.

Note that this feature is only supported if both servers and clients are running
on Kubernetes. We will have better support for other deployment architectures,
as well as bringing your own CA, in the future.

ishustava marked this conversation as resolved.
Show resolved Hide resolved
BUG FIXES:

* Fix graceful termination for servers [[GH-313](https://github.com/hashicorp/consul-helm/pull/313)].
`terminationGracePeriod` is now set to 30 seconds for the servers. The previous setting of 10 seconds
wasn't always enough time for a graceful leave, and in those cases, servers leave the cluster
in a "failed" state. Additionally, clients always set `leave_on_terminate` to `true`.
This replaces the `preStop` hook that was calling `consul leave`. Note that `leave_on_terminate` defaults
to true for clients as of Consul `0.7`, so this change only affects earlier versions.

## 0.15.0 (Dec 17, 2019)

BREAKING CHANGES:
Expand Down
104 changes: 92 additions & 12 deletions templates/client-daemonset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,19 @@ spec:
- name: config
ishustava marked this conversation as resolved.
Show resolved Hide resolved
configMap:
name: {{ template "consul.fullname" . }}-client-config
{{- if .Values.global.tls.enabled }}
- name: tls-ca-cert
secret:
secretName: {{ template "consul.fullname" . }}-ca-cert
- name: tls-ca-key
secret:
secretName: {{ template "consul.fullname" . }}-ca-key
- name: tls-client-cert
emptyDir:
# We're using tmpfs here so that
# client certs are not written to disk
medium: "Memory"
ishustava marked this conversation as resolved.
Show resolved Hide resolved
{{- end }}
{{- range .Values.client.extraVolumes }}
- name: userconfig-{{ .name }}
{{ .type }}:
Expand All @@ -80,7 +93,6 @@ spec:
- name: aclconfig
emptyDir: {}
{{- end }}

containers:
- name: consul
image: "{{ default .Values.global.image .Values.client.image }}"
Expand Down Expand Up @@ -110,6 +122,12 @@ spec:
name: {{ .Values.global.gossipEncryption.secretName }}
key: {{ .Values.global.gossipEncryption.secretKey }}
{{- end }}
{{- if .Values.global.tls.enabled }}
- name: CONSUL_HTTP_ADDR
value: https://localhost:8501
- name: CONSUL_CACERT
value: /consul/tls/ca/tls.crt
{{- end }}
{{- include "consul.extraEnvironmentVars" .Values.client | nindent 12 }}
command:
- "/bin/sh"
Expand All @@ -123,8 +141,23 @@ spec:
-bind=0.0.0.0 \
-client=0.0.0.0 \
-node-meta=pod-name:${HOSTNAME} \
-hcl='leave_on_terminate = true' \
{{- if .Values.global.tls.enabled }}
ishustava marked this conversation as resolved.
Show resolved Hide resolved
-hcl='ca_file = "/consul/tls/ca/tls.crt"' \
-hcl='cert_file = "/consul/tls/client/tls.crt"' \
-hcl='key_file = "/consul/tls/client/tls.key"' \
{{- if .Values.global.tls.verify }}
-hcl='verify_incoming_rpc = true' \
-hcl='verify_outgoing = true' \
-hcl='verify_server_hostname = true' \
{{- end }}
-hcl='ports { https = 8501 }' \
{{- if .Values.global.tls.httpsOnly }}
-hcl='ports { http = -1 }' \
{{- end }}
{{- end }}
{{- if .Values.client.grpc }}
-hcl="ports { grpc = 8502 }" \
-hcl='ports { grpc = 8502 }' \
{{- end }}
-config-dir=/consul/config \
{{- range .Values.client.extraVolumes }}
Expand Down Expand Up @@ -157,6 +190,14 @@ spec:
mountPath: /consul/data
- name: config
mountPath: /consul/config
{{- if .Values.global.tls.enabled }}
- name: tls-ca-cert
mountPath: /consul/tls/ca
readOnly: true
- name: tls-client-cert
mountPath: /consul/tls/client
readOnly: true
{{- end }}
{{- range .Values.client.extraVolumes }}
- name: userconfig-{{ .name }}
readOnly: true
Expand All @@ -166,17 +207,17 @@ spec:
- name: aclconfig
mountPath: /consul/aclconfig
{{- end }}
lifecycle:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not something that needs to be changed but we might be adding this back as consul force-leave -prune due to hashicorp/consul#6897 (comment)

preStop:
exec:
command:
- /bin/sh
- -c
- consul leave
ports:
{{- if (or (not .Values.global.tls.enabled) (not .Values.global.tls.httpsOnly)) }}
- containerPort: 8500
hostPort: 8500
name: http
{{- end }}
{{- if .Values.global.tls.enabled }}
- containerPort: 8501
hostPort: 8501
name: https
{{- end }}
- containerPort: 8502
hostPort: 8502
name: grpc
Expand Down Expand Up @@ -210,14 +251,21 @@ spec:
- "/bin/sh"
- "-ec"
- |
curl http://127.0.0.1:8500/v1/status/leader 2>/dev/null | \
grep -E '".+"'
{{- if .Values.global.tls.enabled }}
curl \
--cacert /consul/tls/ca/tls.crt \
https://127.0.0.1:8501/v1/status/leader \
{{- else }}
curl http://127.0.0.1:8500/v1/status/leader \
{{- end }}
2>/dev/null | grep -E '".+"'
{{- if .Values.client.resources }}
resources:
{{ tpl .Values.client.resources . | nindent 12 | trim }}
{{- end }}
{{- if .Values.global.bootstrapACLs }}
{{- if (or .Values.global.bootstrapACLs .Values.global.tls.enabled) }}
initContainers:
{{- if .Values.global.bootstrapACLs }}
- name: client-acl-init
image: {{ .Values.global.imageK8S }}
command:
Expand All @@ -232,6 +280,38 @@ spec:
- name: aclconfig
mountPath: /consul/aclconfig
{{- end }}
{{- if .Values.global.tls.enabled }}
- name: client-tls-init
image: "{{ default .Values.global.image .Values.client.image }}"
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
command:
- "/bin/sh"
- "-ec"
- |
cd /consul/tls/client
consul tls cert create -client \
-additional-ipaddress=${HOST_IP} \
-dc={{ .Values.global.datacenter }} \
ishustava marked this conversation as resolved.
Show resolved Hide resolved
-domain={{ .Values.global.domain }} \
ishustava marked this conversation as resolved.
Show resolved Hide resolved
-ca=/consul/tls/ca/cert/tls.crt \
-key=/consul/tls/ca/key/tls.key
mv {{ .Values.global.datacenter }}-client-{{ .Values.global.domain }}-0.pem tls.crt
mv {{ .Values.global.datacenter }}-client-{{ .Values.global.domain }}-0-key.pem tls.key
volumeMounts:
- name: tls-client-cert
mountPath: /consul/tls/client
- name: tls-ca-cert
mountPath: /consul/tls/ca/cert
readOnly: true
- name: tls-ca-key
mountPath: /consul/tls/ca/key
readOnly: true
{{- end }}
{{- end }}
{{- if .Values.client.nodeSelector }}
nodeSelector:
{{ tpl .Values.client.nodeSelector . | indent 8 | trim }}
Expand Down
9 changes: 9 additions & 0 deletions templates/client-podsecuritypolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,21 @@ spec:
{{- end }}
hostNetwork: false
hostPorts:
{{- if (not (and .Values.global.tls.enabled .Values.global.tls.httpsOnly)) }}
# HTTP Port
- min: 8500
max: 8500
{{- end }}
{{- if .Values.global.tls.enabled }}
# HTTPS port
- min: 8501
max: 8501
{{- end }}
{{- if .Values.client.grpc }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🍺

# gRPC Port
- min: 8502
max: 8502
{{- end }}
{{- if .Values.client.exposeGossipPorts }}
- min: 8301
max: 8301
Expand Down
25 changes: 21 additions & 4 deletions templates/client-snapshot-agent-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,10 @@ spec:
{{- end }}
terminationGracePeriodSeconds: 10
serviceAccountName: {{ template "consul.fullname" . }}-snapshot-agent

{{- if .Values.client.priorityClassName }}
priorityClassName: {{ .Values.client.priorityClassName | quote }}
{{- end }}
{{- if (or .Values.global.bootstrapACLs (and .Values.client.snapshotAgent.configSecret.secretName .Values.client.snapshotAgent.configSecret.secretKey)) }}
{{- if (or .Values.global.bootstrapACLs .Values.global.tls.enabled (and .Values.client.snapshotAgent.configSecret.secretName .Values.client.snapshotAgent.configSecret.secretKey)) }}
volumes:
{{- if (and .Values.client.snapshotAgent.configSecret.secretName .Values.client.snapshotAgent.configSecret.secretKey) }}
- name: snapshot-config
Expand All @@ -52,6 +51,11 @@ spec:
- name: aclconfig
emptyDir: {}
{{- end }}
{{- if .Values.global.tls.enabled }}
- name: consul-ca-cert
secret:
secretName: {{ template "consul.fullname" . }}-ca-cert
{{- end }}
{{- end }}
containers:
- name: consul-snapshot-agent
Expand All @@ -61,6 +65,15 @@ spec:
valueFrom:
fieldRef:
fieldPath: status.hostIP
{{- if .Values.global.tls.enabled }}
- name: CONSUL_HTTP_ADDR
value: https://$(HOST_IP):8501
- name: CONSUL_CACERT
value: /consul/tls/ca/tls.crt
{{- else }}
- name: CONSUL_HTTP_ADDR
value: http://$(HOST_IP):8500
{{- end }}
{{- if .Values.global.bootstrapACLs }}
- name: CONSUL_HTTP_TOKEN
valueFrom:
Expand All @@ -79,8 +92,7 @@ spec:
{{- if .Values.global.bootstrapACLs}}
-config-dir=/consul/aclconfig \
{{- end }}
-http-addr=http://${HOST_IP}:8500
{{- if (or .Values.global.bootstrapACLs (and .Values.client.snapshotAgent.configSecret.secretName .Values.client.snapshotAgent.configSecret.secretKey) ) }}
{{- if (or .Values.global.bootstrapACLs .Values.global.tls.enabled (and .Values.client.snapshotAgent.configSecret.secretName .Values.client.snapshotAgent.configSecret.secretKey) ) }}
volumeMounts:
{{- if (and .Values.client.snapshotAgent.configSecret.secretName .Values.client.snapshotAgent.configSecret.secretKey) }}
- name: snapshot-config
Expand All @@ -91,6 +103,11 @@ spec:
- name: aclconfig
mountPath: /consul/aclconfig
{{- end }}
{{- if .Values.global.tls.enabled }}
- name: consul-ca-cert
mountPath: /consul/tls/ca
readOnly: true
{{- end }}
{{- end }}
{{- if .Values.global.bootstrapACLs }}
initContainers:
Expand Down
29 changes: 24 additions & 5 deletions templates/connect-inject-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,19 +60,22 @@ spec:
{{ else if .Values.global.bootstrapACLs }}
-acl-auth-method="{{ template "consul.fullname" . }}-k8s-auth-method" \
{{- end }}
{{- if .Values.global.tls.enabled }}
-consul-ca-cert=/consul/tls/ca/tls.crt \
{{- end }}
{{- if .Values.connectInject.centralConfig.enabled }}
-enable-central-config=true \
{{- end }}
{{- if (and .Values.connectInject.centralConfig.enabled .Values.connectInject.centralConfig.defaultProtocol) }}
-default-protocol="{{ .Values.connectInject.centralConfig.defaultProtocol }}" \
{{- end }}
{{- if .Values.connectInject.certs.secretName }}
{{- if .Values.connectInject.certs.secretName }}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ishustava does the generated envoy config file include the TLS context (cert/key)? I see that the CA is added inline.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The envoy TLS context for the consul client (aka local_agent) doesn't need cert/key since it's not using mTLS client auth to talk to Consul. That's why it only has the CA cert.

If your question is about authentication, we recommend using ACLs for production environments for that purpose, and when they are enabled in the Helm chart, Envoy will include an ACL token when talking to Consul.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. I was more curious about mTLS for service to service (via envoy) communication.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. mTLS for service to service is configured dynamically as you add and remove services by the Consul client agent. These docs talk a bit about that.

Copy link

@fouadsemaan fouadsemaan May 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you referring to the dynamically generated envoy-bootstrap.yaml file via the consul connect envoy -bootrap command? Is that where the cert/key should appear?

Specifically, do I expect something like this in the generated file:

tls_context:
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "/etc/example-com.crt"
private_key:
filename: "/etc/example-com.key"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that config is used only for bootstrapping envoy itself. Consul updates envoy config for services dynamically through the API, so you won't find them in a file. You could look at the config though through the envoy admin endpoint. All you need to do is this:

kubectl port-forward po/<your pod that has envoy sidecar container> 19000

Then go to the browser at http://localhost:19000. There you will see an option to dump the config. Alternatively, you could just curl the endpoint to get the config curl http://localhost:19000/config_dump

You could do other things through that endpoint too, like change log levels. Here are the docs for the admin endpoint in case you'll find them useful.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic! I appreciate your help. Thank you! This is awesome!

-tls-cert-file=/etc/connect-injector/certs/{{ .Values.connectInject.certs.certName }} \
-tls-key-file=/etc/connect-injector/certs/{{ .Values.connectInject.certs.keyName }}
{{- else }}
{{- else }}
-tls-auto=${CONSUL_FULLNAME}-connect-injector-cfg \
-tls-auto-hosts=${CONSUL_FULLNAME}-connect-injector-svc,${CONSUL_FULLNAME}-connect-injector-svc.${NAMESPACE},${CONSUL_FULLNAME}-connect-injector-svc.${NAMESPACE}.svc
{{- end }}
{{- end }}
livenessProbe:
httpGet:
path: /health/ready
Expand All @@ -93,16 +96,32 @@ spec:
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 5
{{- if .Values.connectInject.certs.secretName }}
{{- if (or .Values.connectInject.certs.secretName .Values.global.tls.enabled) }}
volumeMounts:
{{- if .Values.connectInject.certs.secretName }}
- name: certs
mountPath: /etc/connect-injector/certs
readOnly: true
{{- end }}
{{- if .Values.global.tls.enabled }}
- name: consul-ca-cert
mountPath: /consul/tls/ca
readOnly: true
{{- end }}
{{- end }}
{{- if (or .Values.connectInject.certs.secretName .Values.global.tls.enabled) }}
volumes:
{{- if .Values.connectInject.certs.secretName }}
- name: certs
secret:
secretName: {{ .Values.connectInject.certs.secretName }}
{{- end }}
{{- end }}
{{- if .Values.global.tls.enabled }}
- name: consul-ca-cert
secret:
secretName: {{ template "consul.fullname" . }}-ca-cert
{{- end }}
{{- end }}
{{- if .Values.connectInject.nodeSelector }}
nodeSelector:
{{ tpl .Values.connectInject.nodeSelector . | indent 8 | trim }}
Expand Down
22 changes: 21 additions & 1 deletion templates/enterprise-license-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@ spec:
spec:
restartPolicy: Never
serviceAccountName: {{ template "consul.fullname" . }}-enterprise-license
{{- if .Values.global.tls.enabled }}
volumes:
- name: tls-ca-cert
secret:
secretName: {{ template "consul.fullname" . }}-ca-cert
{{- end }}
containers:
- name: apply-enterprise-license
image: "{{ default .Values.global.image .Values.server.image }}"
Expand All @@ -41,19 +47,33 @@ spec:
name: {{ .Values.server.enterpriseLicense.secretName }}
key: {{ .Values.server.enterpriseLicense.secretKey }}
- name: CONSUL_HTTP_ADDR
{{- if .Values.global.tls.enabled }}
value: https://{{ template "consul.fullname" . }}-server:8501
{{- else }}
value: http://{{ template "consul.fullname" . }}-server:8500
{{- end }}
{{- if .Values.global.tls.enabled }}
- name: CONSUL_CACERT
value: /consul/tls/ca/tls.crt
{{- end}}
{{- if .Values.global.bootstrapACLs }}
- name: CONSUL_HTTP_TOKEN
valueFrom:
secretKeyRef:
name: "{{ template "consul.fullname" . }}-enterprise-license-acl-token"
key: "token"
{{- end}}
command:
command:
- "/bin/sh"
- "-ec"
- |
consul license put "${ENTERPRISE_LICENSE}"
{{- if .Values.global.tls.enabled }}
volumeMounts:
- name: tls-ca-cert
mountPath: /consul/tls/ca
readOnly: true
{{- end }}
{{- if .Values.global.bootstrapACLs }}
initContainers:
- name: ent-license-acl-init
Expand Down
Loading