Description
Description
Using x509.certificate_managed to generate certificates works for a few hosts at a time, but not en-mass (and I only have 75 minions, so it's not exactly big scale). If I do a salt run on '*' or on larger groups of minions, salt-master stops responding to minions.
If I remove the x509.certificate_managed state, it all goes back to normal and works fine. This happens even if the minions already have a valid certificate so it's not generating one, but just checking for expiry.
Setup
It's pretty much the setup documented at: https://docs.saltstack.com/en/latest/ref/states/all/salt.states.x509.html
/etc/pki/{{ grains['fqdn'] }}.key:
x509.private_key_managed:
- bits: 4096
- backup: True
- require:
- x509: /etc/pki/ca.crt
{% set ips = [] %}
{% for ip in grains['ipv4'] %}
{% set ips = ips.append('IP:' + ip) %}
{% endfor %}
{% for ip in grains['ipv6'] %}
{% if ip != '::1' %}
{% set ips = ips.append('IP:' + ip) %}
{% endif %}
{% endfor %}
/etc/pki/{{ grains['fqdn'] }}.crt:
x509.certificate_managed:
- ca_server: salt-master.mydomain.net
- signing_policy: mypolicy
- public_key: /etc/pki/{{ grains['fqdn'] }}.key
- CN: {{ grains['fqdn'] }}
- O: MyCorp
- Email: hostmaster@mydomain.net
- C: GB
- L: London
- subjectAltName: '{{ ips|join(',') }}'
- days_valid: 90
- days_remaining: 30
- backup: True
- require:
- x509: /etc/pki/{{ grains['fqdn'] }}.key
x509.conf looks like:
x509_signing_policies:
mypolicy:
- minions: '*'
- signing_private_key: /etc/pki/ca.key
- signing_cert: /etc/pki/ca.crt
- O: MyCorp
- Email: hostmaster@mydomain.net
- C: GB
- L: London
- basicConstraints: "critical CA:false"
- keyUsage: "critical keyEncipherment"
- subjectKeyIdentifier: hash
- authorityKeyIdentifier: keyid,issuer:always
- days_valid: 90
- copypath: /etc/pki/issued_certs/
and /etc/salt/master.d/peer.conf:
peer:
.*:
# Sign a certificate with another minion that has a signing certificate.
# The requesting minion must also be granted access in a signing
# policy on the minion with the certificate.
- x509.sign_remote_certificate
Steps to Reproduce the behavior
My salt runs across all minions consistently take ~39s:
root@salt-master:~# time salt --state-output=changes --state_verbose=false -G 'os:Debian' state.highstate
[snip]
real 0m39.309s
user 0m2.165s
sys 0m0.194s
If I add in the /etc/pki/{{ grains['fqdn'] }}.crt state above, it all stops working:
root@salt-master:~# time salt --state-output=changes --state_verbose=false \* state.highstate
[ERROR ] Message timed out
Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.
real 3m32.809s
user 0m4.533s
sys 0m0.322s
If I remove that one state, it all returns to normal and works fine again.
Expected behavior
Checking a certificate's validity/expiry should not add significant time/load to a salt run.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)root@salt-master:~# salt --versions-report
Salt Version:
Salt: 3002.2
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10
libgit2: Not Installed
M2Crypto: 0.31.0
Mako: Not Installed
msgpack: 0.5.6
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: 3.6.1
pygit2: Not Installed
Python: 3.7.3 (default, Jul 25 2020, 13:03:44)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.1.2
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.1
System Versions:
dist: debian 10 buster
locale: UTF-8
machine: x86_64
release: 4.19.0-12-amd64
system: Linux
version: Debian GNU/Linux 10 buster
Additional context
Add any other context about the problem here.