[BUG] Master startup performance using consul cache #58325
Labels
Bug
broken, incorrect, or confusing behavior
Performance
Salt-Syndic
severity-high
2nd top severity, seen by most users, causes major problems
Milestone
Description
Master workers load cache too much during startup.
Each worker generates at least one request per minion to consul.
Restarting all/few masters at a time makes consul unresponsive for an hour(s).
MemCache does not help at all
Setup
Master config:
zmq_backlog : 8192
consul.dc : dc1
master_sign_pubkey : False
consul.consistency : stale
state_output : terse
log_level : debug
consul.port : 8500
cache : consul
con_cache : False
ipv6 : False
master_id : master
order_masters : True
event_publisher_pub_hwm : 64000
consul.host : 127.0.0.1
syndic_wait : 30
worker_threads : 144
pub_hwm : 8192
user : salt
state_verbose : False
sock_pool_size : 4096
consul.token : xxx
zmq_filtering : False
keep_jobs : 4
consul.verify : True
salt_event_pub_hwm : 128000
max_event_size : 1572864
consul.scheme : http
memcache_expire_seconds: 300
memcache_max_items: 1000
memcache_debug: True
Steps to Reproduce the behavior
restarted master at 17:20
Expected behavior
There should be a separate process to query consul only once.
Versions Report
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 16.0.4
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.2.5
System Versions:
dist: centos 7.8.2003 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1127.19.1.el7.x86_64
system: Linux
version: CentOS Linux 7.8.2003 Core
The text was updated successfully, but these errors were encountered: