salt-api can't bear a lot of concurrency when using tornado backend?

[liu624862560]

Description of Issue

here is question, we want use salt-api to send some shell command to minions, we use tornado_rest salt-api, we have one master and about 800 minions, while we use requests and multiprocessing/threading to make requests to salt-api, when we set about 100 minions and do test.ping function, we get about ten returners like [{}], and if we set about 400 minions, we get many [{}],we use request like this

res = requests.post(
            #url=salt_api_url,
            url=salt_api_url,
            data=json.dumps([{
                "client": "local",
                "tgt": target,
                "fun": "test.ping",
                "tgt_type": tgt_type,
                "timeout": 10
            }]),
            headers={
                "Accept": "application/json",
                "X-Auth-Token": token,
                "Content-Type": "application/json"
            },
            verify=False,
            timeout=300
        )

and we just want get results synchronously, don't want use async/returners/event,so is there something I don't know with config? can salt-api do as we want? and anyone has some ideas, please leave you message, thanks for all of you.

Setup

saltmaster config
default_include: master.d/*.conf
timeout: 20
worker_threads: 20
auto_accept: True
file_roots:
  base:
    - /datasalt/srv/salt/base
  dev:
    - /datasalt/srv/salt/dev
  test:
    - /datasalt/srv/salt/test
  prod:
    - /datasalt/srv/salt/prod
pillar_roots:
  base:
    - /datasalt/srv/pillar/base
  dev:
    - /datasalt/srv/pillar/dev
  test:
    - /datasalt/srv/pillar/test
  prod:
    - /datasalt/srv/pillar/prod
pillar_opts: True
log_level_logfile: debug

salt-api config

rest_tornado:
    port: 8001
    address: 0.0.0.0
    #backlog: 128
    ssl_crt: /etc/pki/tls/certs/saltcert.crt
    ssl_key: /etc/pki/tls/private/saltcert.key
    debug: True
    disable_ssl: False
    cors_origin: null
    webhook_url: /hook
    webhook_disable_auth: True
    #num_processes: 4

Steps to Reproduce Issue

some logs like this:

2020-01-24 14:24:12,337 [salt.transport.ipc:254 ][DEBUG   ][4850] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
2020-01-24 14:24:12,340 [salt.transport.zeromq:1084][DEBUG   ][4850] SaltReqTimeoutError, retrying. (1/3)
2020-01-24 14:24:12,380 [salt.transport.zeromq:1084][DEBUG   ][4850] SaltReqTimeoutError, retrying. (1/3)
2020-01-24 14:24:12,411 [salt.transport.zeromq:138 ][DEBUG   ][4850] Re-using AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'sy-centos7.3.1611-xxxxx_master', u'tcp://1
27.0.0.1:4506', u'clear')
2020-01-24 14:24:12,412 [salt.transport.ipc:254 ][DEBUG   ][4850] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc

it looks like something wrong with SaltReqTimeoutError, we don't have much error like this with cherrypy_rest salt-api，cherrypy_rest is good but not fast we when have many minions to control at once

Versions Report

Salt Version:
Salt: 2018.3.3

Dependency Versions:
cffi: 1.6.0
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: 0.24.6
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.2.5
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.24.2
Python: 2.7.5 (default, Aug 4 2017, 00:39:18)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.3.1611 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-693.5.2.el7.x86_64
system: Linux
version: CentOS Linux 7.3.1611 Core

[liu624862560]

another thing is about salt-master, we just do this with development environment，the salt-master of product environment is more cpus and ram than dev env.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[DmitryKuzmenko]

@liu624862560 this sounds like your master is overloaded by a bunch of minion returns coming at once. There are a number of options in this case.

• Check your master performance and the network bandwidth. 400 minions sounds not like a verdict.
• Increase the timeout value and allowed retry count to make the system waiting more till the last one minion is responded. This is the simplest thing to do.
• Use splay executor that spreads the execution in time to not overload master by minions, but it constantly increases the execution time.
• Change your salt layout. You can use a number of Syndics to spread the main Master load. Syndics are collecting minions responses and sending them in a batch.

What way looks better for you?

I also want to hear someone else @saltstack/team-core.

[DmitryKuzmenko]

BTW, how it works if you're doing the same request not using salt-api?