You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Description
Salt-master times out when doing salt-api /auth endpoint request without username or password fields but with eauth field present. Can lead to salt-api DDoS.
Setup
Really the most generic salt setup you can ever create, just enabled salt-api with external_auth (config below)
VM KVM Master and slave
Steps to Reproduce the behavior
Install salt-master, salt-minion and salt-api packages
- See salt-api getting stuck, master errors in logs, retries in salt-api (every retry takes 1 minute - why is this default?)
Salt-api stuck logs
2022-06-17 10:20:11,965 [salt.loaded.int.netapi.rest_cherrypy.app:747 ][INFO ][3027991] [api_acl] Authentication not checked for user admin from IP 192.168.41.101
2022-06-17 10:20:11,974 [salt.transport.zeromq:158 ][DEBUG ][3027991] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
2022-06-17 10:20:11,976 [salt.transport.zeromq:1233][DEBUG ][3027991] Trying to connect to: tcp://127.0.0.1:4506
2022-06-17 10:21:12,036 [salt.transport.zeromq:1291][DEBUG ][3027991] SaltReqTimeoutError, retrying. (1/3)
2022-06-17 10:22:12,094 [salt.transport.zeromq:1291][DEBUG ][3027991] SaltReqTimeoutError, retrying. (2/3)
2022-06-17 10:23:12,156 [salt.transport.zeromq:1291][DEBUG ][3027991] SaltReqTimeoutError, retrying. (3/3)
2022-06-17 10:24:12,212 [salt.transport.zeromq:1261][DEBUG ][3027991] Re-init ZMQ socket: Message timed out
2022-06-17 10:24:12,214 [salt.transport.zeromq:1233][DEBUG ][3027991] Trying to connect to: tcp://127.0.0.1:4506
2022-06-17 10:24:12,216 [salt.transport.zeromq:179 ][DEBUG ][3027991] Closing AsyncZeroMQReqChannel instance
2022-06-17 10:24:12,218 [cherrypy.access.140230875023728:284 ][INFO ][3027991] 192.168.41.101 - - [17/Jun/2022:10:24:12] "POST /login HTTP/1.1" 503 742 "" "curl/7.79.1"
Salt-master logs
022-06-17 10:20:09,059 [salt.config :1917][DEBUG ][3027939] Reading configuration from /etc/salt/master
2022-06-17 10:20:09,062 [salt.config :2080][DEBUG ][3027939] Including configuration from '/etc/salt/minion.d/_schedule.conf'
2022-06-17 10:20:09,062 [salt.config :1917][DEBUG ][3027939] Reading configuration from /etc/salt/minion.d/_schedule.conf
2022-06-17 10:20:09,063 [salt.config :2080][DEBUG ][3027939] Including configuration from '/etc/salt/minion.d/local_master.conf'
2022-06-17 10:20:09,063 [salt.config :1917][DEBUG ][3027939] Reading configuration from /etc/salt/minion.d/local_master.conf
2022-06-17 10:20:09,064 [salt.config :3520][DEBUG ][3027939] Using cached minion ID from /etc/salt/minion_id: vm-3d-salt-test.pine.ly
2022-06-17 10:20:09,066 [salt.loader :868 ][DEBUG ][3027939] Grains refresh requested. Refreshing grains.
2022-06-17 10:20:09,066 [salt.config :1917][DEBUG ][3027939] Reading configuration from /etc/salt/master
2022-06-17 10:20:09,086 [salt.utils.entrypoints:57 ][DEBUG ][3027939] Using importlib_metadata to load entry points
2022-06-17 10:20:09,141 [salt.loader.lazy :857 ][DEBUG ][3027939] Override __utils__: <module 'salt.loaded.int.grains.zfs' from '/usr/lib/python3/dist-packages/salt/grains/zfs.py'>
2022-06-17 10:20:09,170 [salt.modules.network:2132][DEBUG ][3027939] Elapsed time getting FQDNs: 0.017315387725830078 seconds
2022-06-17 10:20:09,335 [salt.utils.lazy :99 ][DEBUG ][3027939] LazyLoaded zfs.is_supported
2022-06-17 10:20:09,398 [salt.utils.lazy :99 ][DEBUG ][3027939] LazyLoaded local_cache.clean_old_jobs
2022-06-17 10:20:09,435 [salt.utils.lazy :99 ][DEBUG ][3027939] LazyLoaded localfs.list_tokens
2022-06-17 10:20:09,437 [salt.utils.verify:454 ][DEBUG ][3027939] This salt-master instance has accepted 1 minion keys.
2022-06-17 10:20:11,981 [salt.utils.lazy :99 ][DEBUG ][3027950] LazyLoaded auto.auth
2022-06-17 10:20:11,982 [salt.transport.zeromq:795 ][ERROR ][3027950] Some exception handling a payload from minion
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 791, in handle_message
ret, req_opts = yield self.payload_handler(payload)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 294, in wrapper
result = func(*args, **kwargs)
File "/usr/lib/python3.8/types.py", line 278, in wrapped
coro = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/master.py", line 1023, in _handle_payload
ret = {"aes": self._handle_aes, "clear": self._handle_clear}[key](load)
File "/usr/lib/python3/dist-packages/salt/master.py", line 1064, in _handle_clear
ret = method(load), {"fun": "send_clear"}
File "/usr/lib/python3/dist-packages/salt/master.py", line 2130, in mk_token
token = self.loadauth.mk_token(clear_load)
File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 209, in mk_token
if not self.authenticate_eauth(load):
File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 312, in authenticate_eauth
if not self.time_auth(load):
File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 115, in time_auth
ret = self.__auth_call(load)
File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 98, in __auth_call
fcall = salt.utils.args.format_call(
File "/usr/lib/python3/dist-packages/salt/utils/args.py", line 470, in format_call
raise SaltInvocationError(
salt.exceptions.SaltInvocationError: auth takes at least 2 arguments (1 given)
2022-06-17 10:20:11,990 [tornado.general :467 ][ERROR ][3027950] Uncaught exception in zmqstream callback
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 435, in send_multipart
memoryview(msg)
TypeError: memoryview: a bytes-like object is required, not 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 460, in _handle_events
self._handle_send()
File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 499, in _handle_send
status = self.socket.send_multipart(msg, **kwargs)
File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 440, in send_multipart
raise TypeError(
TypeError: Frame 0 ('Some exception handling minion ...) does not support the buffer interface.
2022-06-17 10:20:11,994 [tornado.application:640 ][ERROR ][3027950] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f8617aeb5e0>)
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 435, in send_multipart
memoryview(msg)
TypeError: memoryview: a bytes-like object is required, not 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
ret = callback()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 542, in <lambda>
self.io_loop.add_callback(lambda : self._handle_events(self.socket, 0))
File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 460, in _handle_events
self._handle_send()
File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 499, in _handle_send
status = self.socket.send_multipart(msg, **kwargs)
File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 440, in send_multipart
raise TypeError(
TypeError: Frame 0 ('Some exception handling minion ...) does not support the buffer interface.
- All further requests to salt-api via rest_cherrypy salt-api server become stuck
- Further bogus requests prolong server downtime
- If we stop doing bad requests server will come back
Expected behavior
Do bad request to salt-api (/login without password field)
See bad request error and not connection getting stuck
Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
Salt: 3004.1Dependency Versions:
cffi: Not Installedcherrypy: unknowndateutil: 2.7.3docker-py: Not Installedgitdb: 2.0.6gitpython: 3.0.7Jinja2: 2.10.1libgit2: 0.28.3M2Crypto: Not InstalledMako: Not Installedmsgpack: 0.6.2msgpack-pure: Not Installedmysql-python: Not Installedpycparser: Not Installedpycrypto: 2.6.1pycryptodome: 3.6.1pygit2: 1.0.3Python: 3.8.10 (default, Mar 15 2022, 12:22:08)python-gnupg: 0.4.5PyYAML: 5.3.1PyZMQ: 18.1.1smmap: 2.0.5timelib: Not InstalledTornado: 4.5.3ZMQ: 4.3.2System Versions:
dist: ubuntu 20.04 focallocale: utf-8machine: x86_64release: 5.13.0-41-genericsystem: Linuxversion: Ubuntu 20.04 focal
Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:
There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!
Description
Salt-master times out when doing salt-api
/auth
endpoint request withoutusername
orpassword
fields but witheauth
field present. Can lead to salt-api DDoS.Setup
Really the most generic salt setup you can ever create, just enabled salt-api with
external_auth
(config below)Steps to Reproduce the behavior
external_auth
in master configExample master config
Example curl
Salt-api stuck logs
Salt-master logs
Expected behavior
/login
without password field)Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Additional context
Problem can be easily solved by placing try block before
format_call()
func here: https://github.com/saltstack/salt/blob/master/salt/auth/__init__.py#L98-L101The text was updated successfully, but these errors were encountered: