[BUG] salt-ssh AttributeError: 'NoneType' object has no attribute 'count'

[xmsk]

Description of Issue

I'm getting a stack trace when deploying a large spring boot jar file to a target host using salt-ssh. Relevant stack trace

[ERROR   ] An un-handled exception was caught by salt's global exception handler:
AttributeError: 'NoneType' object has no attribute 'count'
Traceback (most recent call last):
  File "/usr/bin/salt-ssh", line 33, in <module>
    sys.exit(load_entry_point('salt==3003.3', 'console_scripts', 'salt-ssh')())
  File "/usr/lib/python3.9/site-packages/salt/scripts.py", line 479, in salt_ssh
    client.run()
  File "/usr/lib/python3.9/site-packages/salt/cli/ssh.py", line 26, in run
    ssh.run()
  File "/usr/lib/python3.9/site-packages/salt/client/ssh/__init__.py", line 800, in run
    ret = self.key_deploy(host, ret)
  File "/usr/lib/python3.9/site-packages/salt/client/ssh/__init__.py", line 454, in key_deploy
    if ret[host].get("stderr", "").count("Permission denied"):
AttributeError: 'NoneType' object has no attribute 'count'
Traceback (most recent call last):
  File "/usr/bin/salt-ssh", line 33, in <module>
    sys.exit(load_entry_point('salt==3003.3', 'console_scripts', 'salt-ssh')())
  File "/usr/lib/python3.9/site-packages/salt/scripts.py", line 479, in salt_ssh
    client.run()
  File "/usr/lib/python3.9/site-packages/salt/cli/ssh.py", line 26, in run
    ssh.run()
  File "/usr/lib/python3.9/site-packages/salt/client/ssh/__init__.py", line 800, in run
    ret = self.key_deploy(host, ret)
  File "/usr/lib/python3.9/site-packages/salt/client/ssh/__init__.py", line 454, in key_deploy
    if ret[host].get("stderr", "").count("Permission denied"):
AttributeError: 'NoneType' object has no attribute 'count'

Setup

A salt state forcing the described error has been provided in this GitHub repository https://github.com/xmsk/salt-ssh-stdout-json-parse.

The salt state is deploying a spring boot jar file to a target host using salt-ssh. The given salt state has been crafted in a way that forces the error each time. Usually the error only happens on the first deployment as subsequent deployments with the same jar file do not generate a diff in the file.managed state.

Setting the option show_changes: False on the relevant file.managed state fixes the issue as far as I can tell. However, getting to the point where I understood what was causing the issue was quite hard which is why I was hoping that some code changes could be made to make it more obvious how to fix the issue.

Steps to Reproduce Issue

Run the provided salt state using the command

salt-ssh -w -i 'salt-test' state.apply

Versions Report

I've used the latest salt-ssh version from the master branch running in a virtualenv

Salt Version:
          Salt: 3005+0na.edd8f78e
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.4
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.15.0
        pygit2: Not Installed
        Python: 3.10.5 (main, Jun  6 2022, 18:49:26) [GCC 12.1.0]
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: arch  
        locale: utf-8
       machine: x86_64
       release: 5.18.9-arch1-1
        system: Linux
       version: Arch Linux  

[xmsk]

After looking at the code I've found that the error is coming from this line of code

salt/salt/client/ssh/__init__.py

Line 470 in edd8f78

if ret[host].get("stderr", "").count("Permission denied"):

where the dict entry for stderr seems to be None which is why the default entry "" does not help in this case. In my opinion this is the first bug leading to the issue.

From what I understand the reason that stderr is None in this case is that the parsing of the minion response in the handle_routine method

salt/salt/client/ssh/__init__.py

Lines 533 to 567 in edd8f78

	def handle_routine(self, que, opts, host, target, mine=False):
	"""
	Run the routine in a "Thread", put a dict on the queue
	"""
	opts = copy.deepcopy(opts)
	single = Single(
	opts,
	opts["argv"],
	host,
	mods=self.mods,
	fsclient=self.fsclient,
	thin=self.thin,
	mine=mine,
	**target
	)
	ret = {"id": single.id}
	stdout, stderr, retcode = single.run()
	# This job is done, yield
	try:
	data = salt.utils.json.find_json(stdout)
	if len(data) < 2 and "local" in data:
	ret["ret"] = data["local"]
	else:
	ret["ret"] = {
	"stdout": stdout,
	"stderr": stderr,
	"retcode": retcode,
	}
	except Exception: # pylint: disable=broad-except
	ret["ret"] = {
	"stdout": stdout,
	"stderr": stderr,
	"retcode": retcode,
	}
	que.put(ret)

is failing. Although I don't understand why this is failing as the string returned in the stdout seems to be valid json (json.loads on it produces a normal python dict).

This leads me to believe that there is a second bug in the function find_json

salt/salt/utils/json.py

Lines 28 to 50 in edd8f78

	def find_json(raw):
	"""
	Pass in a raw string and load the json when it starts. This allows for a
	string to start with garbage and end with json but be cleanly loaded
	"""
	ret = {}
	lines = __split(raw)
	for ind, _ in enumerate(lines):
	try:
	working = "\n".join(lines[ind:])
	except UnicodeDecodeError:
	working = "\n".join(salt.utils.data.decode(lines[ind:]))
	try:
	ret = json.loads(working)
	except ValueError:
	continue
	if ret:
	return ret
	if not ret:
	# Not json, raise an error
	raise ValueError

that incorrectly throws a ValueError for a valid json file.

[lkubb]

This should be fixed in 3007+ by #64542.