[BUG] [3006.0] Regression: Include sls that includes jinja macros leads to Rendering SLS error

[babs]

Description

Include of sls that includes jinja macros leads to Rendering SLS 'base:common.file1' failed: Jinja error: 'NoneType' object has no attribute 'done'
Regression introduced in 3006.0, working fine on 3005.1 and previous.

state.apply common gives the following error while state.apply common.file1 works fine.

local:
    Data failed to compile:
----------
    Rendering SLS 'base:common.file1' failed: Jinja error: 'NoneType' object has no attribute 'done'
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 471, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 185, in get_source
    self.check_cache(_template)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 138, in check_cache
    ret = self.cache_file(template)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 131, in cache_file
    return fcl.get_file(saltpath, "", True, self.saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1167, in get_file
    hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1414, in hash_and_stat_file
    hash_result = self.hash_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1407, in hash_file
    return self.__hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1399, in __hash_and_stat_file
    return self.channel.send(load)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 457, in run_sync
    if not future_cell[0].done():
AttributeError: 'NoneType' object has no attribute 'done'

; line 1

---
{% from 'common/macros.jinja' import a_jinja_macro with context %}    <======================

a state id:
  cmd.run:
    - name: echo {{ a_jinja_macro("hello world") }}

[...]
---

Setup

One master and one minion running 3006.

• VM (vmware)
• onedir packaging

Steps to Reproduce the behavior

On the master side in a common folder:

• macros.jinja

  {% macro a_jinja_macro(arg) -%}
  {{ arg }}
  {%- endmacro %}
  

• init.sls

  include:
  - common.file1
  

• file1.sls

  {% from 'common/macros.jinja' import a_jinja_macro with context %}
  
  a state id:
    cmd.run:
      - name: echo {{ a_jinja_macro("hello world") }}
  
  

Expected behavior

On the minion side as from the master side, being able to call state.apply common without issue (as it was on 3005.1 and previous)

local:
----------
          ID: a state id
    Function: cmd.run
        Name: echo hello world
      Result: True
     Comment: Command "echo hello world" run
     Started: 15:58:45.411296
    Duration: 5.243 ms
     Changes:   
              ----------
              pid:
                  1259842
              retcode:
                  0
              stderr:
              stdout:
                  hello world

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   5.243 ms

Versions Report

salt --versions-report

Salt Version:
          Salt: 3006.0
 
Python Version:
        Python: 3.10.11 (main, Apr 14 2023, 05:57:16) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.11.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-21-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

[jbouter]

Affecting us too, following code:

AttributeError: 'NoneType' object has no attribute 'done'

; line 1

---
{% from "warpnet/map.jinja" import warpnet with context %}    <======================

/etc/profile.d:
  file.directory:
    - name: /etc/profile.d
    - mode: '0755'
[...]

[hurzhurz]

I think this change in the salt/utils/templates.py is causing it:
e29485e#diff-f77e29083ee4c1500d638fc2771ce95f3c76f5bf4172fa3cab7f57ea06f50318

If I manually replace the file with the version from before (https://github.com/saltstack/salt/blob/4ecfd3d3d549e655027a36746f9cf3bc9e11c6e9/salt/utils/templates.py), it seems to work.

[dwoz]

I am able to consistently reproduce this using the states in @babs's issue and state.apply common

[zbukhari-apex]

I'm running into this too but with yaml_import. Pretty much everywhere I use that, this is failing.

Surprisingly it seems to be specific to client side, I have a multi-master setup and the other minions are on 3003.2, everything is just fine there and all minions are running 3003.2 as well.

One master and one minion are on 3006.0 (latest of the repo fetched April 19th 2023).

master - 3006 aka mas6
minion - 3006 aka min6
master - 3003.2 aka mas3
minion - 3003.2 aka min3

Here's a test run of a formula/recipe/etc from min3 to mas6 (success):

username@min3:~$ sudo salt-call --master=mas6 state.apply formulas.salt -l warning --state-output=changes
local:
  Name: /usr/local/bin/salt-restart.sh - Function: file.managed - Result: Clean Started: - 01:26:05.543681 Duration: 184.929 ms
  Name: /etc/salt/minion - Function: file.managed - Result: Clean Started: - 01:26:05.728844 Duration: 153.434 ms
  Name: /etc/salt/minion.d/masters.conf - Function: file.absent - Result: Clean Started: - 01:26:05.882470 Duration: 0.901 ms
  Name: /etc/salt/minion.d/master.conf - Function: file.absent - Result: Clean Started: - 01:26:05.883528 Duration: 0.812 ms
  Name: echo '*/3 * * * * root /usr/local/bin/apex-salt-restart.sh minion && rm /etc/cron.d/salt-minion-restart' > /etc/cron.d/apex-salt-minion-restart
chmod 444 /etc/cron.d/salt-minion-restart
 - Function: cmd.run - Result: Clean Started: - 01:26:05.885512 Duration: 0.005 ms
  Name: /usr/local/bin/salt-upgrader.sh - Function: file.managed - Result: Clean Started: - 01:26:05.885586 Duration: 212.082 ms
  Name: /etc/cron.d/salt-upgrader - Function: file.absent - Result: Clean Started: - 01:26:06.098022 Duration: 1.312 ms

Summary for local
------------
Succeeded: 7
Failed:    0
------------
Total states run:     7
Total run time: 553.475 ms

Here's a run of min6 to mas6 (fails):

user@min6:~$ sudo salt-call --master=mas6 state.apply formulas.salt -l warning --state-output=changes
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 471, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 4, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 185, in get_source
    self.check_cache(_template)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 138, in check_cache
    ret = self.cache_file(template)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 131, in cache_file
    return fcl.get_file(saltpath, "", True, self.saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1167, in get_file
    hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1414, in hash_and_stat_file
    hash_result = self.hash_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1407, in hash_file
    return self.__hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1399, in __hash_and_stat_file
    return self.channel.send(load)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 457, in run_sync
    if not future_cell[0].done():
AttributeError: 'NoneType' object has no attribute 'done'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 218, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 515, in render_jinja_tmpl
    raise SaltRenderError(
salt.exceptions.SaltRenderError: Jinja error: 'NoneType' object has no attribute 'done'
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 471, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 4, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 185, in get_source
    self.check_cache(_template)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 138, in check_cache
    ret = self.cache_file(template)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 131, in cache_file
    return fcl.get_file(saltpath, "", True, self.saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1167, in get_file
    hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1414, in hash_and_stat_file
    hash_result = self.hash_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1407, in hash_file
    return self.__hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1399, in __hash_and_stat_file
    return self.channel.send(load)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 457, in run_sync
    if not future_cell[0].done():
AttributeError: 'NoneType' object has no attribute 'done'

; line 4

---
---
{#- # Use lsb_release -c -s to get the short oscodename or grains.oscodename
    # as we test against to determine if we need to update / upgrade. #}
{%- import_yaml "formulas/salt/data.yaml" as data %}    <======================
{#- # We're also going to put the logic for saltversion here as I wanted
    # to simplify the YAML and reduce redundancy yet make it more detailed
    # as we know have redundancy and test hosts. #}
{#- # First we account for the actual masters. #}
{%- if grains.id in data['lower']['masters'].keys() %}
[...]
---
[CRITICAL] Rendering SLS 'base:formulas.salt.upgrader' failed: Jinja error: 'NoneType' object has no attribute 'done'
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 471, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 4, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 185, in get_source
    self.check_cache(_template)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 138, in check_cache
    ret = self.cache_file(template)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 131, in cache_file
    return fcl.get_file(saltpath, "", True, self.saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1167, in get_file
    hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1414, in hash_and_stat_file
    hash_result = self.hash_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1407, in hash_file
    return self.__hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1399, in __hash_and_stat_file
    return self.channel.send(load)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 457, in run_sync
    if not future_cell[0].done():
AttributeError: 'NoneType' object has no attribute 'done'

; line 4

---
---
{#- # Use lsb_release -c -s to get the short oscodename or grains.oscodename
    # as we test against to determine if we need to update / upgrade. #}
{%- import_yaml "formulas/salt/data.yaml" as data %}    <======================
{#- # We're also going to put the logic for saltversion here as I wanted
    # to simplify the YAML and reduce redundancy yet make it more detailed
    # as we know have redundancy and test hosts. #}
{#- # First we account for the actual masters. #}
{%- if grains.id in data['lower']['masters'].keys() %}
[...]
---
local:
    Data failed to compile:
----------
    Rendering SLS 'base:formulas.salt.upgrader' failed: Jinja error: 'NoneType' object has no attribute 'done'
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 471, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 4, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 185, in get_source
    self.check_cache(_template)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 138, in check_cache
    ret = self.cache_file(template)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 131, in cache_file
    return fcl.get_file(saltpath, "", True, self.saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1167, in get_file
    hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1414, in hash_and_stat_file
    hash_result = self.hash_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1407, in hash_file
    return self.__hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1399, in __hash_and_stat_file
    return self.channel.send(load)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 457, in run_sync
    if not future_cell[0].done():
AttributeError: 'NoneType' object has no attribute 'done'

; line 4

---
---
{#- # Use lsb_release -c -s to get the short oscodename or grains.oscodename
    # as we test against to determine if we need to update / upgrade. #}
{%- import_yaml "formulas/salt/data.yaml" as data %}    <======================
{#- # We're also going to put the logic for saltversion here as I wanted
    # to simplify the YAML and reduce redundancy yet make it more detailed
    # as we know have redundancy and test hosts. #}
{#- # First we account for the actual masters. #}
{%- if grains.id in data['lower']['masters'].keys() %}
[...]
---

Anyway - this is not an issue unless you think it is. This is what happens when a 3005/3006 minion tries to talk to any 3003.2 master (not following the best practice of upgrading the master first so I think this is expected, but meh):

2023-04-15 23:59:48,913 [salt.minion      :1146][ERROR   ][362612] Error while bringing up minion for multi-master. Is master at mas3 responding?
2023-04-15 23:59:58,934 [salt.crypt       :788 ][ERROR   ][362612] Sign-in attempt failed: {'enc': 'pub', 'pub_key': '_da_pub_key_', 'publish_port': 4505, 'token': '_da_token_', 'aes' : '_da_aes_', 'sig': "_da_sig_"}

[hkbakke]

Even the official salt-formula breaks completely with 3006.0 with this error. Although mistakes can happen, it is surprising that an issue as showstoppingly grave as this one manages to go undetected in a LTS release. It is almost like this release has never been tested on an actual production-like system before release. Also, the upgrade was a mess as the salt-master now runs as a normal user that has no access to lots of stuff like ssh keys with correct permissions for root etc,. This is good for security and I applaud it, however this should be clearly noted in the release notes so we have some pointers to why everything breaks.

Hopefully a quick fix release can happen for this specific issue so 3006 is at least usable with actual formulas.

[jbouter]

@hkbakke Even though it is off-topic, I couldn't agree more with the salt-master process now running as a salt user. I too applaud the increase in security, but it rendered part of our environment unreachable as the master process could no longer read our stack repository, resulting in configured firewall rules being removed... I honestly wish it would have been more clearly documented as it is a possibly breaking change.

As for not being tested, it's really surprising indeed. It could have to do with the commit that causes this bug being only two weeks old, but it should have been caught. I'm all for updating software as quickly as feasible within an organisation, but it doesn't seem like salt has had a good .0 release in a long while.