saltclass fails to expand all variables

[WackyOne]

Saltclass variable expansion does not work if the variable is in a list, such as if you have a list of dictionaries in the pillar. This functionality worked in reclass.

Make a pillar with a list of dictionaries and put a variable that you want expanded in the list

pillars:
  substitute:
    key: value
  list_of_dicts:
    - id: dict1
      a: ${substitute:key}
    - id: dict2
      b: ${substitute:key}

Run a pillar.get on list_of_dicts and the variable ${substitute:key} will not be expanded.

# salt 'minion' pillar.get list_of_dicts
minion:
    |_
      ----------
      a:
          ${substitute:key}
      id:
          dict1
    |_
      ----------
      b:
          ${substitute:key}
      id:
          dict2

# salt-call --versions-report
Salt Version:
           Salt: 2018.3.0
 
Dependency Versions:
           cffi: 1.5.2
       cherrypy: 3.6.0
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 1.0.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: 1.2.5
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.13 (default, Jan 03 2017, 17:41:54) [GCC]
   python-gnupg: 0.3.7
         PyYAML: 3.12
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.2.2
 
System Versions:
           dist: SuSE 42.3 x86_64
         locale: UTF-8
        machine: x86_64
        release: 4.4.114-42-default
         system: Linux
        version: openSUSE  42.3 x86_64

[Ch3LL]

can you include how you have reclass setup with information sanitized?

Also was this working previously with your current setup and if so what version?

[WackyOne]

I have reclass working with Salt 2017.7.2 and reclass version 1,4,1.

I use a YAML anchor in my master config:

reclass: &reclass
  storage_type: yaml_fs
  inventory_base_uri: /srv/reclass

and then use it like:

master_tops:
  reclass: *reclass

ext_pillar:
  - reclass: *reclass

For the node files something like /srv/reclass/nodes/minion.yml:

classes:
  - some_app
environment: base
parameters:
  hostname: minion

and the class files something like /srv/reclass/classes/some_app.yml

applications:
  - some_app
parameters:
  substitute:
    key: value
  list_of_dicts:
    - id: dict1
      a: ${substitute:key}
    - id: dict2
      a: ${substitute:key}

Then in /srv/salt/some_app/init.sls you could have:

{% set data = salt['pillar.get']('list_of_dicts', {}) %}

{% for item in data -%}

touch /tmp/{{ item.id }}_{{ item.a }}:
  cmd.run:
    - creates: /tmp/{{ item.id }}_{{ item.a }}
{% endfor %}

You can see the variables are expanded properly with pillar_get:

# salt 'minion' pillar.get list_of_dicts
minion:
    |_
      ----------
      a:
          value
      id:
          dict1
    |_
      ----------
      a:
          value
      id:
          dict2

and that the state some_app runs fine:

# salt 'minion' state.sls some_app
minion:
----------
          ID: touch /tmp/dict1_value
    Function: cmd.run
      Result: True
     Comment: Command "touch /tmp/dict1_value" run
     Started: 09:28:23.855013
    Duration: 24.864 ms
     Changes:   
              ----------
              pid:
                  5311
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: touch /tmp/dict2_value
    Function: cmd.run
      Result: True
     Comment: Command "touch /tmp/dict2_value" run
     Started: 09:28:23.880336
    Duration: 17.711 ms
     Changes:   
              ----------
              pid:
                  5312
              retcode:
                  0
              stderr:
              stdout:

Summary for minion
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  42.575 ms

[Ch3LL]

i'm actually running into this: #47126

did you happen to run into this issue before running into the issue you reported?

[WackyOne]

Yes after updating to 2018.3.0 I had that problem and so that's why I was trying saltclass. I am now back at 2017.7.2 until I can get saltclass or reclass working with the newer version.

[Ch3LL]

oh gotchya i was trying reclass thanks for clarifying. ping @olivier-mauras looks like you added this feature. any chance you are seeing the same behavior in saltclass in 2018.3.0 or any ideas here as well?

[olivier-mauras]

I completely overlooked that case - all lists actually :(
Seems to work after modification in my tests and your example:

classes:
  - default.users
  - default.motd

states:
  - openssh

pillars:
  default:
    network:
      dns:
        srv1: 192.168.0.1
        srv2: 192.168.0.2
        domain: example.com
      ntp:
        srv1: 192.168.10.10
        srv2: 192.168.10.20
    test_on_list:
      - a: ${default:network:dns:srv1}
      - b: ${default:network:dns:srv2}
      - ${default:network:dns:domain}
  list_of_dicts:
    - id: dict1
      a: ${default:network:ntp:srv1}
    - id: dict2
      a: ${default:network:ntp:srv2}

Now gives

(...)
  test_on_list:
  - a: 192.168.0.1
  - b: 192.168.0.2
  - example.com
(...)
list_of_dicts:
- a: 192.168.10.10
  id: dict1
- a: 192.168.10.20
  id: dict2

Can you please test this attached patched and confirm this is correctly working for you before I send the patch in?
https://gist.github.com/olivier-mauras/248c02615e12fea687ee4b7929946d46

[WackyOne]

I am off until Monday and will test it then. I appreciate the speedy response!

[WackyOne]

The patch doesn't work and I believe the problem is related to unicode strings. If I change the checks for instances of 'str' to 'basestring' it works. Perhaps there is something in my setup that needs to be changed?

[olivier-mauras]

Ah indeed using only str for type checking was a bit naive and Python3 centric.
Salt normally handles 2 and 3 support with only basestring through the internal six librairy.
Let me give a couple more testing under Python3 and I'll push a PR with basestring

[olivier-mauras]

@WackyOne I've updated the patch on https://gist.github.com/olivier-mauras/248c02615e12fea687ee4b7929946d46
I've tested it on both python 2.7 and 3.6 with the same result. Let me know how this works for you.

[WackyOne]

@olivier-mauras that patch does the trick. Thanks so much!

[olivier-mauras]

Perfect I'll send a PR with a list expansion test included