[FEATURE REQUEST] Filtering and templating the vault pillar module

[dgengtek]

Is your feature request related to a problem? Please describe.
The current vault pillar module is pretty static and only allows filtering by the minion id. #47817 and #47483 are probably related.

I opened this issue to see if this feature would be desired to be merged into the tree before I open a pull request since it would require a bit more work to do some cleanup, additional documentation, making tests work from the salt repository and removing environment specific code.

Describe the solution you'd like
I have written a pillar extension_module for this with tests which allows filtering results by keys and values for a vault lookup which allows filtering by either static values for each key or dynamic values(only grains and pillar lookups currently).

Also templating of the mapping of a vault path -> pillar path with either nested lookups(via jinja2 templating), grains, pillars lookups(using results or exact matches of a single value of the result of the lookup) or wildcards.

Additional context

I could keep support for the old config style since filtering and templates are just an addition and this mapping is just basically the same. I just liked it more explict with using path and pillarpath as keys and iterating over each config as a list instead of adding multiple vault module definitions in master config.

The following examples of possible configurations with the module show the configs from the current implementation.

Single value lookup

Mapping a single value of vault path to a pillarpath

ext_pillar:
  - vault:
      config:
        # single mapping
        - path: secret/salt
          pillarpath: keyname1

equivalent of the old config

ext_pillar:
  - vault:
       conf: path=secret/salt
       nesting_key: keyname1

vault lookup:

private: ...
public: ...

mapped to pillar:

keyname1:
  private: ...
  public: ...

Wildcards

Example of using wildcards. There must be an equal amount of wildcards for both paths.

ext_pillar:
  - vault:
      config:
        # map each entry from list lookup of ../workers/* to ../worker_keys/* in pillar
        # c1
        - path: secret/workers/*
          pillarpath: somepath/*
        # c2
        - path: secret/nested/*/secrets/*
          pillarpath: mapped/*/secret/*/value

For c1.

path: secret/workers/*
pillarpath: somepath/*

path wildcard of vault contains ["w1", "w2"]. For each item a lookup will be done and the result mapped to pillarpath

secret/workers/w1 -> somepath/w1
secret/workers/w2 -> somepath/w2

For c2.

path: secret/*/items/* 
pillarpath: mapped/*/secret/*/value

The products of each result will be combined and mapped to the pillarpath

1. secret/* returns ["s1", "s2"]

secret/s1/items/* returns ["i1", "i2"]
secret/s2/items/* returns ["i3", "i4"]

secret/s1/items/i1 -> mapped/s1/secret/i2/value
secret/s1/items/i2 -> mapped/s1/secret/i2/value
secret/s2/items/i3 -> mapped/s2/secret/i3/value
secret/s2/items/i4 -> mapped/s2/secret/i4/value

Filtering with keys and values

Example of filtering with single lookup

ext_pillar:
  - vault:
      config:
        - path: secret/path
          pillarpath: path/value
          # use all keys from the vault path lookup
          keys: []
          matches:
            - keys: 
                - host
              # if any keys of this match definition matches any of the values defined here, keep them in the keyset from the vault lookup
              values: 
                - host3
                - host7
                - host9
              # otherwise remove the listed keys here from the keyset defined in the outer definition 'keys'
              add_keys:
                - private

vault lookup of secret/path

host: host3
private: pkey
anotherkey: anothervalue

value of the key host is in the values of the matchers so the key 'private' from the vault lookup data will not be removed

path:
  value:
    host: host3
    private: pkey
    anotherkey: anothervalue

if the vault lookup was

  host: host1
  private: pkey
  anotherkey: anothervalue

then the mapped pillar would look

path:
  value:
    host: host1
    anotherkey: anothervalue

Grains and pillars in filtering

Using grains or pillars in values of matchers

ext_pillar:
  - vault:
      config:
      # c1.
        - path: secret/path
          pillarpath: path/value
          # only select the keys named 'public' and 'private' from the vault lookup keyset
          keys:
            - public
            - private
          matches:
            - keys: 
                - host
              # lookup grain id to check against the value of the vault lookup
              values: 
                - G@id
              # add keys if any of the values matched the lookup of the path, otherwise remove listed keys from the list of keys to add
              add_keys:
                - private
            - keys: 
                - location
              # lookup pillar path 'locations:west:hosts' to check against the value of the vault lookup of the key 'location'
              values: 
                - I@locations:west:hosts
                - I@hosts
              # add keys if any of the values matched the lookup of the path, otherwise remove listed keys from the list of keys to add
              add_keys:
                - public
      # c2.
        - path: secret/app1
          pillarpath: app1
          keys:
            - token
          matches:
            - keys: 
                - slave
              values: 
                - I@appslaves
              add_keys:
                - token

vault lookup of c1. secret/path

host: host3
public: pubkey
private: pkey
anotherkey: anothervalue

if the value of the key host matches the result of G@id then the pillar would be

path:
  value:
    public: pubkey
    private: pkey

if the value of G@id was host1 then instead the mapped pillar would look

path:
  value:
    public: pubkey

Filtering all or none

Example of adding all keys or none if the matcher is not true.

ext_pillar:
  - vault:
      config:
        - path: "secret/path"
          pillarpath: "path/values"
          # use all keys from the vault path lookup
          keys: []
          matches:
          # if keys empty and values are not empty then merge all keys
            - keys: []
              values: 
                # only add all keys if the pillar lookup path 'somevalue:defined' is not empty
                # otherwise no keys will be added
                - 'I@somevalue:defined'
              add_keys: []

Matching against a value of a pillar lookup

Example of matching against an exact value of a pillar lookup

ext_pillar:
  - vault:
      config:
        - path: "secret/salt/concourse/master/session_signing_key"
          pillarpath: "concourse/web/session_signing_key"
          # use all keys from the vault path lookup
          keys: []
          matches:
          # if keys empty and values are not empty then merge all keys, otherwise none are added
            - keys: []
              values: 
                # only do it if the pillar lookup of 'tags' contains 'concourse-master'
                - 'I@tags=concourse-master'
              add_keys: []

Using grains or pillars in the paths

These will expand and create multiple pairs matching their result. A lookup
result which contains a list will iterate over it. For a dictionary only the
keynames will be replaced for it.

If only one or the other of the paths contains a lookup definition its lookup value
must be a single value. If the lookup value contains a list or dict then both
paths need to have the same lookup definition otherwise it will be unable to
create pairs to map against.

ext_pillar:
  - vault:
      config:
        # single value lookup is fine
        - path: "secret/salt/concourse/workers/G@id"
          pillarpath: "concourse/worker/G@id/worker_key"
        # single value lookup is fine even if the other path contains no lookups
        - path: "secret/salt/concourse/workers/G@id"
          pillarpath: "concourse/worker/secret"
        # this lookup must be a single value 
        - path: "secret/personal/I@some:path"
          pillarpath: "private/secret"
        # if this lookup contains a list or dict then the other path must have the same lookup definition
        - path: "secret/personal/I@persons"
          pillarpath: "private/I@persons"
        # if pillar clustermembers returns a list, run a vault lookup for each item
        #  or if it is a value only a single lookup
        - path: hosts/I@clustermembers/secret
          # if the lookup was a single value, map to pillar 'myapp'
          #   if it was a list then the lookup is required to be in pillarpath too
          # pillarpath: myapp/I@clustermembers
          #   otherwise there is no way to know how to map
          pillarpath: myapp

Vault lookup for path: "secret/personal/I@persons" to pillarpath: "private/I@persons"

I@persons returns a list of ["p1", "p2"] or if it can return a dict of {"p1": {...}, "p2": ...}

secret/personal/p1 -> private/p1
secret/personal/p2 -> private/p2

Mapped pillar would be merged to

private:
  p1:
    # result from the vault lookup mapped
    # key: value 
    ...
  p2:
    ...

Matching against a value of a lookup result in paths

Using different amounts for each path or multiple this way is fine since it
will map to one path each.

ext_pillar:
  - vault:
      config:
      # do a lookup of the pillar I@cluster and continue if it resolves to master1
      # do a lookup of the pillar I@role and continue only if it is 'backup'
          # path -> hosts/master1/webtoken
          # pillarpath -> myapp/config/backup/token
        - path: hosts/I@cluster=master1/webtoken
          pillarpath: myapp/config/I@role=backup/token
      # using multiple different amounts
        - path: hosts/I@cluster=master1/webtoken
          pillarpath: myapp/I@app=app1/I@role=backup/token
        - path: hosts/I@backupserver=b1/secret
          pillarpath: backup/hosts/G@id/I@location/token

With for example

path: hosts/I@cluster=master1/webtoken
pillarpath: myapp/I@app=app1/I@role=backup/token

If I@cluster returns ["master1", "master2"] and I@role returns
["server", "backup"] and I@app returns ["app1", ...] then the mapping will be hosts/master1/webtoken -> myapp/app1/backup/token. Note the lookup result can be a single value or a dict in which case the keys of the dict will be used to compare.

Jinja2 templates

Templates are mainly used to do lookups of a result of a previous lookup(a
template depends on a previous templates result).

The constraint for using the keys from a template definition in paths is that
all keys are of the same root template definition and either

1. the used keys are from the same templates used
   k1 and k2 are from the same template

    - path: secret/{{k1}}/{{k2}}
      pillarpath: new/{{k2}}
      template:
        - keys:
          - k1
          - k2
          lookup: "I@somelookup"

2. or all the templates of the keys used in pillarpath contain all the
   templates from path
   ie is the set of the templates of the keys of path a subset of the set of
   the templates of the keys of pillarpath

Practically one has to make sure that the templates can be resolved in order of
the list they are defined in. Currently did not test multiple template
dependency resolves with a different root in the same config(since there was no need for it in my environment).

Example of using jinja2 templates in paths and template definitions which depend
on a previous template definition.

ext_pillar:
  - vault:
      config:
        # lookup keys for templates {{key}} defined in paths
        - path: "secret/ssh/users/{{user}}/{{id}}"
          pillarpath: "users/present/{{puser}}/ssh_keys/{{id}}"
          keys:
            - public
            - private
          matches:
            - keys: 
                - host
              values: 
                - G@id
              add_keys:
                - private
        # keys from a lookup can be used for the following lookups in the list
          template:
            # keys gets the values from the lookup and is used for templating paths or templates which depend on this key
            # if 'ext_pillar_vault_ssh_keys:present' is a dict, puser will be the keyname for each key of the dict lookup result
            # if 'ext_pillar_vault_ssh_keys:present' is a list, puser will be the name for each item of the list
            # invalid if there are multiple keys listed and the lookup only resolves to a single value, list of values and not a dict
          # t1.
            - keys:
              - puser
              lookup: "I@ext_pillar_vault_ssh_keys:present"
            # multiple keys can be used to match the result
            # if the lookup is a dict the listed keys will be used for the lookup of the dict
            # if the lookup is a list of dict then the keys named here will be used from lookup for each dict in the list
          # t2.
            - keys:
              - id
              - user
              lookup: "I@ext_pillar_vault_ssh_keys:present:{{puser}}"

The pillar lookup of template t1 'ext_pillar_vault_ssh_keys:present' returns

admin:
  - user: backup
    id: id_backup
  - user: root
    id: intranet
root:
  - user: root
    id: id_root

Key 'puser' will be ["admin", "root"]

The pillar lookup of template t2 'ext_pillar_vault_ssh_keys:present:admin' returns the list

- user: backup
  id: id_backup
- user: root
  id: intranet

Keys and value pairs will consist of the pairs in a list [(id: id_backup, user: backup), (id: intranet, user: root)]

The pillar lookup of template t2 'ext_pillar_vault_ssh_keys:present:root' returns

- user: root
  id: intranet

Keys and value pairs will be [(id: id_root, user: root)]

mapping of values for each pair:

secret/ssh/users/backup/id_backup -> users/present/admin/ssh_keys/id_backup
secret/ssh/users/root/intranet -> users/present/admin/ssh_keys/intranet

secret/ssh/users/root/id_root -> users/present/root/ssh_keys/intranet

Mixing of templates, lookups, wildcards.

Order of rendering to get the resolved paths for vault lookup and pillarpath:

1. templates in the order of definition will be resolved first
2. then the lookups will be rendered on the results
3. wildcards are iterated over last

ext_pillar:
  - vault:
      config:
        - path: "secret/apps/*/{{worker}}/I@appslaves"
          pillarpath: "app/*/users/{{keyid}}/ssh_keys/I@appslaves"
          template:
          # t1.
            - keys:
              - worker
              lookup: "I@workers"
          # t2.
            - keys:
              - keyid
              lookup: "I@workers:{{worker}}:config:keyid"

Rendering in order

1. Original pillar definition

workers:
  w1:
    config:
      keyid: | ...
  w2:
    config:
      keyid: | ...

Result of template t1.

worker: ["w1", "w2"]

Result of template t2.

keyid: ["k1", "k2"]

2. Salt pillar lookup of I@appslaves is ["s1", "s2"]

3. wildcard vault listing secret/apps/* -> ["app1", "app2"]

path: "secret/apps/*/{{worker}}/I@appslaves"
pillarpath: "app/*/users/{{keyid}}/ssh_keys/I@appslaves"

Mapping the resolved paths

Result of rendering in order for the paths:

  secret/apps/*/w1/I@appslaves -> app/*/users/k1/ssh_keys/I@appslaves
  secret/apps/*/w2/I@appslaves -> app/*/users/k2/ssh_keys/I@appslaves

  secret/apps/*/w1/s1 -> app/*/users/k1/ssh_keys/s1
  secret/apps/*/w1/s2 -> app/*/users/k1/ssh_keys/s2
  secret/apps/*/w2/s1 -> app/*/users/k2/ssh_keys/s1
  secret/apps/*/w2/s2 -> app/*/users/k2/ssh_keys/s2

  secret/apps/app1/w1/s1 -> app/app1/users/k1/ssh_keys/s1
  secret/apps/app2/w1/s1 -> app/app2/users/k1/ssh_keys/s1
  secret/apps/app1/w1/s2 -> app/app1/users/k1/ssh_keys/s2
  secret/apps/app2/w1/s2 -> app/app2/users/k1/ssh_keys/s2
  secret/apps/app1/w2/s1 -> app/app1/users/k2/ssh_keys/s1
  secret/apps/app2/w2/s1 -> app/app2/users/k2/ssh_keys/s1
  secret/apps/app1/w2/s2 -> app/app1/users/k2/ssh_keys/s2
  secret/apps/app2/w2/s2 -> app/app2/users/k2/ssh_keys/s2

If anything fails the pair will not be considered for pillar merging.

Any feedback welcome

[dgengtek]

If anyone is interested in this I created a repository with the pillar extension module which implements what is written in this feature request.

https://github.com/dgengtek/salt-ext-pillar-vault-lookup