[BUG] pillar_roots goes intermittent when git_pillar is also enabled

[ichilton]

Description

If I use pillar_roots alone, it works fine.
If I use git_pillar alone, it seems to work fine.

However, if I enable both at the same time, variables defined in pillar_roots are intermittently available - it appears and disappears on subsequent salt runs.

If I comment out the git_pillar config, it's all fine again.

Setup

Have spent hours and hours going through the documentation and trying different things/config and can not stop this happening. This is the basic config though:

pillar_roots:
  __env__:
    - /srv/salt/pillar

ext_pillar:
  - git:
    - __env__ https://git.mydomain.com/myrepo.git:
      - root: pillar

root@salt-master:/srv/salt/pillar# cat top.sls
{{ saltenv }}:
  '*':
    - default

root@salt-master:/srv/salt/pillar# cat default.sls
test1: true
testvar: blah

Have tried hardcoding 'base' instead of env and {{ saltenv }}.

It happens with ext_pillar_first: True or False.

Steps to Reproduce the behavior

Use the above config and just keep repeatedly running:

salt-call pillar.item testvar

or:

salt-call pillar.items

It also happens using testvar in jinja with: {{ pillar.get('testvar', '') }} - so it's not just the pillar execution module.

If it gets to the point where it disappears all together and won't come back - restarting the salt-master process seems to fix it for a few runs.

Expected behavior

The pillar value should be returned every time and not just disappear.

Screenshots

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:
        blah

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:
        blah

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:
        blah

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:
        blah

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:
        blah

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:
        blah

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:
        blah

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

root@salt-master:~# salt --versions-report
Salt Version:
          Salt: 3002.5

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10
       libgit2: 0.27.7
      M2Crypto: 0.31.0
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.19
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: 0.27.4
        Python: 3.7.3 (default, Jul 25 2020, 13:03:44)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.1.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.1

System Versions:
          dist: debian 10 buster
        locale: UTF-8
       machine: x86_64
       release: 4.19.0-12-amd64
        system: Linux
       version: Debian GNU/Linux 10 buster

Additional context
Add any other context about the problem here.

[ichilton]

Another interesting thing about this is, they can seem to work on one minion in the same moment it's not working on another - which is odd when pillars are parsed on the master?:

root@salt-master:~# salt test.mydomain.com pillar.item z_users
test.mydomain.com:
    ----------
    z_users:
        True

root@salt-master:~# salt test.mydomain.com pillar.item z_users
test.mydomain.com:
    ----------
    z_users:
        True

root@salt-master:~# salt test.mydomain.com pillar.item z_users
test.mydomain.com:
    ----------
    z_users:
        True

root@salt-master:~# salt test.mydomain.com pillar.item z_users
test.mydomain.com:
    ----------
    z_users:
        True

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:
        True

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:
        True

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:
        True

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:
        True

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:
        True

root@salt-master:~# salt-call pillar.item testvar
local:
    ----------
    testvar:

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:

root@salt-master:~# salt test.mydomain.com pillar.item z_users
test.mydomain.com:
    ----------
    z_users:
        True

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:

root@salt-master:~# salt test.mydomain.com pillar.item z_users
test.mydomain.com:
    ----------
    z_users:
        True

root@salt-master:~# salt-call pillar.item z_users
local:
    ----------
    z_users:

[ichilton]

Someone on Slack just suggested to use salt-run pillar.show_pillar <minion> to check the master render of the pillars.

Just ran that 20 times and the pillars were there every time - so sounds like maybe it's a minion bug??

[sagetherage]

@ichilton
The Salt version is vulnerable. We recommend upgrading as soon as possible to at least latest point release or to latest. You can find those here https://github.com/saltstack/salt/releases or the packages repo here: https://repo.saltproject.io/

This does sound like it is a bug, we will get it triaged, thank you!

[ichilton]

Great, thanks.

PS: Use Debian packages, so already on 3003 now.