External auth with pam does not work when the salt-master is NOT running as root

[anitakrueger]

My salt-master process is running as user salt, like configured in /etc/salt/master:

# The user to run the salt-master as. Salt will update all permissions to
# allow the specified user to run the master. If the modified files cause
# conflicts set verify_env to False.
user: salt

The salt system user was created as follows:

groupadd salt
useradd -m -g salt salt

I also have external_auth enabled for other users (originally I was trying to access the halite gui) like below:

external_auth:
  pam:
    vagrant:
      - .*
    anita:
      - .*
      - '@runner'
      - '@wheel'

With that configuration, pam authentication for the users vagrant and anita always fail:

root@salt:/etc/salt# salt -a pam '*' test.ping
username: vagrant
password: 
Failed to authenticate, is this user permitted to execute commands?

As soon as I run salt-master as root, everything starts working.

Am I missing some configuration? I tried adding my local users to the "salt" group, but no avail.

I've reproduced this with a mint salt master and minions from https://github.com/elasticdog/salt-sandbox running Salt 0.17.0.1.

[basepi]

Just to verify: this didn't work on a previous version did it? Wondering if it's just an outstanding bug or a regression.

In any case, thanks for finding this, hopefully it's an easy fix.

[anitakrueger]

Wanted to test this on 0.16.4 real quick with the salt-sandbox, only to find out that the 0.16.4 precise packages have been pulled from the Ubuntu repository :( need a little more time to test on an earlier version.
I only found it because I wanted to use halite and hence never tested before 0.17.

[basepi]

Ah, gotcha.

Easy way to test on a previous version is to just do a manual install from git:

git clone git://github.com/saltstack/salt
cd salt
git checkout v0.16.4
sudo python setup.py install --force

[anitakrueger]

Looks like this never worked, not even in 0.16.4.
Also when testing with 0.17.2 the behavior changed to not reporting an error back, but the salt call simply runs into a timeout:

root@salt:~# salt -a pam '*' test.ping
username: vagrant
password: 
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
SaltReqTimeoutError: Waited 60 seconds
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 117, in salt_main
    client.run()
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 133, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/lib/pymodules/python2.7/salt/client/__init__.py", line 471, in cmd_cli
    **kwargs)
  File "/usr/lib/pymodules/python2.7/salt/client/__init__.py", line 228, in run_job
    **kwargs)
  File "/usr/lib/pymodules/python2.7/salt/client/__init__.py", line 1216, in pub
    payload = sreq.send('clear', payload_kwargs)
  File "/usr/lib/pymodules/python2.7/salt/payload.py", line 193, in send
    timeout * tried
SaltReqTimeoutError: Waited 60 seconds
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 117, in salt_main
    client.run()
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 133, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/lib/pymodules/python2.7/salt/client/__init__.py", line 471, in cmd_cli
    **kwargs)
  File "/usr/lib/pymodules/python2.7/salt/client/__init__.py", line 228, in run_job
    **kwargs)
  File "/usr/lib/pymodules/python2.7/salt/client/__init__.py", line 1216, in pub
    payload = sreq.send('clear', payload_kwargs)
  File "/usr/lib/pymodules/python2.7/salt/payload.py", line 193, in send
    timeout * tried
salt.exceptions.SaltReqTimeoutError: Waited 60 seconds

[basepi]

Thanks for the update.

[jcockhren]

I've verified this behavior.

[jcockhren]

More on this. The stem of the problem seems to lie within the the polling part of the client logic. In order words, it's solved by #8581. It's been merged as of a couple days ago. I'll check the master branch to see if this issues disappears. Stay tuned.

[cachedout]

@jcockhren Great work on hunting this down. I'll go ahead and set this to 'Fixed Pending Verification'. If you wouldn't mind closing this after you've verified that it's fixed in the current develop branch, that would be great. Thank you!

[wari]

Just FYI about this bug. PAM will work for you if you run salt as the user salt and you authenticate with the user salt.

My test cases for my code does this - runs as vagrant, and pam auth as vagrant. I've not tested this with other users as I thought normal that this might not work, which is why I've never tried this other setup.

[alexandredacosta]

This command fixes these issues:

setfacl -m u:salt:r /etc/shadow

For "salt" substitute the salt master process run user.

The problem is that pam auth won't work for non-root users trying to authenticate another user. In this case our non-root user is the user under which the salt master runs. Non-root users can't read "/etc/shadow", which would be required for pam authentication of other users.

Solution: Add an access control rule enabling the salt master run user to read "/etc/shadow".

[basepi]

So does this mean that this is not a bug in salt but rather a "feature" in PAM (or, more accurately, in the permissions for /etc/shadow)?

[alexandredacosta]

To me it means not to use pam auth unless you're running your salt master as root. :-) I say this because access control rules on /etc/shadow will get dropped every time someone uses the "passwd" command, likely because the command regenerates the entire file on a new inode. This makes an access control rule here rather fragile. You might also be able to change file permissions permanently, but as you hint at, this compromises built-in security.

For these cases it's probably best to use an alternate authentication method. There's an included LDAP auth method, and it's easy to cook your own by following the existing salt auth modules.

[basepi]

I'm going to close this one, as there's not much we can do from a salt standpoint. As @alexandredacosta pointed out, you just need to give the salt-master user access to the shadow file so that PAM can auth.