On salt-ssh, cmd.script does not copy scripts to remote server / cp.cache_* broken on salt-ssh

[guptasakshi01]

while executing script on target server using salt-ssh i am getting following error
cmd:- sudo salt-ssh minion cmd.script source=salt://test.sh
minion:

cache_error:
    True
pid:
    0
retcode:
    1
stderr:
stdout:

Setup

test.sh script is present at master on the /srv/salt/
We are executing command considering script will be executed remotely on target server(minion).

Versions Report

Salt Version:
Salt: 2017.7.1

Dependency Versions:
cffi: 1.11.5
cherrypy: 3.5.0
dateutil: 2.7.0
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.3.7
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.12 (default, Dec 4 2017, 14:50:18)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.4.0-112-generic
system: Linux
version: Ubuntu 16.04 xenial

[gtmanfred]

I am unable to replicate this behavior.

[root@salt ~]# cat /srv/salt/test.sh
#!/usr/bin/env bash
echo 'Hello, World!'
[root@salt ~]# salt-ssh -i \* cmd.script source=salt://test.sh
server:
    ----------
    pid:
        1744
    retcode:
        0
    stderr:
    stdout:
        Hello, World!
[root@salt ~]# salt-ssh --versions-report
Salt Version:
           Salt: 2017.7.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Apr 11 2018, 07:36:10)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 5.0.2
            ZMQ: Not Installed

System Versions:
           dist: centos 7.5.1804 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-862.3.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.5.1804 Core

It is working correctly for me. and copying the script over to the minion and then running the command.

[sumeetisp]

Minion based approach works but when i am using salt ssh there is no output.

minion:
    ----------
    cache_error:
        True
    pid:
        0
    retcode:
        1
    stderr:
    stdout:

Any way to debug when using salt ssh?

[sumeetisp]

The only different in output is cache error.

[gtmanfred]

I am unable to replicate this at all, can you provide me more information on your setup? any config changes you have made from the basic setup?

I am not able to replicate this, and it shouldn't be happening, so I don't know where the problem is for you.

It would also be worth while to share the -l debug output of the salt-ssh command.

Daniel

[liucy1983]

i got the same problem.
there is my debug info.

salt-ssh --no-host-keys --user=root --passwd=xxxx --roster=scan 10.0.10.200 cmd.script salt://5848f863e4b051f7b59fa4ce.sh -l debug
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: openstack
[DEBUG ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] salt.utils.network.ip_to_host(u'10.0.10.200') failed: [Errno 1] Unknown host
[DEBUG ] LazyLoaded scan.targets
[DEBUG ] Matched minions: {u'10.0.10.200': {u'host': u'10.0.10.200', u'port': 22}}
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG ] Updating roots fileserver cache
[DEBUG ] LazyLoaded local_cache.prep_jid
[DEBUG ] Adding minions for job 20180706104235103147: [u'10.0.10.200']
[DEBUG ] Could not LazyLoad cmd.script: 'cmd.script' is not available.
[DEBUG ] Performing shimmed, blocking command as follows:
cmd.script salt://5848f863e4b051f7b59fa4ce.sh
[DEBUG ] Executed SHIM command. Command logged to TRACE
[DEBUG ] Child Forked! PID: 30898 STDOUT_FD: 13 STDERR_FD: 15
[DEBUG ] VT: Salt-SSH SHIM Terminal Command executed. Logged to TRACE
[DEBUG ] RETCODE 10.0.10.200: 0
[DEBUG ] LazyLoaded nested.output
10.0.10.200:
----------
cache_error:
True
pid:
0
retcode:
1
stderr:
stdout:
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Sending event: tag = salt/job/20180706104235103147/ret/10.0.10.200; data = {u'fun_args': [u'salt://5848f863e4b051f7b59fa4ce.sh'], u'jid': u'20180706104235103147', u'return': {u'stderr': u'', u'pid': 0, u'retcode': 1, u'cache_error': True, u'stdout': u''}, u'retcode': 0, u'_stamp': '2018-07-06T02:42:38.132159', u'fun': u'cmd.script', u'id': u'10.0.10.200'}

Salt Version:
Salt: 2018.3.1

Dependency Versions:
cffi: 1.11.2
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Aug 4 2017, 00:39:18)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 14.7.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.5

System Versions:
dist: centos 7.4.1708 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-693.2.2.el7.x86_64
system: Linux
version: CentOS Linux 7.4.1708 Core

[The-Loeki]

Same problem 2019.2.0rc2

Related/probably duplicate: #11352

[gtmanfred]

@saltstack/team-triage can yall take a look at this?

[The-Loeki]

Single State cmd.script works fine btw (???)

bastion:~/.salt % sssh minion1 cmd.script salt://myscript.sh
minion1:
    ----------
    cache_error:
        True
    pid:
        0
    retcode:
        1
    stderr:
    stdout:
bastion:~/.salt % sssh minion1 state.single name=domyscript cmd.script source=salt://myscript.sh
minion1:
----------
          ID: domyscript 
    Function: cmd.script
      Result: True
     Comment: Command 'domyscript' run
     Started: 11:43:08.510654
    Duration: 7694.469 ms
     Changes:   
              ----------
              pid:
                  4653
              retcode:
                  0
              stderr:
                 <OUTPUT FOLLOWS>

[waynew]

I'll take a look at this today

[waynew]

Okay, yeah, I got the same behavior using salt-ssh mini -i cmd.script source=salt://test.sh -l debug. Using a couple of Docker images this my debug output:

[root@fa7d47a25c89 /]# salt-ssh mini -i cmd.script source=salt://test.sh -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] LazyLoaded flat.targets
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] compile template: /etc/salt/roster
[DEBUG   ] Jinja search path: ['/var/cache/salt/master/files/base']
[PROFILE ] Time (in seconds) to render '/etc/salt/roster' using 'jinja' renderer: 0.00340509414673
[DEBUG   ] Rendered data from file: /etc/salt/roster:
# Sample salt-ssh config file
#web1:
#  host: 192.168.42.1 # The IP addr or DNS hostname
#  user: fred         # Remote executions will be executed as user fred
#  passwd: foobarbaz  # The password to use for login, if omitted, keys are used
#  sudo: True         # Whether to sudo to root, not enabled by default
#web2:
#  host: 192.168.42.2
mini:
  host: 172.17.0.5
  user: root
  passwd: test

[DEBUG   ] Results of YAML rendering:
OrderedDict([('mini', OrderedDict([('host', '172.17.0.5'), ('user', 'root'), ('passwd', 'test')]))])
[PROFILE ] Time (in seconds) to render '/etc/salt/roster' using 'yaml' renderer: 0.00223207473755
[DEBUG   ] Matched minions: {'mini': OrderedDict([('host', '172.17.0.5'), ('user', 'root'), ('passwd', 'test')])}
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] LazyLoaded local_cache.prep_jid
[DEBUG   ] Adding minions for job 20190207204022735139: ['mini']
[DEBUG   ] Could not LazyLoad cmd.script
[DEBUG   ] Performing shimmed, blocking command as follows:
cmd.script source=salt://test.sh
[DEBUG   ] Executed SHIM command. Command logged to TRACE
[DEBUG   ] Child Forked! PID: 159  STDOUT_FD: 10  STDERR_FD: 12
[DEBUG   ] VT: Salt-SSH SHIM Terminal Command executed. Logged to TRACE
[DEBUG   ] RETCODE 172.17.0.5: 0
[DEBUG   ] LazyLoaded nested.output
mini:
    ----------
    cache_error:
        True
    pid:
        0
    retcode:
        1
    stderr:
    stdout:

I also tried salt-call cmd.script salt://test.sh using the minion'd source, which also failed. Working on tracking down the ultimate cause.

[waynew]

Interesting - I just added /srv/salt/test.sh to the minion and calling it worked... but with the contents of that script and not the one stored on the master. I suspect the cause is a lack of /srv/salt on the minion.

[waynew]

Huh. Nope - I was wrong, it looks like it's not passing the script to the minion...

[waynew]

Okay, I'm not quite sure exactly why, but I've managed to isolate the behavior. Using this Dockerfile for the minion:

FROM ubuntu:18.04

RUN apt-get update && apt-get install -y openssh-server
RUN mkdir /var/run/sshd
RUN echo 'root:test' | chpasswd
RUN sed -i 's/.*PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config

# SSH login fix. Otherwise user is kicked off after login
RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd

ENV NOTVISIBLE "in users profile"
RUN echo "export VISIBLE=now" >> /etc/profile

RUN apt-get install -y python-minimal

EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

And this Dockerfile for the master

FROM ubuntu:18.04 as testing
MAINTAINER Wayne Werner <wwerner@saltstack.com>

RUN echo "tzdata tzdata/Areas select Etc" >/tmp/fnord
RUN echo "tzdata tzdata/Zones/Etc select UTC" >>/tmp/fnord
RUN debconf-set-selections /tmp/fnord
RUN apt-get update && apt-get install -y tzdata
RUN echo "deb http://archive.ubuntu.com/ubuntu bionic main universe multiverse restricted" > /etc/apt/sources.list && \
echo "deb http://archive.ubuntu.com/ubuntu bionic-security main universe multiverse restricted" >> /etc/apt/sources.list && \
apt-get update && \
apt-get install -y git && \
apt-get upgrade -y -o DPkg::Options::=--force-confold


## Add the Salt PPA
RUN echo "deb http://repo.saltstack.com/apt/ubuntu/18.04/amd64/latest bionic main" > /etc/apt/sources.list.d/saltstack.list
RUN apt-get install -y -o DPkg::Options::=--force-confold  software-properties-common curl && \
curl https://repo.saltstack.com/apt/ubuntu/18.04/amd64/latest/SALTSTACK-GPG-KEY.pub | apt-key add - && \
apt-get update

## Install Salt Dependencies
RUN apt-get install -y -o DPkg::Options::=--force-confold \
python \
python-yaml \
python-m2crypto \
python-crypto \
msgpack-python \
python-zmq \
python-jinja2 \
python-requests \
salt-ssh

RUN echo "auto_accept: True" >> /etc/salt/master

# TODO will I need to futz with the master config?
#ADD default_minion /etc/salt/minion

FROM testing
## Install dev tools
RUN apt-get install -y ed vim-tiny python-pip
RUN pip install pudb

WORKDIR /srv/salt

Run the minion like so:

docker run -d -P --name test_sshd salt-sshd-ubuntu-minion:18.04   # or whatever you've tagged it

And the master:

docker run --rm -it --link test_sshd salt-ssh:ubuntu1804  # or whatever you've tagged it

Then create a script in /srv/salt/ like so:

#!/bin/sh
echo "This might work!"

And try to run it:

salt-ssh --user=root --passwd=test --roster=scan $TEST_SSHD_PORT_22_TCP_ADDR -i cmd.script salt://test.sh

This fails with a cache error. Sad. All is not lost, however!

salt-ssh --user=root --passwd=test --roster=scan $TEST_SSHD_PORT_22_TCP_ADDR -i cp.get_file salt://test.sh /srv/salt/ makedirs=True

Now re-run the cmd.script and it works!

From what I've been able to tell the problem is that the minion never asks the master for the file - I've got some tests I want to run about caching, but ultimately you can at least work around it by copying the file directly to /srv/salt/ on the minion (which seems kind of weird, considering that it puts the salt sources under /var/tmp from what I'm able to see. Especially odd if you do cp.cache_file - which only works after you've copied the file to /srv/salt/ on the minion.)