crmsh-4.6: bootstrap in ssh-agent: false positive sshd error messages and waste 10+ seconds on scp operations

[zzhou1]

Overall two issues
#######################################

can be improved for bootstrapping user experience in the ssh-agent mode for non-root sudoer environment.

1. sshd on the init node reports red error messages in journalctl. There are two failure scp attempts from the joining node
2. such failures waste 10+ seconds for bootstrapping

REPRODUCER
#######################################

adm@15sp6-1:~> sudo -E crm cluster init --use-ssh-agent -y 

adm@15sp6-2:~> sudo -E crm -d cluster join --use-ssh-agent -y -c adm@15sp6-1

Dec 25 18:11:12 15sp6-2 crmsh.sh: DEBUG: su_subprocess_run: ['su', 'adm', '--login', '-s', '/bin/sh', '-c', "scp adm@15sp6-1:'/etc/csync2/csync2.cfg' /etc/csync2"], {'input': None, 'stdout': -1, 'stderr': -1}

Dec 25 18:11:19 15sp6-2 crmsh.sh: DEBUG: su_subprocess_run: ['su', 'adm', '--login', '-s', '/bin/sh', '-c', "scp adm@15sp6-1:'/etc/csync2/key_hagroup' /etc/csync2"], {'input': None, 'stdout': -1, 'stderr': -1}


Observation-1: `su --login` blocks "ssh-agent" functionality 

adm@15sp6-2:~> sudo -E su
15sp6-2:/home/adm # env|grep SOCK
SSH_AUTH_SOCK=/tmp/ssh-XXXX1rGs9V/agent.27604

15sp6-2:/home/adm # su --login adm -s /bin/sh -c "ssh adm@15sp6-1 hostname" 
Permission denied, please try again.
Permission denied, please try again.
Received disconnect from 192.168.156.101 port 22:2: Too many authentication failures
Disconnected from 192.168.156.101 port 22

15sp6-2:/home/adm # su adm -s /bin/sh -c "ssh adm@15sp6-1 hostname" 
15sp6-1


Observation-2: file permission is not possible for non-sudoer 

15sp6-2:/home/adm # su adm -s /bin/sh -c "scp adm@15sp6-1:'/etc/csync2/csync2.cfg' /etc/csync2"
scp: open local "/etc/csync2/csync2.cfg": Permission denied


Observation-3: file permission is not possible for non-sudoer 

adm@15sp6-1:~> ls -l /etc/csync2/key_hagroup
-rw------- 1 root root   65 Dec 30 16:10 key_hagroup

adm@15sp6-2:~> sudo -E su adm -s /bin/sh -c "scp adm@15sp6-1:'/etc/csync2/key_hagroup' /tmp/"
scp: remote open "/etc/csync2/key_hagroup": Permission denied

SYMPTOMS & LOG
#######################################

adm@15sp6-1:~> sudo -E crm cluster init --use-ssh-agent -y -N adm@15sp6-2
...
INFO: Adding node 15sp6-2 to cluster
INFO: Running command on 15sp6-2: crm cluster join -y  --use-ssh-agent -c adm@15sp6-1

Dec 25 13:47:47 15sp6-1 sshd[15736]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.156.102  user=adm
Dec 25 13:47:50 15sp6-1 sshd[15734]: error: PAM: Authentication failure for adm from 192.168.156.102
Dec 25 13:47:50 15sp6-1 sshd[15779]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.156.102  user=adm
Dec 25 13:47:51 15sp6-1 sshd[15734]: error: PAM: Authentication failure for adm from 192.168.156.102
Dec 25 13:47:51 15sp6-1 sshd[15780]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.156.102  user=adm
Dec 25 13:47:53 15sp6-1 sshd[15734]: error: PAM: Authentication failure for adm from 192.168.156.102
Dec 25 13:47:53 15sp6-1 sshd[15734]: Failed none for adm from 192.168.156.102 port 57496 ssh2
Dec 25 13:47:53 15sp6-1 sshd[15734]: Failed password for adm from 192.168.156.102 port 57496 ssh2
Dec 25 13:47:53 15sp6-1 sshd[15734]: Failed password for adm from 192.168.156.102 port 57496 ssh2
Dec 25 13:47:53 15sp6-1 sshd[15734]: error: maximum authentication attempts exceeded for adm from 192.168.156.102 port 57496 ssh2 [preauth]
Dec 25 13:47:53 15sp6-1 sshd[15734]: Disconnecting authenticating user adm 192.168.156.102 port 57496: Too many authentication failures [preauth]
Dec 25 13:47:53 15sp6-1 sshd[15781]: Accepted publickey for adm from 192.168.156.102 port 38834 ssh2: RSA SHA256:Ch61e2yqy5Tkw08UEYeNsi9YXBWoujjtYncF+XbuJ3w
Dec 25 13:47:53 15sp6-1 systemd-logind[771]: New session 25 of user adm.
Dec 25 13:47:53 15sp6-1 systemd[1]: Started Session 25 of User adm.
Dec 25 13:47:53 15sp6-1 sshd[15781]: pam_unix(sshd:session): session opened for user adm by (uid=0)
Dec 25 13:47:53 15sp6-1 sudo[15784]:      adm : PWD=/home/adm ; USER=root ; COMMAND=/bin/sh
Dec 25 13:47:53 15sp6-1 sudo[15784]: pam_unix(sudo:session): session opened for user root by (uid=1001)
Dec 25 13:47:53 15sp6-1 sudo[15784]: pam_unix(sudo:session): session closed for user root
Dec 25 13:47:53 15sp6-1 sshd[15783]: Received disconnect from 192.168.156.102 port 38834:11: disconnected by user
Dec 25 13:47:53 15sp6-1 sshd[15783]: Disconnected from user adm 192.168.156.102 port 38834
Dec 25 13:47:53 15sp6-1 sshd[15781]: pam_unix(sshd:session): session closed for user adm
Dec 25 13:47:53 15sp6-1 systemd[1]: session-25.scope: Deactivated successfully.
Dec 25 13:47:53 15sp6-1 systemd-logind[771]: Session 25 logged out. Waiting for processes to exit.
Dec 25 13:47:53 15sp6-1 systemd-logind[771]: Removed session 25.

Dec 25 13:47:54 15sp6-1 sshd[15812]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.156.102  user=adm
Dec 25 13:47:55 15sp6-1 sshd[15810]: error: PAM: Authentication failure for adm from 192.168.156.102
Dec 25 13:47:55 15sp6-1 sshd[15813]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.156.102  user=adm
Dec 25 13:47:57 15sp6-1 sshd[15810]: error: PAM: Authentication failure for adm from 192.168.156.102
Dec 25 13:47:57 15sp6-1 sshd[15814]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.156.102  user=adm
Dec 25 13:47:59 15sp6-1 sshd[15810]: error: PAM: Authentication failure for adm from 192.168.156.102
Dec 25 13:47:59 15sp6-1 sshd[15810]: Failed none for adm from 192.168.156.102 port 38850 ssh2
Dec 25 13:47:59 15sp6-1 sshd[15810]: Failed password for adm from 192.168.156.102 port 38850 ssh2
Dec 25 13:47:59 15sp6-1 sshd[15810]: Failed password for adm from 192.168.156.102 port 38850 ssh2
Dec 25 13:47:59 15sp6-1 sshd[15810]: error: maximum authentication attempts exceeded for adm from 192.168.156.102 port 38850 ssh2 [preauth]
Dec 25 13:47:59 15sp6-1 sshd[15810]: Disconnecting authenticating user adm 192.168.156.102 port 38850: Too many authentication failures [preauth]
Dec 25 13:47:59 15sp6-1 sshd[15815]: Accepted publickey for adm from 192.168.156.102 port 39370 ssh2: RSA SHA256:Ch61e2yqy5Tkw08UEYeNsi9YXBWoujjtYncF+XbuJ3w
Dec 25 13:47:59 15sp6-1 systemd-logind[771]: New session 26 of user adm.
Dec 25 13:47:59 15sp6-1 systemd[1]: Started Session 26 of User adm.
Dec 25 13:47:59 15sp6-1 sshd[15815]: pam_unix(sshd:session): session opened for user adm by (uid=0)
Dec 25 13:47:59 15sp6-1 sudo[15818]:      adm : PWD=/home/adm ; USER=root ; COMMAND=/bin/sh
Dec 25 13:47:59 15sp6-1 sudo[15818]: pam_unix(sudo:session): session opened for user root by (uid=1001)
Dec 25 13:47:59 15sp6-1 sudo[15818]: pam_unix(sudo:session): session closed for user root
Dec 25 13:47:59 15sp6-1 sshd[15817]: Received disconnect from 192.168.156.102 port 39370:11: disconnected by user
Dec 25 13:47:59 15sp6-1 sshd[15817]: Disconnected from user adm 192.168.156.102 port 39370
Dec 25 13:47:59 15sp6-1 sshd[15815]: pam_unix(sshd:session): session closed for user adm
Dec 25 13:47:59 15sp6-1 systemd[1]: session-26.scope: Deactivated successfully.
Dec 25 13:47:59 15sp6-1 systemd-logind[771]: Session 26 logged out. Waiting for processes to exit.
Dec 25 13:47:59 15sp6-1 systemd-logind[771]: Removed session 26.

[zzhou1]

One idea could be to replace "scp" related code with the prun() code which is used for "crm cluster copy"