Bug: provision VM + ansible

[LecrisUT]

From Fedora CI: https://matrix.to/#/!cfWVeczGVJbiKSlrwi:fedoraproject.org/$pWT2oRW18ucYgnv6-Yyqfbx-9M6UJdLtF1-3lLM1T8A?via=fedoraproject.org&via=fedora.im&via=matrix.org.

It seems when provision --how virtual, and there is a prepare --how ansible step, then the ansible-playbook command is run on the host, and not within the virtual machine. That would be fine, except it is using -i root@localhost

        Command 'ansible-playbook --ssh-common-args '-oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p10023 -i /var/tmp/tmt/run-022/plans/behave/provision/default-0/id_ecdsa -S/run/user/17122/tmt/tmppk62lju8' -i root@127.0.0.1, /home/praiskup/rh/projects/mock/mock/worktree/praiskup/TF/testing-farm/../mock/integration-tests/setup-playbook/play-tf.yml' returned 4.

        stdout (7/7 lines)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        PLAY [Prepare the testing machine for running Mock tests] **********************

        TASK [Gathering Facts] *********************************************************
        fatal: [root@127.0.0.1]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via scp: scp: SSH protocol v.1 is no longer supported\r\n", "unreachable": true}

[happz]

I don't think the use of localhost is the problem here. The VM spawned by virtual might be accessible via localhost, with the right port, and that's what ansible-playbook seems to be doing with -i & -p10023 in --ssh-common-args.

The error message also suggests a different cause: I'm seeing scp there, and scp definitely does not accept -p$PORT, so maybe the trouble is here - -p works for ssh but not for scp, and we should do --ssh-extra-args "-p10023" --scp-extra-args "-P10023" instead of including a port in common args.

[LecrisUT]

How about making a proper inventory file with ansible_port. Iiuc, that one will be used for all ssh-like commands

[happz]

How about making a proper inventory file with ansible_port. Iiuc, that one will be used for all ssh-like commands

Also possible, sure. It might be possible to create it once for each guest and re-use it, it's probably read-only from that point anyway.

[praiskup]

The -P 10023 doesn't help:

$ ansible-playbook --scp-extra-args '-P 10023' ...
...
fatal: [127.0.0.1]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: SSH protocol v.1 is no longer supported", "unreachable": true}

[LecrisUT]

How about making a proper inventory file with ansible_port. Iiuc, that one will be used for all ssh-like commands

Also possible, sure. It might be possible to create it once for each guest and re-use it, it's probably read-only from that point anyway.

Probably it is worth putting in /groups/all.yaml unless it is manually defined. Iirc there is a system-wide ansible inventory, where we could use /groups/tmt.yaml?

[happz]

SSH protocol v.1 is no longer supported

Can you run it with more verbose Ansible, -vvvv or something on that level? The SSH protocol mentioned in the error message is puzzling, I assumed it comes from the -p vs -P issue, but maybe there's something more. Verbose logging could show the full SSH commands.

[praiskup]

$ ansible-playbook -vvvv  --scp-extra-args '-P 10023'  --ssh-common-args '-oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p10023  -i /var/tmp/tmt/run-048/testing-farm/plans/behave/provision/default-0/id_ecdsa -S/run/user/17122/tmt/tmplcumhybv ' -i 127.0.0.1, /home/praiskup/rh/projects/mock/mock/worktree/praiskup/TF/testing-farm/../mock/integration-tests/setup-playbook/play-tf.yml
ansible-playbook [core 2.16.2]
  config file = /home/praiskup/.ansible.cfg
  configured module search path = ['/home/praiskup/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.12/site-packages/ansible
  ansible collection location = /home/praiskup/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
Using /home/praiskup/.ansible.cfg as config file
setting up inventory plugins
Loading collection ansible.builtin from 
Set default localhost to 127.0.0.1
Parsed 127.0.0.1, inventory source with host_list plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3.12/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: play-tf.yml **********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Positional arguments: /home/praiskup/rh/projects/mock/mock/worktree/praiskup/TF/testing-farm/../mock/integration-tests/setup-playbook/play-tf.yml
verbosity: 4
connection: ssh
ssh_common_args: -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p10023  -i /var/tmp/tmt/run-048/testing-farm/plans/behave/provision/default-0/id_ecdsa -S/run/user/17122/tmt/tmplcumhybv 
scp_extra_args: -P 10023
become_method: sudo
tags: ('all',)
inventory: ('127.0.0.1,',)
forks: 5
1 plays in /home/praiskup/rh/projects/mock/mock/worktree/praiskup/TF/testing-farm/../mock/integration-tests/setup-playbook/play-tf.yml

PLAY [Prepare the testing machine for running Mock tests] **********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
task path: /home/praiskup/rh/projects/mock/mock/worktree/praiskup/TF/mock/integration-tests/setup-playbook/play-tf.yml:2
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: root
<127.0.0.1> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p10023 -i /var/tmp/tmt/run-048/testing-farm/plans/behave/provision/default-0/id_ecdsa -S/run/user/17122/tmt/tmplcumhybv -o 'ControlPath="/home/praiskup/.ansible/cp/842df7d42c"' 127.0.0.1 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<127.0.0.1> (0, b'/root\n', b'OpenSSH_9.3p1, OpenSSL 3.1.1 30 May 2023\r\ndebug1: Reading configuration data /home/praiskup/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: /home/praiskup/.ssh/config line 58: Applying options for *\r\ndebug2: checking match for \'User copr-libvirt-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 77: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug2: checking match for \'User copr-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 79: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for \'final all\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched \'final\'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 127.0.0.1 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /home/praiskup/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: /home/praiskup/.ssh/config line 58: Applying options for *\r\ndebug2: add_identity_file: ignoring duplicate key ~/.ssh/id_rsa_long\r\ndebug2: add_identity_file: ignoring duplicate key ~/.ssh/id_rsa\r\ndebug2: checking match for \'User copr-libvirt-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 77: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug2: checking match for \'User copr-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 79: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for \'final all\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched \'final\'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 785934\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: root
<127.0.0.1> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p10023 -i /var/tmp/tmt/run-048/testing-farm/plans/behave/provision/default-0/id_ecdsa -S/run/user/17122/tmt/tmplcumhybv -o 'ControlPath="/home/praiskup/.ansible/cp/842df7d42c"' 127.0.0.1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1706014152.5577002-787011-31490231856825 `" && echo ansible-tmp-1706014152.5577002-787011-31490231856825="` echo /root/.ansible/tmp/ansible-tmp-1706014152.5577002-787011-31490231856825 `" ) && sleep 0'"'"''
<127.0.0.1> (0, b'ansible-tmp-1706014152.5577002-787011-31490231856825=/root/.ansible/tmp/ansible-tmp-1706014152.5577002-787011-31490231856825\n', b'OpenSSH_9.3p1, OpenSSL 3.1.1 30 May 2023\r\ndebug1: Reading configuration data /home/praiskup/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: /home/praiskup/.ssh/config line 58: Applying options for *\r\ndebug2: checking match for \'User copr-libvirt-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 77: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug2: checking match for \'User copr-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 79: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for \'final all\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched \'final\'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 127.0.0.1 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /home/praiskup/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: /home/praiskup/.ssh/config line 58: Applying options for *\r\ndebug2: add_identity_file: ignoring duplicate key ~/.ssh/id_rsa_long\r\ndebug2: add_identity_file: ignoring duplicate key ~/.ssh/id_rsa\r\ndebug2: checking match for \'User copr-libvirt-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 77: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug2: checking match for \'User copr-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 79: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for \'final all\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched \'final\'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 785934\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<127.0.0.1> Attempting python interpreter discovery
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: root
<127.0.0.1> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p10023 -i /var/tmp/tmt/run-048/testing-farm/plans/behave/provision/default-0/id_ecdsa -S/run/user/17122/tmt/tmplcumhybv -o 'ControlPath="/home/praiskup/.ansible/cp/842df7d42c"' 127.0.0.1 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'python3.12'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.11'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.10'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<127.0.0.1> (0, b'PLATFORM\nLinux\nFOUND\n/usr/bin/python3.12\n/usr/bin/python3\n/usr/bin/python\n/usr/bin/python\nENDFOUND\n', b'OpenSSH_9.3p1, OpenSSL 3.1.1 30 May 2023\r\ndebug1: Reading configuration data /home/praiskup/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: /home/praiskup/.ssh/config line 58: Applying options for *\r\ndebug2: checking match for \'User copr-libvirt-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 77: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug2: checking match for \'User copr-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 79: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for \'final all\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched \'final\'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 127.0.0.1 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /home/praiskup/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: /home/praiskup/.ssh/config line 58: Applying options for *\r\ndebug2: add_identity_file: ignoring duplicate key ~/.ssh/id_rsa_long\r\ndebug2: add_identity_file: ignoring duplicate key ~/.ssh/id_rsa\r\ndebug2: checking match for \'User copr-libvirt-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 77: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug2: checking match for \'User copr-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 79: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for \'final all\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched \'final\'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 785934\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: root
<127.0.0.1> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p10023 -i /var/tmp/tmt/run-048/testing-farm/plans/behave/provision/default-0/id_ecdsa -S/run/user/17122/tmt/tmplcumhybv -o 'ControlPath="/home/praiskup/.ansible/cp/842df7d42c"' 127.0.0.1 '/bin/sh -c '"'"'/usr/bin/python3.12 && sleep 0'"'"''
<127.0.0.1> (0, b'{"platform_dist_result": [], "osrelease_content": "NAME=\\"Fedora Linux\\"\\nVERSION=\\"39 (Cloud Edition)\\"\\nID=fedora\\nVERSION_ID=39\\nVERSION_CODENAME=\\"\\"\\nPLATFORM_ID=\\"platform:f39\\"\\nPRETTY_NAME=\\"Fedora Linux 39 (Cloud Edition)\\"\\nANSI_COLOR=\\"0;38;2;60;110;180\\"\\nLOGO=fedora-logo-icon\\nCPE_NAME=\\"cpe:/o:fedoraproject:fedora:39\\"\\nHOME_URL=\\"https://fedoraproject.org/\\"\\nDOCUMENTATION_URL=\\"https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/\\"\\nSUPPORT_URL=\\"https://ask.fedoraproject.org/\\"\\nBUG_REPORT_URL=\\"https://bugzilla.redhat.com/\\"\\nREDHAT_BUGZILLA_PRODUCT=\\"Fedora\\"\\nREDHAT_BUGZILLA_PRODUCT_VERSION=39\\nREDHAT_SUPPORT_PRODUCT=\\"Fedora\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=39\\nSUPPORT_END=2024-05-14\\nVARIANT=\\"Cloud Edition\\"\\nVARIANT_ID=cloud\\n"}\n', b'OpenSSH_9.3p1, OpenSSL 3.1.1 30 May 2023\r\ndebug1: Reading configuration data /home/praiskup/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: /home/praiskup/.ssh/config line 58: Applying options for *\r\ndebug2: checking match for \'User copr-libvirt-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 77: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug2: checking match for \'User copr-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 79: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for \'final all\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched \'final\'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 127.0.0.1 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /home/praiskup/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: /home/praiskup/.ssh/config line 58: Applying options for *\r\ndebug2: add_identity_file: ignoring duplicate key ~/.ssh/id_rsa_long\r\ndebug2: add_identity_file: ignoring duplicate key ~/.ssh/id_rsa\r\ndebug2: checking match for \'User copr-libvirt-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 77: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug2: checking match for \'User copr-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 79: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for \'final all\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched \'final\'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 785934\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python3.12/site-packages/ansible/modules/setup.py
<127.0.0.1> PUT /home/praiskup/.ansible/tmp/ansible-local-7870084ofq7akq/tmp4jiv8nkj TO /root/.ansible/tmp/ansible-tmp-1706014152.5577002-787011-31490231856825/AnsiballZ_setup.py
<127.0.0.1> SSH: EXEC scp -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p10023 -i /var/tmp/tmt/run-048/testing-farm/plans/behave/provision/default-0/id_ecdsa -S/run/user/17122/tmt/tmplcumhybv -P 10023 -o 'ControlPath="/home/praiskup/.ansible/cp/842df7d42c"' /home/praiskup/.ansible/tmp/ansible-local-7870084ofq7akq/tmp4jiv8nkj '[127.0.0.1]:/root/.ansible/tmp/ansible-tmp-1706014152.5577002-787011-31490231856825/AnsiballZ_setup.py'
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: root
<127.0.0.1> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p10023 -i /var/tmp/tmt/run-048/testing-farm/plans/behave/provision/default-0/id_ecdsa -S/run/user/17122/tmt/tmplcumhybv -o 'ControlPath="/home/praiskup/.ansible/cp/842df7d42c"' 127.0.0.1 '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1706014152.5577002-787011-31490231856825/ > /dev/null 2>&1 && sleep 0'"'"''
<127.0.0.1> (0, b'', b'OpenSSH_9.3p1, OpenSSL 3.1.1 30 May 2023\r\ndebug1: Reading configuration data /home/praiskup/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: /home/praiskup/.ssh/config line 58: Applying options for *\r\ndebug2: checking match for \'User copr-libvirt-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 77: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug2: checking match for \'User copr-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 79: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for \'final all\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched \'final\'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 127.0.0.1 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /home/praiskup/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: /home/praiskup/.ssh/config line 58: Applying options for *\r\ndebug2: add_identity_file: ignoring duplicate key ~/.ssh/id_rsa_long\r\ndebug2: add_identity_file: ignoring duplicate key ~/.ssh/id_rsa\r\ndebug2: checking match for \'User copr-libvirt-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 77: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug2: checking match for \'User copr-controller\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /home/praiskup/.ssh/config line 79: not matched \'User "root"\' \r\ndebug2: match not found\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for \'final all\' host 127.0.0.1 originally 127.0.0.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched \'final\'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 785934\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
fatal: [127.0.0.1]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via scp: scp: SSH protocol v.1 is no longer supported\r\n",
    "unreachable": true
}

PLAY RECAP *********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
127.0.0.1                  : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   

[happz]

How about making a proper inventory file with ansible_port. Iiuc, that one will be used for all ssh-like commands

Also possible, sure. It might be possible to create it once for each guest and re-use it, it's probably read-only from that point anyway.

Probably it is worth putting in /groups/all.yaml unless it is manually defined. Iirc there is a system-wide ansible inventory, where we could use /groups/tmt.yaml?

I for one would stay away from anything system-wide. tmt usually talks to one guest when running ansible-playbook, and no other tmt run talks to those guests either, therefore having inventory files in $WORKDIR/$PLAN/provision/ or somewhere similar, visible to just that tmt run process, seems like the safest way to me. tmt.utils.safe_filename() can provide a nice filename, Guest class can gain a property returning the path constructed by safe_filename('tmt-inventory.yaml', self.step, self), populate it when first accessed, etc.

[LecrisUT]

@praiskup can you try without setting user root and using become: true instead?

@happz I was thinking about the comment of reusing the inventory. But I guess you meant reusing within the plan, in which case it makes more sense like that

[happz]

@happz I was thinking about the comment of reusing the inventory. But I guess you meant reusing within the plan, in which case it makes more sense like that

Ah, yes, definitely in the plan. Re-use by all invocations of ansible-playbook, i.e. not generating the inventory file for each invocation.

[happz]

@praiskup hm, no idea, maybe the server log would help. And maybe SSH is not kidding when it complains about an invalid SSH version...

[praiskup]

Note that we enforce -S/run/user/17122/tmt/tmplcumhybv , but ansible is using -o 'ControlPath="/home/praiskup/.ansible/cp/842df7d42c"', is this a problem?

The ansible-playbook command starts working if when I drop the -S option.

[praiskup]

And, now I noticed I had ~/.ansible.cfg :

[ssh_connection]
transfer_method = scp

Commenting that out also helped. I don't remember the reasons for this...

[happz]

Note that we enforce -S/run/user/17122/tmt/tmplcumhybv , but ansible is using -o 'ControlPath="/home/praiskup/.ansible/cp/842df7d42c"', is this a problem?

The ansible-playbook command starts working if when I drop the -S option.

Yes, that does not seem right, it's definitely sending mixed signals to Ansible.

And, now I noticed I had ~/.ansible.cfg :

[ssh_connection]
transfer_method = scp

Commenting that out also helped. I don't remember the reasons for this...

Would ansible-playbook work without removing -S... after you commented out the transfer_method?

[praiskup]

Yes, it works without -S:

$ /usr/bin/ansible-playbook --ssh-common-args '-oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p10023 -i /var/tmp/tmt/run-052/testing-farm/plans/behave/provision/default-0/id_ecdsa' -i root@127.0.0.1, /home/praiskup/rh/projects/mock/mock/worktree/praiskup/TF/mock/integration-tests/setup-playbook/play-tf.yml
PLAY [Prepare the testing machine for running Mock tests] **********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
[WARNING]: sftp transfer mechanism failed on [root@127.0.0.1]. Use ANSIBLE_DEBUG=1 to see detailed information
[WARNING]: scp transfer mechanism failed on [root@127.0.0.1]. Use ANSIBLE_DEBUG=1 to see detailed information
ok: [root@127.0.0.1]

TASK [include_tasks] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
included: /home/praiskup/rh/projects/mock/mock/worktree/praiskup/TF/mock/integration-tests/setup-playbook/tasks/main.yml for root@127.0.0.1

TASK [install mock and other needed packages] **********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
^C [ERROR]: User interrupted execution

These warnings don't seem to be candidates for ignoring :-/ though

[WARNING]: sftp transfer mechanism failed on [root@127.0.0.1]. Use ANSIBLE_DEBUG=1 to see detailed information
[WARNING]: scp transfer mechanism failed on [root@127.0.0.1]. Use ANSIBLE_DEBUG=1 to see detailed information

[praiskup]

These warnings don't seem to be candidates for ignoring :-/ though

These warnings appear no matter if I use -S<path> or not.

[praiskup]

What helps is to move -p10023 --ssh-common-opts to --ssh-extra-opts - warnings disappeared.