junos_install_os with NSSU get exceptions

[henrchen]

When I use NSSU to install OS on QFX virtual chassis device, my ansible-playbook gets exceptions.

Upgrade from JUNOS 14.1X53.D35 to D40, I get:
TASK [upgrade-os : Install Junos OS package on leaf] **************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: jnpr.junos.exception.RpcError: RpcError(severity: error, bad_element: None, message: Not able to get object by path)
fatal: [ord12-c1u1-leaf-1.uspp1.oraclecloud.com]: FAILED! =>
{"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n File "/tmp/ansible_IC0uSg/ansible_module_junos_install_os.py", line 307, in \n main()\n File "/tmp/ansible_IC0uSg/ansible_module_junos_install_os.py", line 292, in main\n results = junos_install_os(module, dev)\n File "/tmp/ansible_IC0uSg/ansible_module_junos_install_os.py", line 221, in junos_install_os\n ok = sw.install(package, **sw_args)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/utils/sw.py", line 673, in install\n dev_timeout=timeout, **kwargs)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/utils/sw.py", line 193, in pkgaddNSSU\n package_name=remote_package, **kvargs)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/rpcmeta.py", line 336, in _exec_rpc\n return self._junos.execute(rpc, **dec_args)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/decorators.py", line 76, in wrapper\n return function(*args, **kwargs)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/decorators.py", line 23, in wrapper\n result = function(*args, **kwargs)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/device.py", line 722, in execute\n errs=ex)\njnpr.junos.exception.RpcError: RpcError(severity: error, bad_element: None, message: Not able to get object by path)\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}
to retry, use: --limit @/home/yuhechen/.ansible/retry/upgrade-os.retry
PLAY RECAP ********************************************************************************************************************************************
ord12-c1u1-leaf-1.uspp1.oraclecloud.com : ok=2 changed=0 unreachable=0 failed=1

From D40 to D35, I get:
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: jnpr.junos.exception.ConnectClosedError: ConnectClosedError(ord12-c1u1-leaf-3.uspp1.oraclecloud.com)
fatal: [ord12-c1u1-leaf-3.uspp1.oraclecloud.com]: FAILED! =>
{"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n File "/tmp/ansible_OZzX1f/ansible_module_junos_install_os.py", line 307, in \n main()\n File "/tmp/ansible_OZzX1f/ansible_module_junos_install_os.py", line 292, in main\n results = junos_install_os(module, dev)\n File "/tmp/ansible_OZzX1f/ansible_module_junos_install_os.py", line 221, in junos_install_os\n ok = sw.install(package, **sw_args)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/utils/sw.py", line 673, in install\n dev_timeout=timeout, **kwargs)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/utils/sw.py", line 193, in pkgaddNSSU\n package_name=remote_package, **kvargs)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/rpcmeta.py", line 336, in _exec_rpc\n return self._junos.execute(rpc, **dec_args)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/decorators.py", line 76, in wrapper\n return function(*args, **kwargs)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/decorators.py", line 23, in wrapper\n result = function(*args, **kwargs)\n File "/usr/local/lib/python2.7/site-packages/jnpr/junos/device.py", line 708, in execute\n raise EzErrors.ConnectClosedError(self)\njnpr.junos.exception.ConnectClosedError: ConnectClosedError(ord12-c1u1-leaf-3.uspp1.oraclecloud.com)\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}

netconf.txt

[vnitinv]

@henrchen share your playbook content.

[stacywsmith]

I suspect there are two issues here.

1. The NSSU upgrade from D35 to D40 fails due to a Junos bug. The NETCONF traceoptions log shows that the <rpc-reply> for the <rpc ><request-package-nonstop-upgrade><package-name>/var/tmp/jinstall-qfx-5-flex-14.1X53-D40.8-domestic-signed.tgz</package-name></request-package-nonstop-upgrade><rpc> RPC includes an <rpc-error> element:

May 25 17:32:51 [40742] Outgoing: <rpc-error>
May 25 17:32:51 [40742] Outgoing: <error-type>protocol</error-type>
May 25 17:32:51 [40742] Outgoing: <error-tag>operation-failed</error-tag>
May 25 17:32:51 [40742] Outgoing: <error-severity>error</error-severity>
May 25 17:32:51 [40742] Outgoing: <error-message>
May 25 17:32:51 [40742] Outgoing: Not able to get object by path
May 25 17:32:51 [40742] Outgoing: </error-message>
May 25 17:32:51 [40742] Outgoing: </rpc-error>

2. In the case of D40 to D35, it appears the NSSU happened successfully, however, I suspect the reboot option was specified, and the module doesn't cleanly handle the fact that the reboot option results in the device raising a ConnectClosedError exception.

If @henrchen can provide the playbook for this, we can confirm my suspicion about the second point and use this issue to track it to resolution.

[stacywsmith]

@henrchen

Please retest this with the latest juniper_junos_software module.

You can install the latest modules with:

sudo ansible-galaxy remove Juniper.junos
sudo ansible-galaxy install git+https://github.com/Juniper/ansible-junos-stdlib.git,,Juniper.junos

Your new playbook might look something like:

---
- name: 'Perform an NSSU upgrade'
  hosts: junos-all
  connection: local
  gather_facts: no
  roles:
    - Juniper.junos

  tasks:
    - name: 'Perform an NSSU on a QFX VC'
      juniper_junos_software:
        local_package: '/path/to/jinstall-qfx-5-flex-14.1X53-D40.8-domestic-signed.tgz'
        nssu: true
      register: response
    - name: 'Print result'
      debug:
        var: response

(That's only a guestimate at the new playbook since we never received the content of your original playbook where this problem was seen.)

If this problem is seen on the new module, please do open a new issue.