Junos_Install_Os Failed to upgrade the JUNOS Host Software

[nzkller]

Running junos_install_os will successfully install the Junos Base OS Software Suite but its does not upgrade the Juniper Host Software in Juniper's QFX Series switches.

Juniper's Advanced JTAC Agent specified that this might create issues in the performance of the switch and it could be responsible for some of the problem we were having.

Example:

Hostname: tesw1
Model: qfx5100-48t-6q
Junos: 14.1X53-D30.3
JUNOS Base OS Software Suite [14.1X53-D30.3]
JUNOS Base OS boot [14.1X53-D30.3]
JUNOS Crypto Software Suite [14.1X53-D30.3]
JUNOS Online Documentation [14.1X53-D30.3]
JUNOS Kernel Software Suite [14.1X53-D30.3]
JUNOS Packet Forwarding Engine Support (qfx-ex-x86-32) [14.1X53-D30.3]
JUNOS Routing Software Suite [14.1X53-D30.3]
JUNOS SDN Software Suite [14.1X53-D30.3]
JUNOS Enterprise Software Suite [14.1X53-D30.3]
JUNOS Web Management Platform Package [14.1X53-D30.3]
JUNOS py-base-i386 [14.1X53-D30.3]
JUNOS Host Software [13.2X51-D25.2]

Normally, if I'd perform the upgrade by hand I will use the command

request system software add reboot no-validate force-host /var/tmp/jinstall.......

But I need this to be perform by the Ansible playbook that will provision new racks

Is there anyway to make the junos_install_os upgrade the host software as well?

Thanks in advance and best regards,

Nzkller

[stacywsmith]

Hi @nzkller

Sorry, I'm not familiar enough with the specific nuances of the QFX platform, so I have a couple of questions for you.

When upgrading these devices manually from the CLI, is there any difference in the actual package file that you supply to the request system software add command? If not, is the only difference the addition of the force-host argument to the command?

[nzkller]

Hi @stacywsmith,

Yes you are absolutely right the only difference is the force-host argument that you add at the end.

Thank you for your reply and best regards,

Nzkller

[stacywsmith]

Thanks @nzkller

It's not currently possible to pass the force-host argument via the junos_install_os module, but I will work on adding that capability.

Would you mind emailing me your JTAC case number to stacys at juniper dot net?

[nzkller]

Yes, I will let me get back to the office and I'll forward that to you.

Thanks,

Nzkller

[stacywsmith]

@nzkller Can you can try out this proposed fix.

sudo ansible-galaxy remove Juniper.junos
sudo ansible-galaxy install  git+https://github.com/stacywsmith/ansible-junos-stdlib.git@force_host_arg,,Juniper.junos -vvv

You'll need to add the force_host: True argument to the junos_install_os task in your playbook.

[nzkller]

Hi @stacywsmith

I sent you an email with the ATAC cases as you requested.

I will try your proposed fix today. I update to Ansible 2.3 and I'm having some issues running my playbooks including paramiko giving me this error

localhost:ERROR:login:No existing session

Even if I run the netconify by itself

TTY:connecting to TTY:62.91.101.241:1234:XXXX:XXXXX ...
ERROR:login:No existing session

Traceback (most recent call last):
  File "/usr/local/bin/netconify", line 7, in <module>
    results = nc.run()
  File "/usr/local/lib/python2.7/dist-packages/netconify/cmdo.py", line 268, in run
    self._hook_exception('login', err)
  File "/usr/local/lib/python2.7/dist-packages/netconify/cmdo.py", line 266, in run
    self._tty_login()
  File "/usr/local/lib/python2.7/dist-packages/netconify/cmdo.py", line 352, in _tty_login
    self._tty.login(notify=notify)
  File "/usr/local/lib/python2.7/dist-packages/netconify/tty.py", line 104, in login
    self._tty_open()
  File "/usr/local/lib/python2.7/dist-packages/netconify/tty_ssh.py", line 43, in _tty_open
    username=self.s_user, password=self.s_passwd, timeout=self.timeout, allow_agent=False, look_for_keys=False)
  File "/usr/local/lib/python2.7/dist-packages/paramiko/client.py", line 342, in connect
    server_key = t.get_remote_server_key()
  File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 675, in get_remote_server_key
    raise SSHException('No existing session')
paramiko.ssh_exception.SSHException: No existing session

So I will try to solve this and try your fix.

Best regards,

Nzkller

[nzkller]

Ok its fixed with Paramiko version 2.0.5 now im going to try your fix. Ill keep you posted.

Best regards,

Nzkller

[nzkller]

HI @stacywsmith

Im getting this error while installing it

root@finn:/etc/ansible# ansible-galaxy install  git+https://github.com/stacywsmith/ansible-junos-stdlib.git@force_host_arg,,Juniper.junos -vvv
Using /etc/ansible/ansible.cfg as config file
Opened /root/.ansible_galaxy
Installing role Juniper.junos
 [WARNING]: - Juniper.junos was NOT installed successfully: - command git clone https://github.com/stacywsmith/ansible-junos-stdlib.git@force_host_arg
Juniper.junos failed in directory /tmp/user/0/tmppfmZQY (rc=128)

ERROR! - you can use --ignore-errors to skip failed roles and finish processing the list.

Any ideas?

Best regards,

Nzkller

[stacywsmith]

@nzkller Sorry. I messed up the syntax for installing from a specific branch. Please try:

sudo ansible-galaxy remove Juniper.junos
sudo ansible-galaxy install  git+https://github.com/stacywsmith/ansible-junos-stdlib.git,force_host_arg,Juniper.junos

[nzkller]

@stacywsmith

No problem I figured it out so I ran it and it successfully upgraded the Junos Host Software

These are the logs for the install

2017-04-26 15:28:38,919:10.238.0.181:Starting the software upgrade process: /data/repository/Juniper/jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz
2017-04-26 15:28:38,919:10.238.0.181:computing local checksum on: /data/repository/Juniper/jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz
2017-04-26 15:28:40,377:10.238.0.181:cleaning filesystem ...
2017-04-26 15:28:40,378:ncclient.operations.rpc:Requesting 'ExecuteRpc'
2017-04-26 15:29:02,968:paramiko.transport:Connected (version 2.0, client OpenSSH_6.4)
2017-04-26 15:29:03,960:paramiko.transport:Authentication (password) successful!
2017-04-26 15:30:56,074:10.238.0.181:jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz: 46120960 / 461111477 (10%)
2017-04-26 15:32:50,668:10.238.0.181:jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz: 92225536 / 461111477 (20%)
2017-04-26 15:34:44,433:10.238.0.181:jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz: 138346496 / 461111477 (30%)
2017-04-26 15:36:37,053:10.238.0.181:jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz: 184451072 / 461111477 (40%)
2017-04-26 15:38:31,156:10.238.0.181:jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz: 230572032 / 461111477 (50%)
2017-04-26 15:40:26,243:10.238.0.181:jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz: 276676608 / 461111477 (60%)
2017-04-26 15:42:19,847:10.238.0.181:jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz: 322781184 / 461111477 (70%)
2017-04-26 15:44:14,030:10.238.0.181:jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz: 368902144 / 461111477 (80%)
2017-04-26 15:46:06,393:10.238.0.181:jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz: 415006720 / 461111477 (90%)
2017-04-26 15:47:59,808:10.238.0.181:jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz: 461111477 / 461111477 (100%)
2017-04-26 15:48:02,499:10.238.0.181:computing remote checksum on: /var/tmp/jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz
2017-04-26 15:48:02,500:ncclient.operations.rpc:Requesting 'ExecuteRpc'
2017-04-26 15:48:07,039:10.238.0.181:checksum check passed.
2017-04-26 15:48:07,040:10.238.0.181:installing software on VC member: 1 ... please be patient ...
2017-04-26 15:48:07,040:ncclient.operations.rpc:Requesting 'ExecuteRpc'
2017-04-26 15:56:47,049:10.238.0.181:software pkgadd package-result: 0
Output:

[Apr 26 15:48:07]: Checking pending install on fpc1

Pushing /var/tmp/jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz to fpc1:/var/tmp/jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz
Verified jinstall-vjunos-14.1X53-D40.8-domestic.tgz signed by PackageProductionSwitchingEc_2015
Verified jinstall-vjunos-14.1X53-D40.8-domestic.tgz signed by JuniperSwitchingProduction_2015
Adding vjunos...
Saving contents of boot area prior to installation

WARNING:     This package will load JUNOS 14.1X53-D40.8 software.
WARNING:     It will save JUNOS configuration files, and SSH keys
WARNING:     (if configured), but erase all other files and information
WARNING:     stored on this machine.  It will attempt to preserve dumps
WARNING:     and log files, but this can not be guaranteed.  This is the
WARNING:     pre-installation stage and all the software is loaded when
WARNING:     you reboot the system.

POST-INSTALL...
Saving the config files ...
NOTICE: uncommitted changes have been saved in /var/db/config/juniper.conf.pre-install
Pushing installation package to host...
Upgrade Host OS to 14.1X53-D40.8...
./jinstall-qfx-5-14.1X53-D40.8-domestic.img.gz: OK
/boot/boot
Extracting image...
Host upgrade staging completed. Need reboot to complete upgrade installation.
WARNING: Changing next boot to SSD0
WARNING: Changing boot device enable from  0x3F to 0x3F
Install jinstall-vjunos completed

WARNING:     A REBOOT IS REQUIRED TO LOAD THIS SOFTWARE CORRECTLY. Use the
WARNING:     'request system reboot' command when software installation is
WARNING:     complete. To abort the installation, do not reboot your system,
WARNING:     instead use the 'request system software delete jinstall'
WARNING:     command as soon as this operation completes.

Saving state for rollback ...
2017-04-26 15:56:47,049:10.238.0.181:installing software on VC member: 0 ... please be patient ...
2017-04-26 15:56:47,050:ncclient.operations.rpc:Requesting 'ExecuteRpc'
2017-04-26 16:01:04,296:10.238.0.181:software pkgadd package-result: 0
Output:
Verified jinstall-vjunos-14.1X53-D40.8-domestic.tgz signed by PackageProductionSwitchingEc_2015
Verified jinstall-vjunos-14.1X53-D40.8-domestic.tgz signed by JuniperSwitchingProduction_2015
Adding vjunos...
Saving contents of boot area prior to installation

WARNING:     This package will load JUNOS 14.1X53-D40.8 software.
WARNING:     It will save JUNOS configuration files, and SSH keys
WARNING:     (if configured), but erase all other files and information
WARNING:     stored on this machine.  It will attempt to preserve dumps
WARNING:     and log files, but this can not be guaranteed.  This is the
WARNING:     pre-installation stage and all the software is loaded when
WARNING:     you reboot the system.

POST-INSTALL...
Saving the config files ...
NOTICE: uncommitted changes have been saved in /var/db/config/juniper.conf.pre-install
Pushing installation package to host...
Upgrade Host OS to 14.1X53-D40.8...
./jinstall-qfx-5-14.1X53-D40.8-domestic.img.gz: OK
/boot/boot
Extracting image...
Host upgrade staging completed. Need reboot to complete upgrade installation.
WARNING: Changing next boot to SSD0
WARNING: Changing boot device enable from  0x3F to 0x3F
Install jinstall-vjunos completed

WARNING:     A REBOOT IS REQUIRED TO LOAD THIS SOFTWARE CORRECTLY. Use the
WARNING:     'request system reboot' command when software installation is
WARNING:     complete. To abort the installation, do not reboot your system,
WARNING:     instead use the 'request system software delete jinstall'
WARNING:     command as soon as this operation completes.

Saving state for rollback ...

2017-04-26 16:01:04,297:10.238.0.181:<REBOOT>
2017-04-26 16:01:04,297:ncclient.operations.rpc:Requesting 'ExecuteRpc'
2017-04-26 16:01:05,218:10.238.0.181:upgrade pending reboot cycle, please be patient.
2017-04-26 16:01:05,218:ncclient.operations.rpc:Requesting 'CloseSession'
2017-04-26 16:01:05,801:10.238.0.181:pausing: 10
2017-04-26 16:01:15,811:10.238.0.181:process completed OK.

And the result is this one

local@tesw1> show version
fpc0:
--------------------------------------------------------------------------
Hostname: tesw1
Model: qfx5100-48t-6q
Junos: 14.1X53-D40.8
JUNOS Base OS boot [14.1X53-D40.8]
JUNOS Base OS Software Suite [14.1X53-D40.8]
JUNOS Online Documentation [14.1X53-D40.8]
JUNOS Crypto Software Suite [14.1X53-D40.8]
JUNOS Kernel Software Suite [14.1X53-D40.8]
JUNOS Packet Forwarding Engine Support (qfx-ex-x86-32) [14.1X53-D40.8]
JUNOS Routing Software Suite [14.1X53-D40.8]
JUNOS SDN Software Suite [14.1X53-D40.8]
JUNOS Enterprise Software Suite [14.1X53-D40.8]
JUNOS Web Management Platform Package [14.1X53-D40.8]
JUNOS py-base-i386 [14.1X53-D40.8]
**JUNOS Host Software [14.1X53-D40.8]**

fpc1:
--------------------------------------------------------------------------
Hostname: tesw1
Model: qfx5100-48t-6q
Junos: 14.1X53-D40.8
JUNOS Base OS boot [14.1X53-D40.8]
JUNOS Base OS Software Suite [14.1X53-D40.8]
JUNOS Online Documentation [14.1X53-D40.8]
JUNOS Crypto Software Suite [14.1X53-D40.8]
JUNOS Kernel Software Suite [14.1X53-D40.8]
JUNOS Packet Forwarding Engine Support (qfx-ex-x86-32) [14.1X53-D40.8]
JUNOS Routing Software Suite [14.1X53-D40.8]
JUNOS SDN Software Suite [14.1X53-D40.8]
JUNOS Enterprise Software Suite [14.1X53-D40.8]
JUNOS Web Management Platform Package [14.1X53-D40.8]
JUNOS py-base-i386 [14.1X53-D40.8]
**JUNOS Host Software [14.1X53-D40.8]**

I really appreciate the help. Works great!

Are you going to implement it into the master branch?

Thank you very much and best regards,

Nzkller

[nzkller]

@stacywsmith

So it did the upgrade but then I tried it in a non virtual-chassis standalone switch i got the following error

TASK [upgrade-tors : Install Junos OS package in IPMI Switches] ****************
task path: /etc/ansible/roles/upgrade-tors/tasks/main.yaml:21
Using module file /etc/ansible/roles/Juniper.junos/library/junos_install_os
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1493229365.95-131200277699358 `" && echo ansible-tmp-1493229365.95-131200277699358="` echo ~/.ansible/tmp/ansible-tmp-1493229365.95-131200277699358 `" ) && sleep 0'
<localhost> PUT /tmp/user/0/tmpw3m3WP TO /root/.ansible/tmp/ansible-tmp-1493229365.95-131200277699358/junos_install_os
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1493229365.95-131200277699358/ /root/.ansible/tmp/ansible-tmp-1493229365.95-131200277699358/junos_install_os && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1493229365.95-131200277699358/junos_install_os; rm -rf "/root/.ansible/tmp/ansible-tmp-1493229365.95-131200277699358/" > /dev/null 2>&1 && sleep 0'
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/tmp/user/0/ansible_zaP05h/ansible_module_junos_install_os.py", line 317, in <module>
    main()
  File "/tmp/user/0/ansible_zaP05h/ansible_module_junos_install_os.py", line 302, in main
    results = junos_install_os(module, dev)
  File "/tmp/user/0/ansible_zaP05h/ansible_module_junos_install_os.py", line 230, in junos_install_os
    ok = sw.install(package, **sw_args)
  File "/usr/local/lib/python2.7/dist-packages/jnpr/junos/utils/sw.py", line 699, in install
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/jnpr/junos/utils/sw.py", line 176, in pkgadd
    rsp = self.rpc.request_package_add(**args)
  File "/usr/local/lib/python2.7/dist-packages/jnpr/junos/rpcmeta.py", line 336, in _exec_rpc
    return self._junos.execute(rpc, **dec_args)
  File "/usr/local/lib/python2.7/dist-packages/jnpr/junos/decorators.py", line 76, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/jnpr/junos/decorators.py", line 23, in wrapper
    result = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/jnpr/junos/device.py", line 722, in execute
    errs=ex)
jnpr.junos.exception.RpcError: RpcError(severity: error, bad_element: None, message: Could not connect to fpc1 : Can't assign requested address)

failed: [localhost] 
=> {
    "failed": true,
    "invocation": {
        "module_name": "junos_install_os"
    },
    "item": {
        "hostnameosw": "osw1",
        "hostnametesw": "tesw1",
        "iposw": "10.238.0.182",
        "iptesw": "10.238.0.181",
        "packageosw": "/data/repository/Juniper/jinstall-ex-2200-12.3R12.4-domestic-signed.tgz",
        "packagetesw": "/data/repository/Juniper/jinstall-qfx-5-14.1X53-D40.8-domestic-signed.tgz",
        "versionosw": "12.3R12.4",
        "versiontesw": "14.1X53-D40.8"
    },
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/user/0/ansible_zaP05h/ansible_module_junos_install_os.py\", line 317, in <module>\n    main()\n  File \"/tmp/user/0/ansible_zaP05h/ansible_module_junos_install_os.py\", line 302, in main\n    results = junos_install_os(module, dev)\n  File \"/tmp/user/0/ansible_zaP05h/ansible_module_junos_install_os.py\", line 230, in junos_install_os\n    ok = sw.install(package, **sw_args)\n  File \"/usr/local/lib/python2.7/dist-packages/jnpr/junos/utils/sw.py\", line 699, in install\n    **kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/jnpr/junos/utils/sw.py\", line 176, in pkgadd\n    rsp = self.rpc.request_package_add(**args)\n  File \"/usr/local/lib/python2.7/dist-packages/jnpr/junos/rpcmeta.py\", line 336, in _exec_rpc\n    return self._junos.execute(rpc, **dec_args)\n  File \"/usr/local/lib/python2.7/dist-packages/jnpr/junos/decorators.py\", line 76, in wrapper\n    return function(*args, **kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/jnpr/junos/decorators.py\", line 23, in wrapper\n    result = function(*args, **kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/jnpr/junos/device.py\", line 722, in execute\n    errs=ex)\njnpr.junos.exception.RpcError: RpcError(severity: error, bad_element: None, message: Could not connect to fpc1 : Can't assign requested address)\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE"
}

Is this related to the changes?

Best regards,

Nzkller

[nzkller]

Ok

Now it's working fine. It was the dependency py-junos-eznc. I was using version 2.1.1 and I install version 2.0.1 and it works fine. I'm using pip to install.

Should I open an issue in https://github.com/Juniper/py-junos-eznc/issues for this?

Best regards,

Nzkller

[stacywsmith]

@nzkller Thanks for confirming this issue is resolved by my change.

Yes. Please do file an issue against py-junos-eznc for the other problem you are seeing.

[stacywsmith]

@nzkller

I was thinking about your issue with PyEZ 2.1.1 a little more.

I think it MAY be addressed by Juniper/py-junos-eznc#700 which was recently committed. Can you please upgrade to the latest development version of PyEZ from GitHub using:

sudo pip install --upgrade git+https://github.com/Juniper/py-junos-eznc.git

Please let me know if that resolves your problem. If not, you can open an issue against PyEZ.