Junos ansible module breaks on Ansible 2.9

[mtucker502]

Issue Type

• Bug Report

module_utils/juniper_junos_common.py

Info

Juniper.Junos role and Python libraries version

$ ansible --version
ansible 2.9.0
  config file = /Users/matucker/Downloads/git/junos-ansible-examples/junos/ansible.cfg
  configured module search path = [u'/Users/matucker/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/matucker/Downloads/git/junos-ansible-examples/venv/lib/python2.7/site-packages/ansible
  executable location = /Users/matucker/Downloads/git/junos-ansible-examples/venv/bin/ansible
  python version = 2.7.10 (default, Feb 22 2019, 21:55:15) [GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]

$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
ansible==2.9.0
bcrypt==3.1.7
cffi==1.13.2
cryptography==2.8
enum34==1.1.6
ipaddress==1.0.23
Jinja2==2.10.3
junos-eznc==2.3.0
lxml==4.4.1
MarkupSafe==1.1.1
ncclient==0.6.6
netaddr==0.7.19
paramiko==2.6.0
pycparser==2.19
PyNaCl==1.3.0
pyparsing==2.4.2
pyserial==3.4
PyYAML==5.1.2
scp==0.13.2
selectors2==2.0.1
six==1.12.0
transitions==0.7.1
yamlordereddictloader==0.4.0

$ ansible-config dump --only-changed
DEFAULT_HOST_LIST(/Users/matucker/Downloads/git/junos-ansible-examples/junos/ansible.cfg) = [u'/Users/matucker/Downloads/git/junos-ansible-examples/junos/hosts']
HOST_KEY_CHECKING(/Users/matucker/Downloads/git/junos-ansible-examples/junos/ansible.cfg) = False

OS / Environment

N/A

Summary

ansible/ansible#61621
ansible/ansible#64324

Steps to reproduce

Try to use any juniper_junos_* module with Ansible 2.9.

    - name: Run Commands
      juniper_junos_command:
        user: "{{ ansible_user }}"
        passwd: "{{ ansible_password }}"
        host: "{{ ansible_host }}"
        port: "{{ ansible_port }}"
        commands: "show version"
        formats: text
        dest: "{{playbook_dir}}/output/{{ inventory_hostname }}.txt"

Expected results

No import errors.

Actual results

The full traceback is:
Traceback (most recent call last):
  File "/Users/matucker/.ansible/tmp/ansible-tmp-1572894354.81-100059189556901/AnsiballZ_juniper_junos_command.py", line 102, in <module>
    _ansiballz_main()
  File "/Users/matucker/.ansible/tmp/ansible-tmp-1572894354.81-100059189556901/AnsiballZ_juniper_junos_command.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/matucker/.ansible/tmp/ansible-tmp-1572894354.81-100059189556901/AnsiballZ_juniper_junos_command.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.juniper_junos_command', init_globals=None, run_name='__main__', alter_sys=False)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 180, in run_module
    fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/var/folders/04/g24gg21176zflhxqzl_82y9c0000gn/T/ansible_juniper_junos_command_payload_A6fDzi/ansible_juniper_junos_command_payload.zip/ansible/modules/juniper_junos_command.py", line 312, in <module>
  File "/var/folders/04/g24gg21176zflhxqzl_82y9c0000gn/T/ansible_juniper_junos_command_payload_A6fDzi/ansible_juniper_junos_command_payload.zip/ansible/module_utils/juniper_junos_common.py", line 39, in <module>
  File "/Users/matucker/Downloads/git/junos-ansible-examples/venv/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 22, in <module>
    from ansible.executor.module_common import modify_module
  File "/Users/matucker/Downloads/git/junos-ansible-examples/venv/lib/python2.7/site-packages/ansible/executor/module_common.py", line 38, in <module>
    from ansible.executor.powershell import module_manifest as ps_manifest
  File "/Users/matucker/Downloads/git/junos-ansible-examples/venv/lib/python2.7/site-packages/ansible/executor/powershell/module_manifest.py", line 26, in <module>
    from ansible.plugins.loader import ps_module_utils_loader
  File "/Users/matucker/Downloads/git/junos-ansible-examples/venv/lib/python2.7/site-packages/ansible/plugins/loader.py", line 22, in <module>
    from ansible.parsing.utils.yaml import from_yaml
  File "/Users/matucker/Downloads/git/junos-ansible-examples/venv/lib/python2.7/site-packages/ansible/parsing/utils/yaml.py", line 19, in <module>
    from ansible.parsing.ajson import AnsibleJSONDecoder
  File "/Users/matucker/Downloads/git/junos-ansible-examples/venv/lib/python2.7/site-packages/ansible/parsing/ajson.py", line 11, in <module>
    from ansible.module_utils.common.json import AnsibleJSONEncoder
ImportError: No module named json

[mtucker502]

@wikwinia

I too have run into this issue. Let's see what Nitin thinks and I will work on a PR.

[nertwork]

I have run into this issue as well - wondering what the priority/progress of this fix is.

Thanks!

[paulinster]

Also having the same issue... Downgrading to ansible 2.8 solve the issue..

[sliddjur]

Is this related? I just upgraded from Ubuntu 19.04 to 19.10. And at the same time I also upgraded to Ansible 2.9, from launchpad PPA. I dont know what actually broke. I also put interpreter_python=auto in my ansible.cfg

olof@olof-ubuntu ~/l/g/juniper> ansible-playbook compare-config.yml --limit lab -vvv
ansible-playbook 2.9.1
  config file = /home/olof/lo@g.com/git/juniper/ansible.cfg
  configured module search path = [u'/home/olof/ntc-ansible']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.17rc1 (default, Oct 10 2019, 10:26:01) [GCC 9.2.1 20191008]
Using /home/olof/lo@g.com/git/juniper/ansible.cfg as config file
host_list declined parsing /home/olof/lo@g.com/git/juniper/hosts.yml as it did not pass its verify_file() method
script declined parsing /home/olof/lo@g.com/git/juniper/hosts.yml as it did not pass its verify_file() method
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. 
This feature will be removed in version 2.10. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

Parsed /home/olof/lo@g.com/git/juniper/hosts.yml inventory source with yaml plugin

PLAYBOOK: compare-config.yml ********************************************************************************************************************************************************************
1 plays in compare-config.yml

PLAY [Playbook to compare running config with local files.] *************************************************************************************************************************************
META: ran handlers

TASK [compare-config : Loading configuration and performing a diff] *****************************************************************************************************************************
task path: /home/olof/lo@g.com/git/juniper/roles/compare-config/tasks/main.yml:2
<10.18.0.36> ESTABLISH LOCAL CONNECTION FOR USER: olof
<10.18.0.36> EXEC /bin/sh -c 'echo ~olof && sleep 0'
<10.18.0.36> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248 `" && echo ansible-tmp-1573822676.7-257586551633248="` echo /home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248 `" ) && sleep 0'
<o12-ls03> Attempting python interpreter discovery
<10.18.0.36> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<10.18.0.36> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
Using module file /home/olof/.ansible/roles/Juniper.junos/library/juniper_junos_config.py
<10.18.0.36> PUT /home/olof/.ansible/tmp/ansible-local-15812jIBh8M/tmpt8jNCl TO /home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/AnsiballZ_juniper_junos_config.py
<10.18.0.36> EXEC /bin/sh -c 'chmod u+x /home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/ /home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/AnsiballZ_juniper_junos_config.py && sleep 0'
<10.18.0.36> EXEC /bin/sh -c '/usr/bin/python3 /home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/AnsiballZ_juniper_junos_config.py && sleep 0'
<10.18.0.36> EXEC /bin/sh -c 'rm -f -r /home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/AnsiballZ_juniper_junos_config.py", line 102, in <module>
    _ansiballz_main()
  File "/home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/AnsiballZ_juniper_junos_config.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/AnsiballZ_juniper_junos_config.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.juniper_junos_config', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_juniper_junos_config_payload_9u09lqtc/ansible_juniper_junos_config_payload.zip/ansible/modules/juniper_junos_config.py", line 741, in <module>
  File "/tmp/ansible_juniper_junos_config_payload_9u09lqtc/ansible_juniper_junos_config_payload.zip/ansible/module_utils/juniper_junos_common.py", line 40, in <module>
ModuleNotFoundError: No module named 'ansible.plugins'

fatal: [o12-ls03]: FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    }, 
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/AnsiballZ_juniper_junos_config.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/AnsiballZ_juniper_junos_config.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/olof/.ansible/tmp/ansible-tmp-1573822676.7-257586551633248/AnsiballZ_juniper_junos_config.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.juniper_junos_config', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib/python3.7/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.7/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib/python3.7/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_juniper_junos_config_payload_9u09lqtc/ansible_juniper_junos_config_payload.zip/ansible/modules/juniper_junos_config.py\", line 741, in <module>\n  File \"/tmp/ansible_juniper_junos_config_payload_9u09lqtc/ansible_juniper_junos_config_payload.zip/ansible/module_utils/juniper_junos_common.py\", line 40, in <module>\nModuleNotFoundError: No module named 'ansible.plugins'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", 
    "rc": 1
}

PLAY RECAP **************************************************************************************************************************************************************************************
o12-ls03                   : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[rsmekala]

I have raised the pull request #467. I am sorry for the delay.

[rsmekala]

@mtucker Thanks for the references, it really helped me to narrow it down.

The following comment summarizes the issue:
ansible/ansible#61621 (comment)

[toghraee]

Hi
I have the same issue on an updated Ansible box.

ansible --version
ansible 2.9.1
config file = /etc/ansible/ansible.cfg
configured module search path = ['/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.6.9 (default, Sep 3 2019, 16:17:03) [GCC 4.4.7 20120313 (Red Hat 4.4.7-23)]

The full traceback is:
Traceback (most recent call last):
File "/home/rtoghrae/.ansible/tmp/ansible-tmp-1575565955.4441965-177832034690863/AnsiballZ_juniper_junos_software.py", line 102, in
_ansiballz_main()
File "/home/rtoghrae/.ansible/tmp/ansible-tmp-1575565955.4441965-177832034690863/AnsiballZ_juniper_junos_software.py", line 94, in _ansiballz_main
invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
File "/home/rtoghrae/.ansible/tmp/ansible-tmp-1575565955.4441965-177832034690863/AnsiballZ_juniper_junos_software.py", line 40, in invoke_module
runpy.run_module(mod_name='ansible.modules.juniper_junos_software', init_globals=None, run_name='main', alter_sys=True)
File "/usr/local/lib/python3.6/runpy.py", line 205, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/ansible_juniper_junos_software_payload_zg6euena/ansible_juniper_junos_software_payload.zip/ansible/modules/juniper_junos_software.py", line 394, in
File "/tmp/ansible_juniper_junos_software_payload_zg6euena/ansible_juniper_junos_software_payload.zip/ansible/module_utils/juniper_junos_common.py", line 39, in
File "/usr/local/lib/python3.6/site-packages/ansible/plugins/action/init.py", line 22, in
from ansible.executor.module_common import modify_module
File "/usr/local/lib/python3.6/site-packages/ansible/executor/module_common.py", line 38, in
from ansible.executor.powershell import module_manifest as ps_manifest
File "/usr/local/lib/python3.6/site-packages/ansible/executor/powershell/module_manifest.py", line 26, in
from ansible.plugins.loader import ps_module_utils_loader
File "/usr/local/lib/python3.6/site-packages/ansible/plugins/loader.py", line 22, in
from ansible.parsing.utils.yaml import from_yaml
File "/usr/local/lib/python3.6/site-packages/ansible/parsing/utils/yaml.py", line 19, in
from ansible.parsing.ajson import AnsibleJSONDecoder
File "/usr/local/lib/python3.6/site-packages/ansible/parsing/ajson.py", line 11, in
from ansible.module_utils.common.json import AnsibleJSONEncoder

[tobzsc]

I am experiencing exactly the same issue. :( Hopefully, it will be fixed soon. #467 still waiting to be tested...