Current implementation doesn't handle multi member stacks where versions of members are inconsistent

[lukebaldan]

Issue Type

• Bug Report

Module Name

juniper.device.software
juniper.device collection and Python libraries version

Juniper device collection 1.0.2

ansible [core 2.14.2]
  config file = /project/junos_utils/tests/xgrade/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.10/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.9 (main, Dec 12 2022, 17:52:15) [GCC 12.2.1 20220924] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = False

ansible==7.2.0
ansible-core==2.14.2
awxkit==3.8.6
bcrypt==4.0.1
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==3.0.1
cryptography==39.0.0
idna==3.4
Jinja2==3.1.2
junos-eznc==2.6.6
jxmlease==1.0.3
lxml==4.9.2
MarkupSafe==2.1.2
ncclient==0.6.13
netaddr==0.8.0
packaging==23.0
paramiko==3.0.0
pycparser==2.21
PyNaCl==1.5.0
pyparsing==3.0.9
pyserial==3.5
PyYAML==6.0
requests==2.28.2
resolvelib==0.8.1
scp==0.14.5
six==1.16.0
transitions==0.9.0
urllib3==1.26.14
xmltodict==0.13.0
yamlordereddictloader==0.4.0

OS / Environment

Juniper EX4200 configured with 3 members and following starting versions:

• FPC0 (Master): 12.3R12-S12
• FPC1 (Backup): 12.3R11.2
• FPC2 (Linecard): 12.3R11.2

Summary

The existing juniper.device.software module seems to work fine if all members of the switch stack are on the same starting version.
If the current re (Member 0) is on version B and members 1 and 2 are on version A. Attempting an upgrade to version B will fail because of the following lines of code which checks the version of the current RE only.

Ideally the module should be able to upgrade all members of the stack regardless of their current version to the target version.

    if target_version is not None:
        if all_re is True:
            junos_info = facts['junos_info']
            for current_re in junos_info:
                current_version = junos_info[current_re]['text']
                if target_version != current_version:
                    junos_module.logger.debug("Current version on %s: %s. "
                                              "Target version: %s.",
                                              current_version, current_re,
                                              target_version)
                    results['changed'] = True
                else:
                    results['msg'] += "Current version on %s: %s same as Targeted " \
                                      "version: %s.\n" % (current_version, current_re,
                                                          target_version)

Steps to reproduce

    - name: Install Junos OS package
      juniper.device.software:
        reboot: false
        # For the purpose of this test, local package points to file containing 12.3R12-S12 release.
        local_package: "{{ pkg_dir }}/{{ chassis_model }}/{{ current_step_file }}"

Expected results

Expect Members 1 and 2 to be upgraded to 12.3R12-S12 to match master / member 0.

Actual results

Instead, debug reveals that current version is the same as targeted and the upgrade is not performed:

ok: [X] => {
"changed": false,
"check_mode": false,
"invocation": {
"module_args": {
"attempts": null,
"baud": null,
"checksum": null,
"checksum_algorithm": "md5",
"checksum_timeout": 300,
"cleanfs_timeout": 300,
"console": null,
"cs_passwd": null,
"cs_user": null,
"force_host": false,
"host": "10.x.x.x",
"huge_tree": false,
"issu": false,
"level": null,
"logdir": null,
"logfile": "/tmp/junosutils/files/logs/x.log",
"mode": null,
"nssu": false,
"passwd": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"port": 830,
"ssh_config": null,
"ssh_private_key_file": null,
"timeout": 10,
"user": "x",
"validate": false,
"vmhost": false
}
},
"msg": "Current version on 12.3R12-S12: fpc0 same as Targeted version: 12.3R12-S12.\n"
}

[chidanandpujar]

Hi @lukebaldan ,

Please try with parameter "member_id" with list of member ids to be installed with software package .
member_id: ['1','2']

---
- name: Test juniper.device.software module
  hosts: all
  collections:
    - juniper.device
  connection: local
  gather_facts: no
  vars:
    wait_time: 3600
    pkg_dir: /var/tmp/
    OS_version: 
    OS_package: 
    log_dir: /var/log/

  tasks:
    - name: Checking NETCONF connectivity
      wait_for: host={{ ansible_ssh_host }} port=830 timeout=5
    - name: Install Junos OS package
      software:
        reboot: False 
        no_copy: True
        version: "{{ OS_version }}"
        package: "{{ pkg_dir }}/{{ OS_package }}"
        logfile: "{{ log_dir }}/software.log"
        member_id: ['1','2']

Thanks & Regards
Chidanand

[chidanandpujar]

Fix is merged - #617

[lukebaldan]

Hi @chidanandpujar. I have tested in my environment and it appears the new code is broken. Even if member_id is not specified, the upgrade no longer completes:

TASK [Install Junos OS package - Specific Member fpc1, jinstall-ex-4200-12.3R11.2-domestic-signed.tgz] ********************************************************************************************************
task path: /project/junos_utils/tests/xgrade/playbook_test.yml:32
The full traceback is:
File "/tmp/ansible_juniper.device.software_payload_khpbe7ik/ansible_juniper.device.software_payload.zip/ansible_collections/juniper/device/plugins/modules/software.py", line 712, in main
File "/usr/lib/python3.10/site-packages/jnpr/junos/utils/sw.py", line 990, in install
add_ok = self.pkgadd(
File "/usr/lib/python3.10/site-packages/jnpr/junos/utils/sw.py", line 240, in pkgadd
rsp = self.rpc.request_package_add(**args)
File "/usr/lib/python3.10/site-packages/jnpr/junos/rpcmeta.py", line 363, in _exec_rpc
return self._junos.execute(rpc, **dec_args)
File "/usr/lib/python3.10/site-packages/jnpr/junos/decorators.py", line 76, in wrapper
return function(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/jnpr/junos/decorators.py", line 23, in wrapper
result = function(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/jnpr/junos/device.py", line 854, in execute
raise EzErrors.RpcError(cmd=rpc_cmd_e, rsp=rsp, errs=ex)
fatal: [switch1_ex4200]: FAILED! => {
"changed": true,
"check_mode": false,
"invocation": {
"module_args":

{ "attempts": null, "baud": null, "checksum": null, "checksum_algorithm": "md5", "checksum_timeout": 300, "cleanfs_timeout": 300, "console": null, "cs_passwd": null, "cs_user": null, "force_host": false, "host": "10.0.52.75", "huge_tree": false, "issu": false, "level": null, "logdir": null, "logfile": "/tmp/junosutils/files/logs/switch1_ex4200_1_software.log", "mode": null, "nssu": false, "passwd": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "port": 830, "ssh_config": null, "ssh_private_key_file": null, "timeout": 10, "user": "svc_ansible", "validate": false, "vmhost": false }
},
"msg": "Installation failed. Error: RpcError(severity: error, bad_element: member-id, message: error: syntax error\nerror: syntax error)"
}

PLAY RECAP ****************************************************************************************************************************************************************************************************
switch1_ex4200 : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

I suspect the rpc does not support the member-id parameter. Interestingly if I omit the member_id parameter to perform an upgrade on all RE's, I still encounter a fail:

fatal: [switch1_ex4200]: FAILED! => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3.10"
},
"changed": false,
"module_stderr": "Traceback (most recent call last):\n File "/root/.ansible/tmp/ansible-tmp-1696480052.7300484-2995-132287938713464/AnsiballZ_software.py", line 107, in \n _ansiballz_main()\n File "/root/.ansible/tmp/ansible-tmp-1696480052.7300484-2995-132287938713464/AnsiballZ_software.py", line 99, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File "/root/.ansible/tmp/ansible-tmp-1696480052.7300484-2995-132287938713464/AnsiballZ_software.py", line 47, in invoke_module\n runpy.run_module(mod_name='ansible_collections.juniper.device.plugins.modules.software', init_globals=dict(_module_fqn='ansible_collections.juniper.device.plugins.modules.software', _modlib_path=modlib_path),\n File "/usr/lib/python3.10/runpy.py", line 224, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File "/usr/lib/python3.10/runpy.py", line 86, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_juniper.device.software_payload_sn03o87y/ansible_juniper.device.software_payload.zip/ansible_collections/juniper/device/plugins/modules/software.py", line 810, in \n File "/tmp/ansible_juniper.device.software_payload_sn03o87y/ansible_juniper.device.software_payload.zip/ansible_collections/juniper/device/plugins/modules/software.py", line 712, in main\n File "/usr/lib/python3.10/site-packages/jnpr/junos/utils/sw.py", line 990, in install\n add_ok = self.pkgadd(\n File "/usr/lib/python3.10/site-packages/jnpr/junos/utils/sw.py", line 240, in pkgadd\n rsp = self.rpc.request_package_add(**args)\n File "/usr/lib/python3.10/site-packages/jnpr/junos/rpcmeta.py", line 342, in _exec_rpc\n raise TypeError(\nTypeError: The value None for argument member-id is of <class 'NoneType'>. Argument values must be a string, boolean, or list/tuple of strings and booleans.\n",
"module_stdout": "",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
"rc": 1
}

[chidanandpujar]

Hi @lukebaldan
Thanks very much for sharing the logs .
I have not checked the upgrade with jinstall-ex-4200-12.3R11.2-domestic-signed.tgz image ,
Let me upgrade and check the issues.

Thanks

[chidanandpujar]

Hi @lukebaldan
Thanks ,
Could you please install the latest PyEZ version 2.6.8 and re-try the installation without member-id .

~/sw_install_member_new_issues/ansible-junos-stdlib/tests# ansible-playbook pb.juniper_junos_software_ex3400.yml -vvvv
ansible-playbook [core 2.15.4]
  config file = /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /root/sw_install_member_new_issues/venv/lib/python3.9/site-packages/ansible
  ansible collection location = /root/sw_install_member_new_issues/ansible-junos-stdlib/ansible_collections
  executable location = /root/sw_install_member_new_issues/venv/bin/ansible-playbook
  python version = 3.9.6 (default, Oct 18 2021, 17:33:10) [GCC 7.5.0] (/root/sw_install_member_new_issues/venv/bin/python)
  jinja version = 3.1.2
  libyaml = True
Using /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/ansible.cfg as config file
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/inventory as it did not pass its verify_file() method
script declined parsing /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/inventory as it did not pass its verify_file() method
auto declined parsing /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/inventory as it did not pass its verify_file() method
Parsed /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/inventory inventory source with ini plugin
Loading collection juniper.device from /root/sw_install_member_new_issues/ansible-junos-stdlib/ansible_collections/juniper/device
Loading callback plugin default of type stdout, v2.0 from /root/sw_install_member_new_issues/venv/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: pb.juniper_junos_software_ex3400.yml ****************************************************************************************************************************
Positional arguments: pb.juniper_junos_software_ex3400.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/root/sw_install_member_new_issues/ansible-junos-stdlib/tests/inventory',)
forks: 5
1 plays in pb.juniper_junos_software_ex3400.yml

PLAY [Test juniper.device.software module] ********************************************************************************************************************************

TASK [Checking NETCONF connectivity] **************************************************************************************************************************************
task path: /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/pb.juniper_junos_software_ex3400.yml:16
<10.48.12.144> ESTABLISH LOCAL CONNECTION FOR USER: root
<10.48.12.144> EXEC /bin/sh -c 'echo ~root && sleep 0'
<10.48.12.144> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1696504006.511924-6803-231908507328892 `" && echo ansible-tmp-1696504006.511924-6803-231908507328892="` echo /root/.ansible/tmp/ansible-tmp-1696504006.511924-6803-231908507328892 `" ) && sleep 0'
Using module file /root/sw_install_member_new_issues/venv/lib/python3.9/site-packages/ansible/modules/wait_for.py
<10.48.12.144> PUT /root/.ansible/tmp/ansible-local-6798cax26_r2/tmpiaef865x TO /root/.ansible/tmp/ansible-tmp-1696504006.511924-6803-231908507328892/AnsiballZ_wait_for.py
<10.48.12.144> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1696504006.511924-6803-231908507328892/ /root/.ansible/tmp/ansible-tmp-1696504006.511924-6803-231908507328892/AnsiballZ_wait_for.py && sleep 0'
<10.48.12.144> EXEC /bin/sh -c '/root/sw_install_member/venv/bin/python /root/.ansible/tmp/ansible-tmp-1696504006.511924-6803-231908507328892/AnsiballZ_wait_for.py && sleep 0'
<10.48.12.144> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1696504006.511924-6803-231908507328892/ > /dev/null 2>&1 && sleep 0'
ok: [test] => {
    "changed": false,
    "elapsed": 0,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 0,
            "exclude_hosts": null,
            "host": "10.48.12.144",
            "msg": null,
            "path": null,
            "port": 830,
            "search_regex": null,
            "sleep": 1,
            "state": "started",
            "timeout": 5
        }
    },
    "match_groupdict": {},
    "match_groups": [],
    "path": null,
    "port": 830,
    "search_regex": null,
    "state": "started"
}

TASK [Install Junos OS package] *******************************************************************************************************************************************
task path: /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/pb.juniper_junos_software_ex3400.yml:18
<10.48.12.144> ESTABLISH LOCAL CONNECTION FOR USER: root
<10.48.12.144> EXEC /bin/sh -c 'echo ~root && sleep 0'
<10.48.12.144> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1696504007.2348301-6830-12089999658631 `" && echo ansible-tmp-1696504007.2348301-6830-12089999658631="` echo /root/.ansible/tmp/ansible-tmp-1696504007.2348301-6830-12089999658631 `" ) && sleep 0'
Using module file /root/sw_install_member_new_issues/ansible-junos-stdlib/ansible_collections/juniper/device/plugins/modules/software.py
<10.48.12.144> PUT /root/.ansible/tmp/ansible-local-6798cax26_r2/tmpx3boyvx0 TO /root/.ansible/tmp/ansible-tmp-1696504007.2348301-6830-12089999658631/AnsiballZ_software.py
<10.48.12.144> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1696504007.2348301-6830-12089999658631/ /root/.ansible/tmp/ansible-tmp-1696504007.2348301-6830-12089999658631/AnsiballZ_software.py && sleep 0'
<10.48.12.144> EXEC /bin/sh -c '/root/sw_install_member/venv/bin/python /root/.ansible/tmp/ansible-tmp-1696504007.2348301-6830-12089999658631/AnsiballZ_software.py && sleep 0'
<10.48.12.144> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1696504007.2348301-6830-12089999658631/ > /dev/null 2>&1 && sleep 0'
Notification for handler wait_reboot has been saved.
changed: [test] => {
    "changed": true,
    "check_mode": false,
    "invocation": {
        "module_args": {
            "attempts": null,
            "baud": null,
            "checksum": null,
            "checksum_algorithm": "md5",
            "checksum_timeout": 300,
            "cleanfs_timeout": 300,
            "console": null,
            "cs_passwd": null,
            "cs_user": null,
            "force_host": false,
            "host": "10.48.12.144",
            "huge_tree": false,
            "issu": false,
            "level": null,
            "logdir": null,
            "logfile": "/var/log//software.log",
            "mode": null,
            "nssu": false,
            "package": "/var/tmp//jinstall-ex-4200-12.3-20180426.0-domestic.tgz",
            "passwd": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "port": 830,
            "ssh_config": null,
            "ssh_private_key_file": null,
            "timeout": 10,
            "user": "root",
            "validate": false,
            "vmhost": false
        }
    },
    "msg": "Package /var/tmp/jinstall-ex-4200-12.3-20180426.0-domestic.tgz successfully installed. Response from device is: \n\n\nWARNING:     The software that is being installed has limited support.\nWARNING:     Run 'file show /etc/notices/unsupported.txt' for details.\n\nWARNING: A reboot is required to install the software\nWARNING:     Use the 'request system reboot' command immediately\n Reboot successfully initiated. Reboot message: \n\nRebooting fpc0\n"
}

TASK [Print response] *****************************************************************************************************************************************************
task path: /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/pb.juniper_junos_software_ex3400.yml:31
ok: [test] => {
    "test1": {
        "changed": true,
        "check_mode": false,
        "failed": false,
        "msg": "Package /var/tmp/jinstall-ex-4200-12.3-20180426.0-domestic.tgz successfully installed. Response from device is: \n\n\nWARNING:     The software that is being installed has limited support.\nWARNING:     Run 'file show /etc/notices/unsupported.txt' for details.\n\nWARNING: A reboot is required to install the software\nWARNING:     Use the 'request system reboot' command immediately\n Reboot successfully initiated. Reboot message: \n\nRebooting fpc0\n"
    }
}

TASK [Check TEST - 1] *****************************************************************************************************************************************************
task path: /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/pb.juniper_junos_software_ex3400.yml:35
ok: [test] => {
    "changed": false,
    "msg": "All assertions passed"
}
NOTIFIED HANDLER wait_reboot for test

RUNNING HANDLER [wait_reboot] *********************************************************************************************************************************************
task path: /root/sw_install_member_new_issues/ansible-junos-stdlib/tests/pb.juniper_junos_software_ex3400.yml:41
<10.48.12.144> ESTABLISH LOCAL CONNECTION FOR USER: root
<10.48.12.144> EXEC /bin/sh -c 'echo ~root && sleep 0'
<10.48.12.144> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1696504052.944833-6891-235381864439700 `" && echo ansible-tmp-1696504052.944833-6891-235381864439700="` echo /root/.ansible/tmp/ansible-tmp-1696504052.944833-6891-235381864439700 `" ) && sleep 0'
Using module file /root/sw_install_member_new_issues/venv/lib/python3.9/site-packages/ansible/modules/wait_for.py
<10.48.12.144> PUT /root/.ansible/tmp/ansible-local-6798cax26_r2/tmp6vvb47i7 TO /root/.ansible/tmp/ansible-tmp-1696504052.944833-6891-235381864439700/AnsiballZ_wait_for.py
<10.48.12.144> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1696504052.944833-6891-235381864439700/ /root/.ansible/tmp/ansible-tmp-1696504052.944833-6891-235381864439700/AnsiballZ_wait_for.py && sleep 0'
<10.48.12.144> EXEC /bin/sh -c '/root/sw_install_member/venv/bin/python /root/.ansible/tmp/ansible-tmp-1696504052.944833-6891-235381864439700/AnsiballZ_wait_for.py && sleep 0'

Thanks

[chidanandpujar]

Hi @lukebaldan ,
Thanks ,
Could you please install the latest PyEZ and execue the playbook .

I am able to execute the playbook without any issues .

~/sw_install_member_new_issues/ansible-junos-stdlib/tests# vi pb.juniper_junos_software_member.yml 
---
- name: Test juniper.device.software module
  hosts: all
  collections:
    - juniper.device
  connection: local
  gather_facts: no
  vars:
    wait_time: 3600
    pkg_dir: /var/tmp/
    OS_version: 12.3
    OS_package: jinstall-ex-4200-12.3-20180426.0-domestic.tgz
    log_dir: /var/log/

  tasks:
    - name: Checking NETCONF connectivity
      wait_for: host={{ ansible_ssh_host }} port=830 timeout=5
    - name: Install Junos OS package
      software:
        reboot: False
        no_copy: True
        version: "{{ OS_version }}"
        package: "{{ pkg_dir }}/{{ OS_package }}"
        logfile: "{{ log_dir }}/software.log"
        all_re: False
        member_id: ['0']
      register: test1
      notify:
        - wait_reboot

    - name: Print response
      debug:
        var: test1

    - name: Check TEST - 1
      assert:
        that:
          - test1.failed == false
"pb.juniper_junos_software_member.yml" 43L, 1041C written                                                                                      

:~/sw_install_member_new_issues/ansible-junos-stdlib/tests# ansible-playbook pb.juniper_junos_software_member.yml 

PLAY [Test juniper.device.software module] ********************************************************************************************************************************

TASK [Checking NETCONF connectivity] **************************************************************************************************************************************
ok: [test]

TASK [Install Junos OS package] *******************************************************************************************************************************************
changed: [test]

TASK [Print response] *****************************************************************************************************************************************************
ok: [test] => {
    "test1": {
        "changed": true,
        "check_mode": false,
        "failed": false,
        "msg": "Package /var/tmp/jinstall-ex-4200-12.3-20180426.0-domestic.tgz successfully installed. Response from device is: \n\n\nWARNING:     The software that is being installed has limited support.\nWARNING:     Run 'file show /etc/notices/unsupported.txt' for details.\n\nWARNING: A reboot is required to install the software\nWARNING:     Use the 'request system reboot' command immediately\n"
    }
}

TASK [Check TEST - 1] *****************************************************************************************************************************************************
ok: [test] => {
    "changed": false,
    "msg": "All assertions passed"
}

RUNNING HANDLER [wait_reboot] *********************************************************************************************************************************************
ok: [test]

PLAY RECAP ****************************************************************************************************************************************************************
test                       : ok=5    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Thanks

[lukebaldan]

Hi @chidanandpujar Based on your feedback, i upgraded PyEZ and i am happy to report that initial testing looks good. I am able to successfully upgrade/downgrade individual members without version checking issues. I will continue to test, but i am 95% confident, this feature has been implemented correctly.
Thanks for your help with this.