RuntimeError: load_plugins() called twice - Ansible 2.10.5, RHEL 8, package module

[dsgnr]

When using the package module with the latest RC (as well as latest commit on master branch) of Mitogen for Ansible 2.10.5, I am receiving an exception;

fatal: [EL8]: FAILED! => {
    ansible_facts":
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    "changed": false
    "module_stderr":
        Traceback (most recent call last):
          File "master:/Users/dsgnr/.virtualenvs/ansible/lib/python3.7/site-packages/ansible_mitogen/runner.py", line 975, in _run
            self._run_code(code, mod)
          File "master:/Users/dsgnr/.virtualenvs/ansible/lib/python3.7/site-packages/ansible_mitogen/runner.py", line 939, in _run_code
            exec(code, vars(mod))
          File "master:/Users/dsgnr/.virtualenvs/py3/lib/python3.7/site-packages/ansible/modules/dnf.py", line 1330, in <module>
          File "master:/Users/dsgnr/.virtualenvs/py3/lib/python3.7/site-packages/ansible/modules/dnf.py", line 1319, in main
          File "master:/Users/dsgnr/.virtualenvs/py3/lib/python3.7/site-packages/ansible/modules/dnf.py", line 1288, in run
          File "master:/Users/dsgnr/.virtualenvs/py3/lib/python3.7/site-packages/ansible/modules/dnf.py", line 620, in _base
          File "/usr/lib/python3.6/site-packages/dnf/base.py", line 300, in init_plugins
            self._plugins._load(self.conf, disabled_glob, enable_plugins)
          File "/usr/lib/python3.6/site-packages/dnf/plugin.py", line 131, in _load
            raise RuntimeError("load_plugins() called twice")
        RuntimeError: load_plugins() called twice
    "module_stdout": ""
    "msg":
        MODULE FAILURE
        See stdout/stderr for the exact error
    "rc": 1
}

+- name: Install package
+  package:
+    name: "{{ package }}"
+    state: present

However, using the individual modules works fine;

+- name: Install package for Ubuntu
+  apt:
+    name: "{{ package }}"
+    state: present
+  when: ansible_os_family == "Ubuntu"
+
+- name: Install package for RHEL up to version 7
+  yum:
+    name: "{{ package }}"
+    state: present
+  when: ansible_os_family == "RedHat" and ansible_distribution_major_version|int <= 7
+
+- name: Install package for RHEL 8
+  dnf:
+    name: "{{ package }}"
+    state: present
+  when: ansible_os_family == "RedHat" and ansible_distribution_major_version|int == 8

• Which version of Ansible are you running?
  Ansible 2.10.5

• Is your version of Ansible patched in any way?

• No

• Are you running with any custom modules, or module_utils loaded?
  No

• Have you tried the latest master version from Git?
  Running v0.3.0-rc.0

• Do you have some idea of what the underlying problem may be?

• Mention your host and target OS and versions
  Red Hat Enterprise Linux release 8.3 (Ootpa)

• Mention your host and target Python versions

# /usr/libexec/platform-python
Python 3.6.8 (default, Aug 18 2020, 08:33:21)
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

EDIT (AW): Formatted error and traceback

[wouterhummelink]

This also occurs if the dnf/yum module is used more than once in the same playbook.
I think because mitogen always keeps the process alive that state from a previous call to libdnf is left around, causing subsequent attempts to initialize libdnf to fail. Some code would need te added to clean up after the dnf module.

ansible_facts: {}
  module_stderr: |-
    Traceback (most recent call last):
      File "master:/home/wouterhummelink/.local/lib/python3.9/site-packages/ansible_mitogen/runner.py", line 975, in _run
        self._run_code(code, mod)
      File "master:/home/wouterhummelink/.local/lib/python3.9/site-packages/ansible_mitogen/runner.py", line 939, in _run_code
        exec(code, vars(mod))
      File "master:/home/wouterhummelink/.local/lib/python3.9/site-packages/ansible/modules/dnf.py", line 1330, in <module>
      File "master:/home/wouterhummelink/.local/lib/python3.9/site-packages/ansible/modules/dnf.py", line 1319, in main
      File "master:/home/wouterhummelink/.local/lib/python3.9/site-packages/ansible/modules/dnf.py", line 1288, in run
      File "master:/home/wouterhummelink/.local/lib/python3.9/site-packages/ansible/modules/dnf.py", line 620, in _base
      File "/usr/lib/python3.6/site-packages/dnf/base.py", line 300, in init_plugins
        self._plugins._load(self.conf, disabled_glob, enable_plugins)
      File "/usr/lib/python3.6/site-packages/dnf/plugin.py", line 131, in _load
        raise RuntimeError("load_plugins() called twice")
    RuntimeError: load_plugins() called twice
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

[wouterhummelink]

This could be solved by calling into dnf.plugin.Plugins._unload() but I'm not quite sure where such cleanup should go... ansible or mitogen

[wouterhummelink]

I've locally patched dnf.py in ansible to call _unload() just before the call to init_plugins() that appears to successfully work around this issue.

[wouterhummelink]

@moreati The ansible project appears to be unwilling to fix this on their end.

[moreati]

thank you, I don't think this will make it into the next release candidate. Possibly the one after.

[fauust]

Hi, I can confirm this on 3 centos8 and a rhel8.
Any help you might need in debugging this, just ping me (I am very happy that mitogen finally is usable with ansible 2.10)!

[Aethylred]

Another confirmation for CentOS8

[Madic-]

As workaround I added the following to the task:

  vars:
    mitogen_task_isolation: fork

Example:

- name: Podman | Installing podman....
  package:
    name:
      - podman
    state: present
  vars:
    mitogen_task_isolation: fork

[fauust]

@Madic- workaround works for me too.

[JiffsMaverick]

I can confirm this bug for Oracle Linux 8. Mitogen 0.3.0-rc1 and ansible-base 2.10.5.

[moreati]

#816 attempting to reproduce as part of #816

[moreati]

I've reproduced this in https://github.com/moreati/mitogen/tree/issue776, after checking out the branch you can use Tox to run these tests, e.g.

ANSIBLE_TAGS=issue_776 tox -e py39-mode_ansible-distros_centos8

[annttu]

Adding ansible.legacy.dnf to ALWAYS_FORK_MODULES list in ansible_mitogen/planner.py seems to fix this issue with yum.

+++ ./ansible_mitogen/planner.py	2021-03-26 14:18:56.000000000 +0200
@@ -321,6 +321,7 @@
     ALWAYS_FORK_MODULES = frozenset([
         'dnf',  # issue #280; py-dnf/hawkey need therapy
         'firewalld',  # issue #570: ansible module_utils caches dbus conn
+        'ansible.legacy.dnf',  # issue #776
     ])

     def should_fork(self):

[pun-ky]

@annttu yep, I could confirm that it works with RHEL8/Python3 and Ansible 2.10 .

thanks for sharing this fix! guys could you somehow release the fix even as an unofficial version?

right now I need to reference fork in my build but it just would be nice to reference your awesome project ;)

btw maybe this ALWAYS_FORK_MODULES param should be a configurable option from ansible.cfg? :)

[DaVince]

Adding ansible.legacy.dnf to ALWAYS_FORK_MODULES list in ansible_mitogen/planner.py seems to fix this issue with yum.

+++ ./ansible_mitogen/planner.py	2021-03-26 14:18:56.000000000 +0200
@@ -321,6 +321,7 @@
     ALWAYS_FORK_MODULES = frozenset([
         'dnf',  # issue #280; py-dnf/hawkey need therapy
         'firewalld',  # issue #570: ansible module_utils caches dbus conn
+        'ansible.legacy.dnf',  # issue #776
     ])

     def should_fork(self):

Can confirm this fixes it on Ansible 2.10.7 on MacOS as well, for both package and dnf.

[bradh352]

If you want to easily install the PR #845 that contains this fix without manually patching, you can run something like:
pip install git+https://github.com/mitogen-hq/mitogen.git@refs/pull/845/merge

[Nothing4You]

if you look at the ALWAYS_FORK_MODULES list you can see that dnf already was in there for a different reason (#280), it just wasn't migrated to the new naming scheme i guess?

[momiji]

Hello, this is a must have!
Please release version 0.3.0 in pypi with this patch, it really a required on redhat8.
Good job !