Migrate ansible_mitogen.plugins to ansible collections

[ITD27M01]

• Which version of Ansible are you running?

= 2.11

• Is your version of Ansible patched in any way?
  No

• Are you running with any custom modules, or module_utils loaded?
  A lot of modules from different Ansible collections

• Have you tried the latest master version from Git?
  No

• Mention your host and target OS and versions
  Ubuntu 22.04

• Mention your host and target Python versions
  3.8 or 3.10

Hi, thank you for such amazing plugin!

The new convenient distribution format for Ansible plugins/modules is Ansible Collections. Any plugin either inventory/action/lookup/strategy plugin or Ansible-module could be packaged to such collection and distributed to end users via Ansible Galaxy or collection git repo. All the python code for Ansible (except SDK libraries) should be distributed as a collection and the user experience will be as follows:

1. Install the python requirements

# pip3 install mitogen
or
# apt-get install python3-mitogen

2. Install Ansible requirements

$ ansible-galaxy install -r requirements.yml
or
$ ansible-galaxy collection install mitogen-hq.mitogen

3. Use mitogen plugin from collection in ansible.cfg by FQCN

strategy_plugins = mitogen-hq.mitogen.strategy
strategy = mitogen_linear

It dramatically simplifies the usage and environment setup.

The good collections examples are collections for cloud providers such as AWS or OpenStack:

https://github.com/ansible-collections/amazon.aws
https://github.com/openstack/ansible-collections-openstack

[amarao]

It may be worth it. Mitogen itself may resides in module_utils. This will solve problems with paths.

But can strategy plugin resides in a collection?

[ITD27M01]

@amarao Hi! Just a simple check: https://github.com/ITD27M01/ansible-strategy-collection

[amarao]

This is really helpful. I believe, this is natural way for mitogen. But those changes requires actions from core maintainers, because galaxy namespace must be allocated, and upload process should be configured.

[ITD27M01]

Hi @amarao,
I've created an example with mitogen itself: https://github.com/ITD27M01/ansible-mitogen-example. It looks like it is easier than I thought, all the plugin directory would be migrated to the collection: https://github.com/ITD27M01/ansible-mitogen-collection (with only modification of BASE_DIR manipulation, I just removed such)

It is a workaround for me for now, I will create such collection in our private repo.

[amarao]

I've just discussed this idea with colleagues, and we found one additional case to consider.

In some projects we run ansible with mitogen like this:

export ANSIBLE_STRATEGY_PLUGINS=/usr/local/lib/python3.10/dist-packages/ansible_mitogen/plugins/strategy
export ANSIBLE_STRATEGY=mitogen_linear

This allows to have unmodified playbooks to run with mitogen or without. I really like to keep this feature.

How to specify env vars if mitogen is installed as a collection?

[ITD27M01]

@amarao As I understand these variables are helpers for settings. So, the same way as plugins defined in ansible.cfg they could be defined there:

export ANSIBLE_STRATEGY_PLUGINS=mitogen_hq.mitogen.mitogen
export ANSIBLE_STRATEGY=mitogen_hq.mitogen.mitogen_linear

[amarao]

We've just released an alternative collection for Mitogen, which allow to to use it via galaxy (you still need to pip install mitogen, but you no longer need to provide pathes). It's also patches Mitogen requirements for Ansible, allowing to use it with Ansible-core 2.14+.

https://galaxy.ansible.com/serverscom/mitogen

[Z9n2JktHlZDmlhSvqc9X2MmL3BwQG7tk]

Is there a way to exclude pip3 install mitogen step ?

Example. I use my ansible repo in several places:

• on my own workstation;
• on one of production VMs (just to run ansible inside the production local network which is much faster than VPN);
• on Jenkins agents during my applications deployment.

Typical workflow is:

• git clone my-ansible-repo
• run ansible-playbook ...

With pip3 install mitogen step I need to install it in all places which is not so convenient.
And moreover, if I don't want to test anything (running ansible with mitogen or without), it is more convenient for me to set

[defaults]
strategy_plugins = ...
strategy = mitogen_linear

just in ansible.cfg in the root of my ansible repo.
But if somebody who knows nothing about mitogen usage clones this repo he can not run ansible, because of strategy settings in ansible.cfg.
I mean that pip3 install mitogen step is not so obvious.
Is there a way to download all mitogen files with one command, for example, ansible-galaxy install -r requirements.yml ?

Another example: if I include mitogen git repo in my ansible repo as a submodule, I will still need to do just 2 steps - git clone + run ansible. Everything is still consistent: if the repo has an ansible.cfg with mitogen settings, then it contains the mitogen itself too.
But I think downloading dependencies via ansible-galaxy is a more correct way.
What do you think ?

[amarao]

You can vendor mitogen into your repo and install it locally (from the repo).

I really don't want to bring mitogen code under maintenance. I love it, I use it, but I don't dare to peek inside.

(Also, there are many Ansible modules which require you to have specific python libraries: netaddr, openstacksdk, requests, etc), so, realistically, you still need to pip install before running the code.

[kainz]

For those dealing with this until it's solved one way or another, I use a python snippet to autoconfigure ANSIBLE_* envvars for mitogen in a way that works with arbitrary venv installations (assuming you have the venv activated or run through whatever wrapper like poetry or whatnot):

#!/bin/env python3

import sys
import ansible_mitogen.plugins.strategy

print(f"export ANSIBLE_STRATEGY_PLUGINS={sys.modules['ansible_mitogen.plugins.strategy'].__path__[0]} ANSIBLE_STRATEGY=mitogen_linear")