azcollection requirements are so old, which azure-cli package will work with them?

[howardjones]

I'm trying to build a virtualenv for awx/tower that contains azcollection and also the azure cli (since playbooks still need that to do things that the azcollection can't, like creating Service Principals)

However, the current azcli uses a very short script wrapper over python modules python3 -Im azure.cli $@

The versions (2.11.1 vs 2.21.0 currently) installed by azcollection are not structured this way:

/var/lib/awx/venv/local/bin/python3: No module named azure.cli.__main__; 'azure.cli' is a package and cannot be directly executed

Which version of azure-cli will actually work with these old azure modules, until you move everything to the newer modules?

[howardjones]

Even cooler - there doesn't appear to be ANY version of azure-cli that matches the requirements for azcollection!
azcollection wants azure-mgmt-batch==5.0.1

azure-cli-2.0.68 wants 6.0
azure-cli-2.0.67 wants 4.0.3

(both of which are nearly two years old at this point)

[Fred-sun]

@howardjones Thanks for your submit this issue! We are preparing to update all depend packages! once upgrade, I will add comment in here! Thank you very much!

[greystitch]

@howardjones Thanks for your submit this issue! We are preparing to update all depend packages! once upgrade, I will add comment in here! Thank you very much!

Does this mean that azcollection will be having the same python module dependencies as azure-cli defined somewhere in recommended requirements file?

[howardjones]

I was wondering this too. At least the same as some version of azure-cli. Currently with the new pip version resolver, pip will just churn and churn forever if you ask it for the azcollection dependencies and azure-cli, since they don't align.

[greystitch]

@howardjones Absolutely! And, as for now, the only solution to combine azcollection and azure-cli on the same machine (or container in my case) is to install CLI binary as package. But this workaround cannot be considered as good, because installation size is about 800MB and contains many other unwanted stuff, like another python installation, its libraries and hundreds of pyc files... Well, I'm really looking forward to see Ansible related functionality aligned with azure-cli, which might lift up the configuration management to the next level.

[paultaiton]

I'm not involved in the update that @Fred-sun mentioned, which is the way forward. However I wanted to know if using two virtual environments, one for azcollection, one for azure-cli, would be possible and mitigate the issue for you guys @howardjones and @greystitch in the meantime? There's no hard technical requirement that they both be installed together since azcollection does not directly call the cli. It is what I have done on my machines, although I do azcollection requirements to a venv and azure-cli to the default --user paths. You can then set "ansible_python_interpreter" to the venv during plays that require azcollection, or if using AWX / Tower you can follow those specific instructions for making virtual environments without a need for azure-cli.

[howardjones]

This is what I'm doing currently, but it's not very efficient... the second venv is another 650MB inside the container once azure-cli is installed with pip.

[paultaiton]

This is what I'm doing currently, but it's not very efficient... the second venv is another 650MB inside the container once azure-cli is installed with pip.

Gotcha. Yeah, not an ideal situation to duplicate so many packages for version differences.

[erikhjensen]

@paultaiton Hi, I'm being asked to look into this for another team where they do use the Azure Modules in some Tasks and they use the Azure CLI in others. When you commented that you're splitting up the virtual environments are you calling into other job-templates each with suitable Venvs from a master job-template? From what I gather, the team I'm working with is executing their job directly on the Ansible Worker and is looking for a venv with both capabilities.

[paultaiton]

@erikhjensen
I do not call Azure CLI from Ansible at any time, and only use the azcollection modules, so my experience there is limited. I have azure cli installed to my user's default python path, and a dedicated .venv for azcollection. Any az command I run is done manually from a regular bash session.

In theory I believe you should be able to use the fully qualified path to the az executable, and it will pick the right python interpreter environment. That's probably the easiest way to get it working with both.

The other option I can think of is to change the "ansible_python_interpreter" on a per play basis, and keep your azcollection tasks and "az" tasks in separate plays.

[howardjones]

@erikhjensen I am doing this with awx at the moment - two extra venvs, one for azcollection (and whatever else we need), and a second for azcli. I also change the az "binary" shellscript to explicitly use the right venv's python instead of the discovery thing it does otherwise:

#!/usr/bin/env bash
AZ_INSTALLER=DEB /var/lib/awx/venv/azlocal/bin/python3 -Im azure.cli "$@"

so the job template runs in the azcollection venv, but it calls binaries in the other one which will use their local modules, since they are using that python.

If it's for local use, it's worth noting that the package-installed (i.e. apt not pip-installed) azure-cli on Ubuntu from the MS PPA does all this for you in /opt/az. Presumably it does the same in other packages they distribute.

[erikhjensen]

Hi @Fred-sun, how goes the planning of the dependency updates? I only ask because we had to back out a change which attempted to work around this limitation and I suspect our team is not quite as resourceful as @howardjones as our yum-installed Azure CLI was failing when called from within the awx virtual environments. Fwiw, this is the error we saw within our ansible_az virtual environment.

"stderr": "/var/lib/awx/venv/ansible_az/bin/python3: No module named azure.cli.__main__; 'azure.cli' is a package and cannot be directly executed",