salt-ssh not working on remote hosts with python versions >3.8 #61276

HappyBuzzCut · 2021-11-21T00:51:47Z

Description of Issue

When trying to highstate a new Fedora 25 machine with Python 3.11 (currently latest in repo), I get the following error:
"ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib64/python3.10/collections/init.py)"

Setup

N/A - Errors with salt using Python module. I have no custom python scripts

Steps to Reproduce Issue

[root@morton ~]
# salt-ssh -i comms state.highstate
comms:
    ----------
    _error:
        Failed to return clean data
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.root_5a678c_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.root_5a678c_salt/pyall/salt/scripts.py", line 426, in salt_call
            import salt.cli.call
          File "/var/tmp/.root_5a678c_salt/pyall/salt/cli/call.py", line 3, in <module>
            import salt.cli.caller
          File "/var/tmp/.root_5a678c_salt/pyall/salt/cli/caller.py", line 15, in <module>
            import salt.minion
          File "/var/tmp/.root_5a678c_salt/pyall/salt/minion.py", line 23, in <module>
            import salt.client
          File "/var/tmp/.root_5a678c_salt/pyall/salt/client/__init__.py", line 40, in <module>
            import salt.utils.minions
          File "/var/tmp/.root_5a678c_salt/pyall/salt/utils/minions.py", line 12, in <module>
            import salt.auth.ldap
          File "/var/tmp/.root_5a678c_salt/pyall/salt/auth/ldap.py", line 11, in <module>
            from jinja2 import Environment
          File "/var/tmp/.root_5a678c_salt/pyall/jinja2/__init__.py", line 6, in <module>
            from markupsafe import escape
          File "/var/tmp/.root_5a678c_salt/py3/markupsafe/__init__.py", line 13, in <module>
            from collections import Mapping
        ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib64/python3.10/collections/__init__.py)
    stdout:

Versions Report

minion:
None - Using salt-ssh to highstate a host over ssh. Remote host running fedora 35

master:

Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 18 2021, 10:07:16)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

System Versions:
          dist: oracle 7.9
        locale: UTF-8
       machine: x86_64
       release: 5.4.17-2136.300.7.el7uek.x86_64
        system: Linux
       version: Oracle Linux Server 7.9

The text was updated successfully, but these errors were encountered:

welcome · 2021-11-21T00:51:49Z

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

OrangeDog · 2021-11-22T10:36:20Z

What version of markupsafe is that? It may simply need updating.

HappyBuzzCut · 2021-11-23T01:59:00Z

What version of markupsafe is that? It may simply need updating.

So in an effort to get this working, I have downgraded salt-ssh to version 3003.3 but the issue remains. I have made sure the package you mentioned was upgraded and I even installed it on remote machine. Here are the versions

salt master

Name         : python36-markupsafe
Version      : 0.23
Release      : 4.el7
Arch         : x86_64
Size         : 99 k
Source       : python3-markupsafe-0.23-4.el7.src.rpm
Repo         : @System
Summary      : Implements a XML/HTML/XHTML Markup safe string for Python
URL          : http://pypi.python.org/pypi/MarkupSafe
License      : BSD
Description  : A library for safe markup escaping.

fedora 35 machine

Installed Packages
Name         : python3-markupsafe
Version      : 2.0.0
Release      : 2.fc35
Architecture : x86_64
Size         : 48 k
Source       : python-markupsafe-2.0.0-2.fc35.src.rpm
Repository   : @System
From repo    : fedora
Summary      : Implements a XML/HTML/XHTML Markup safe string for Python 3
URL          : https://pypi.org/project/MarkupSafe/
License      : BSD
Description  : A library for safe markup escaping. Python 3 version.

OrangeDog · 2021-11-23T09:58:45Z

from collections import Mapping was removed in version 1.1 of markupsafe.
Therefore it's the version from the master that its is using, which is far too old.

Salt-SSH should be using the packages it finds on the minion though, if they're installed.
You say you're trying to run this under Python 3.11, but the stacktrace says /usr/lib64/python3.10/.
Have you got the PATH/set_path configured correctly?

HappyBuzzCut · 2021-11-23T23:38:53Z

from collections import Mapping was removed in version 1.1 of markupsafe. Therefore it's the version from the master that its is using, which is far too old.

Okay the Salt folks need to get this package upgraded on their repo because as of now version 0.23-4 is in their rhel7 repo for versions 3003 and 3004

Salt-SSH should be using the packages it finds on the minion though, if they're installed. You say you're trying to run this under Python 3.11, but the stacktrace says /usr/lib64/python3.10/. Have you got the PATH/set_path configured correctly?

It is the same error for both the versions. 3.10 is the default and I just symlink /usr/bin/python3 to the ver I want. This of course is all in vain since like you said the master uses an older markupsafe

Interesting to note, my internal fedora 35 machine works just great with salt-minion but salt-ssh is problematic it seems with these mismatches

OrangeDog · 2021-11-24T10:09:47Z

The master is fine to be using an older markupsafe. There's nothing wrong there; it's the correct package for el7.

The problem is either:
a. You have not installed markupsafe correctly on the minion for the python that salt-ssh is configured to use
b. There is a bug in salt-ssh where it's not picking up dependencies already available on the minion

Given you've been switching around with dev versions of Python, my guess would be a. If you can rule that out then it is b.

HappyBuzzCut · 2021-11-25T02:45:23Z

The master is fine to be using an older markupsafe. There's nothing wrong there; it's the correct package for el7.

The problem is either: a. You have not installed markupsafe correctly on the minion for the python that salt-ssh is configured to use b. There is a bug in salt-ssh where it's not picking up dependencies already available on the minion

Given you've been switching around with dev versions of Python, my guess would be a. If you can rule that out then it is b.

'a' can be ruled out:

# dnf info python3-markupsafe.x86_64
Last metadata expiration check: 2:28:48 ago on Wed 24 Nov 2021 06:15:06 PM CST.
Installed Packages
Name         : python3-markupsafe
Version      : 2.0.0
Release      : 2.fc35
Architecture : x86_64
Size         : 48 k
Source       : python-markupsafe-2.0.0-2.fc35.src.rpm
Repository   : @System
From repo    : fedora
Summary      : Implements a XML/HTML/XHTML Markup safe string for Python 3
URL          : https://pypi.org/project/MarkupSafe/
License      : BSD
Description  : A library for safe markup escaping. Python 3 version.

# rpm -ql python3-markupsafe
/usr/lib/.build-id
/usr/lib/.build-id/8d
/usr/lib/.build-id/8d/29e1de80aecbb909f342cd7e34ea212b188fd8
/usr/lib64/python3.10/site-packages/MarkupSafe-2.0.0-py3.10.egg-info
/usr/lib64/python3.10/site-packages/MarkupSafe-2.0.0-py3.10.egg-info/PKG-INFO
/usr/lib64/python3.10/site-packages/MarkupSafe-2.0.0-py3.10.egg-info/SOURCES.txt
/usr/lib64/python3.10/site-packages/MarkupSafe-2.0.0-py3.10.egg-info/dependency_links.txt
/usr/lib64/python3.10/site-packages/MarkupSafe-2.0.0-py3.10.egg-info/top_level.txt
/usr/lib64/python3.10/site-packages/markupsafe
/usr/lib64/python3.10/site-packages/markupsafe/__init__.py
/usr/lib64/python3.10/site-packages/markupsafe/__pycache__
/usr/lib64/python3.10/site-packages/markupsafe/__pycache__/__init__.cpython-310.opt-1.pyc
/usr/lib64/python3.10/site-packages/markupsafe/__pycache__/__init__.cpython-310.pyc
/usr/lib64/python3.10/site-packages/markupsafe/__pycache__/_native.cpython-310.opt-1.pyc
/usr/lib64/python3.10/site-packages/markupsafe/__pycache__/_native.cpython-310.pyc
/usr/lib64/python3.10/site-packages/markupsafe/_native.py
/usr/lib64/python3.10/site-packages/markupsafe/_speedups.cpython-310-x86_64-linux-gnu.so
/usr/lib64/python3.10/site-packages/markupsafe/_speedups.pyi
/usr/lib64/python3.10/site-packages/markupsafe/py.typed
/usr/share/doc/python3-markupsafe
/usr/share/doc/python3-markupsafe/CHANGES.rst
/usr/share/doc/python3-markupsafe/README.rst
/usr/share/licenses/python3-markupsafe
/usr/share/licenses/python3-markupsafe/LICENSE.rst

# python3 -V
Python 3.10.0

Pip had markupsafe 2.0.1 installed. I even tried removing that to strictly use the rpm version. still failed...

# pip3.10 list|grep MarkupSafe
MarkupSafe         2.0.0

OrangeDog · 2021-11-25T10:12:56Z

'a' cannot be ruled out until you establish which python installation the minion is using.

Have you got the PATH/set_path configured correctly?

The python binary which that uses should include /usr/lib64/python3.10/site-packages in python -c 'import sys; print(sys.path)'

The original stacktrace had /usr/lib64/python3.10, but you've been changing things since then.

HappyBuzzCut · 2021-11-26T06:00:36Z

'a' cannot be ruled out until you establish which python installation the minion is using.

Have you got the PATH/set_path configured correctly?

The python binary which that uses should include /usr/lib64/python3.10/site-packages in python -c 'import sys; print(sys.path)'

# salt-ssh -i comms -r "python3 -c 'import sys; print(sys.path)'"
comms:
    ----------
    retcode:
        0
    stderr:
    stdout:
        ['', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/local/lib64/python3.10/site-packages', '/usr/local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/site-packages/cloud_init-21.3-py3.10.egg', '/usr/lib/python3.10/site-packages', '/usr/lib64/python3.10/site-packages']

# salt-ssh -i comms -r "ls /usr/lib64/python3.10/site-packages|grep MarkupSafe"
comms:
    ----------
    retcode:
        0
    stderr:
    stdout:
        MarkupSafe-2.0.0-py3.10.egg-info

The original stacktrace had /usr/lib64/python3.10, but you've been changing things since then.

Nothing has changed.....

# salt-ssh -i comms test.ping
comms:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.root_5a678c_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.root_5a678c_salt/pyall/salt/scripts.py", line 426, in salt_call
            import salt.cli.call
          File "/var/tmp/.root_5a678c_salt/pyall/salt/cli/call.py", line 3, in <module>
            import salt.cli.caller
          File "/var/tmp/.root_5a678c_salt/pyall/salt/cli/caller.py", line 15, in <module>
            import salt.minion
          File "/var/tmp/.root_5a678c_salt/pyall/salt/minion.py", line 23, in <module>
            import salt.client
          File "/var/tmp/.root_5a678c_salt/pyall/salt/client/__init__.py", line 40, in <module>
            import salt.utils.minions
          File "/var/tmp/.root_5a678c_salt/pyall/salt/utils/minions.py", line 12, in <module>
            import salt.auth.ldap
          File "/var/tmp/.root_5a678c_salt/pyall/salt/auth/ldap.py", line 11, in <module>
            from jinja2 import Environment
          File "/var/tmp/.root_5a678c_salt/pyall/jinja2/__init__.py", line 6, in <module>
            from markupsafe import escape
          File "/var/tmp/.root_5a678c_salt/py3/markupsafe/__init__.py", line 13, in <module>
            from collections import Mapping
        ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib64/python3.10/collections/__init__.py)
    stdout:

OrangeDog · 2021-11-26T10:19:08Z

Yep, that looks like it's not picking up installed packages corectly then.

Installing a newer markupsafe (e.g. via yum or pip) on the master should work around it.

ESNewmanium · 2023-01-23T18:20:57Z

@HappyBuzzCut I was able to sidestep this by pip installing MarkupSafe 2.0.0 on my salt-ssh master which runs py 3.6. Do be sure to clear out the /var/cache/salt/master files of your salt-ssh master though. Didn't work until I did that.

miukumac · 2023-07-19T10:36:39Z

This problem is still open at least on a mixed openSUSE setup where the Master has Python 3.6.15 with Salt Master 3006.0 and some of the nodes with openSUSE Tumbleweed running on Python 3.11.4 and naturally on Minion 3006.0.

Installing a newer markupsafe did not triage the issue or perhaps it's because of the distribution packages.

OrangeDog · 2023-07-19T10:41:08Z

did not triage the issue

"triage" means to determine the severity and priority of something

miukumac · 2023-07-19T12:27:25Z

"triage" means to determine the severity and priority of something

Sorry, English isn't my primary language so sometimes I have to rely on translators and the verb we use for fixing translated to triage for some reason.

Anyway unfortunately upgrading the python module did not fix the issue for me.

matwey · 2023-08-26T16:36:26Z

The master is fine to be using an older markupsafe. There's nothing wrong there; it's the correct package for el7.

The problem is either: a. You have not installed markupsafe correctly on the minion for the python that salt-ssh is configured to use b. There is a bug in salt-ssh where it's not picking up dependencies already available on the minion

Do I understand correctly, that salt-ssh is able to use python dependencies from the minion side instead of providing it within the tarball?

Given you've been switching around with dev versions of Python, my guess would be a. If you can rule that out then it is b.

matwey · 2023-08-26T17:08:58Z

@OrangeDog from what I see from the sources, markupsafe is unconditionally added to the thin tarball.
salt-call just prepends sys.path with bundled paths, so the target python has no chance to use installed markupsafe.

github-actions bot added the needs-triage label Nov 21, 2021

github-actions bot assigned frogunder Nov 21, 2021

OrangeDog added dependency underlying Salt dependency issue info-needed waiting for more info Salt-SSH labels Nov 22, 2021

OrangeDog added severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around and removed info-needed waiting for more info needs-triage dependency underlying Salt dependency issue labels Nov 26, 2021

OrangeDog added this to the Approved milestone Nov 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

salt-ssh not working on remote hosts with python versions >3.8 #61276

salt-ssh not working on remote hosts with python versions >3.8 #61276

HappyBuzzCut commented Nov 21, 2021 •

edited

Loading

welcome bot commented Nov 21, 2021

OrangeDog commented Nov 22, 2021

HappyBuzzCut commented Nov 23, 2021

OrangeDog commented Nov 23, 2021 •

edited

Loading

HappyBuzzCut commented Nov 23, 2021 •

edited

Loading

OrangeDog commented Nov 24, 2021

HappyBuzzCut commented Nov 25, 2021 •

edited

Loading

OrangeDog commented Nov 25, 2021 •

edited

Loading

HappyBuzzCut commented Nov 26, 2021 •

edited

Loading

OrangeDog commented Nov 26, 2021

ESNewmanium commented Jan 23, 2023

miukumac commented Jul 19, 2023

OrangeDog commented Jul 19, 2023

miukumac commented Jul 19, 2023 •

edited

Loading

matwey commented Aug 26, 2023

matwey commented Aug 26, 2023

salt-ssh not working on remote hosts with python versions >3.8 #61276

salt-ssh not working on remote hosts with python versions >3.8 #61276

Comments

HappyBuzzCut commented Nov 21, 2021 • edited Loading

Description of Issue

Setup

Steps to Reproduce Issue

Versions Report

welcome bot commented Nov 21, 2021

OrangeDog commented Nov 22, 2021

HappyBuzzCut commented Nov 23, 2021

OrangeDog commented Nov 23, 2021 • edited Loading

HappyBuzzCut commented Nov 23, 2021 • edited Loading

OrangeDog commented Nov 24, 2021

HappyBuzzCut commented Nov 25, 2021 • edited Loading

OrangeDog commented Nov 25, 2021 • edited Loading

HappyBuzzCut commented Nov 26, 2021 • edited Loading

OrangeDog commented Nov 26, 2021

ESNewmanium commented Jan 23, 2023

miukumac commented Jul 19, 2023

OrangeDog commented Jul 19, 2023

miukumac commented Jul 19, 2023 • edited Loading

matwey commented Aug 26, 2023

matwey commented Aug 26, 2023

HappyBuzzCut commented Nov 21, 2021 •

edited

Loading

OrangeDog commented Nov 23, 2021 •

edited

Loading

HappyBuzzCut commented Nov 23, 2021 •

edited

Loading

HappyBuzzCut commented Nov 25, 2021 •

edited

Loading

OrangeDog commented Nov 25, 2021 •

edited

Loading

HappyBuzzCut commented Nov 26, 2021 •

edited

Loading

miukumac commented Jul 19, 2023 •

edited

Loading