[BUG] salt-master unable to connect gitfs of ext_pillar #57121

rvvliet · 2020-05-07T10:25:10Z

Description
I'am rebuilding my salt-master and configuring gitfs and ext_pillar git but gitfs is unable to connect to the git repo.

The ssh keys are working and have been tested using ssh client as does a git clone on the server with these keys.

The salt master config has been used on the original server without problems, only the keys are renewed, i have tested ED25519 and RSA type keys.

Setup

fileserver_backend:
  - roots
  - gitfs

gitfs_provider: pygit2

gitfs_remotes:
  - git@git.mydomain.coml:saltstack/salt-gitfs-remote-enviroments.git:
    - pubkey: /usr/local/etc/salt/ssh_keys/saltstack-repo_id_rsa.pub
    - privkey: /usr/local/etc/salt/ssh_keys/saltstack-repo_id_rsa
    - root: states
    - saltenv:
      - base:
        - ref: master

ext_pillar:
  - git:
    - master git@git.newimage.nl:saltstack/salt-gitfs-remote-enviroments.git:
      - pubkey: /usr/local/etc/salt/ssh_keys/saltstack-repo_id_rsa.pub
      - privkey: /usr/local/etc/salt/ssh_keys/saltstack-repo_id_rsa
      - root: pillar
      - env: base

Steps to Reproduce the behavior
Start the salt-master , maybe in debug mode and check log of output.

Expected behavior
Connection to Git

Screenshots

[ERROR   ] Error occurred fetching gitfs remote 'git@git.mydomain.com:saltstack/salt-gitfs-remote-enviroments.git': Failed to retrieve list of SSH authentication methods: Failed getting response
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/salt/utils/gitfs.py", line 1741, in _fetch
    fetch_results = origin.fetch(**fetch_kwargs)
  File "/usr/local/lib/python3.7/site-packages/pygit2/remote.py", line 423, in fetch
    check_error(err)
  File "/usr/local/lib/python3.7/site-packages/pygit2/errors.py", line 61, in check_error
    raise GitError(message)
_pygit2.GitError: Failed to retrieve list of SSH authentication methods: Failed getting response

[DEBUG   ] Set update lock for git_pillar remote 'master git@git.newimage.nl:saltstack/salt-gitfs-remote-enviroments.git'
[DEBUG   ] Fetching git_pillar remote 'master git@git.newimage.nl:saltstack/salt-gitfs-remote-enviroments.git'
[ERROR   ] Error occurred fetching git_pillar remote 'master git@git.newimage.nl:saltstack/salt-gitfs-remote-enviroments.git': Failed to retrieve list of SSH authentication methods: Failed getting response
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/salt/utils/gitfs.py", line 1741, in _fetch
    fetch_results = origin.fetch(**fetch_kwargs)
  File "/usr/local/lib/python3.7/site-packages/pygit2/remote.py", line 423, in fetch
    check_error(err)
  File "/usr/local/lib/python3.7/site-packages/pygit2/errors.py", line 61, in check_error
    raise GitError(message)
_pygit2.GitError: Failed to retrieve list of SSH authentication methods: Failed getting response
[DEBUG   ] Removed update lock for git_pillar remote 'master git@git.mydomain.com:saltstack/salt-gitfs-remote-enviroments.git'

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2019.2.4
 
Dependency Versions:
           cffi: 1.14.0
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: 1.0.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 1.2.0
         Python: 3.7.7 (default, Apr 12 2020, 21:25:06)
   python-gnupg: Not Installed
         PyYAML: 5.3.1
          PyZMQ: 19.0.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1
 
System Versions:
           dist:   
         locale: UTF-8
        machine: amd64
        release: 12.1-RELEASE-p3
         system: FreeBSD
        version: Not Installed

Used relevant OS Packages
git-lite-2.26.2                Distributed source code management tool (lite package)
libgit2-1.0.0                  Portable, pure C implementation of the Git core
py37-pygit2-1.2.0_1            Python bindings for libgit2
libssh2-1.8.2,3                Library implementing the SSH2 protocol

Additional context

The text was updated successfully, but these errors were encountered:

waynew · 2020-05-07T20:59:30Z

Looks like that error is from here: https://github.com/libgit2/libgit2/blob/master/src/transports/ssh.c#L806

Are you sure that the ssh keys are really the ones being used? Use these instructions on StackOverflow to double check.

rvvliet · 2020-05-08T11:07:05Z

Yes i found the reference to the libgit2 error also, witch pointed to key problems, or possibly a unexpected authentication method line.

I have teste with the keys in the location above, but also with the keys in de /root/.ssh/ location none of this helps.

The SSH Server response with below when salt of pygit2 tries to connect. (IP replaced)
Includes are also 2 test python scripts. these have the same result.

May  8 12:23:43 git sshd[17289]: Postponed publickey for git from 1.1.1.1 port 31150 ssh2 [preauth]
May  8 12:23:43 git sshd[17289]: message repeated 2 times: [ Postponed publickey for git from 1.1.1.1 port 31150 ssh2 [preauth]]
May  8 12:23:43 git sshd[17289]: error: maximum authentication attempts exceeded for git from 1.1.1.1 port 31150 ssh2 [preauth]
May  8 12:23:43 git sshd[17289]: Disconnecting: Too many authentication failures [preauth]

test1.py

import pygit2 

class MyRemoteCallbacks(pygit2.RemoteCallbacks):
    def credentials(self, url, username_from_url, allowed_types):
        if allowed_types & pygit2.credentials.GIT_CREDTYPE_USERNAME:
            return pygit2.Username("git")
        elif allowed_types & pygit2.credentials.GIT_CREDTYPE_SSH_KEY:
            return pygit2.Keypair("git", "/root/.ssh/saltstack-repo_id_rsa.pub", "/root/.ssh/saltstack-repo_id_rsa", "")
        else:
            return None

print("Cloning over ssh")
pygit2.clone_repository("ssh://git@git.DOMAIN.nl/saltstack/", "salt-gitfs-remote-enviroments.git",callbacks=MyRemoteCallbacks())

test2.py

import pygit2

print("Cloning over ssh with the username in the URL")
keypair = pygit2.Keypair("git", "/root/.ssh/saltstack-repo_id_rsa.pub", "/root/.ssh/saltstack-repo_id_rsa", "")
callbacks = pygit2.RemoteCallbacks(credentials=keypair)

pygit2.clone_repository("ssh://git@git.DOMAIN.nl/saltstack/", "salt-gitfs-remote-enviroments.git", callbacks=callbacks)

I check the python scripts with truss and that shows the key files being opend.

open("/root/.ssh/saltstack-repo_id_rsa",O_RDONLY,0666) = 4 (0x4)
fstat(4,{ mode=-rw------- ,inode=134332,size=1843,blksize=4096 }) = 0 (0x0)
read(4,"-----BEGIN OPENSSH PRIVATE KEY--"...,4096) = 1843 (0x733)
read(4,0x80136c000,4096)			 = 0 (0x0)
close(4)					 = 0 (0x0)
>>>
open("/root/.ssh/saltstack-repo_id_rsa",O_RDONLY,0666) = 4 (0x4)
fstat(4,{ mode=-rw------- ,inode=134332,size=1843,blksize=4096 }) = 0 (0x0)
read(4,"-----BEGIN OPENSSH PRIVATE KEY--"...,4096) = 1843 (0x733)
read(4,0x80136c000,4096)			 = 0 (0x0)
close(4)					 = 0 (0x0)

Permissions /root/.ssh/

-rw-------  1 root  wheel   411 May  7 09:42 saltstack-repo_id_ed25519
-rw-r--r--  1 root  wheel   102 May  7 09:42 saltstack-repo_id_ed25519.pub
-rw-------  1 root  wheel  1843 May  7 09:42 saltstack-repo_id_rsa
-rw-r--r--  1 root  wheel   410 May  7 09:42 saltstack-repo_id_rsa.pub

Permissions /usr/local/etc/salt/ssh_keys/

-rw-------  1 root  wheel   411 May  6 19:10 saltstack-repo_id_ed25519
-rw-r--r--  1 root  wheel   102 May  6 19:10 saltstack-repo_id_ed25519.pub
-rw-------  1 root  wheel  1843 May  7 09:05 saltstack-repo_id_rsa
-rw-r--r--  1 root  wheel   410 May  7 09:05 saltstack-repo_id_rsa.pub

As said, using a git clone as root is using the key and clones without problems.
I'll keep testing, but suggestions are welcome.

rvvliet · 2020-05-13T08:13:02Z

After another 2 days of trying i decided to switch to https:// for gitfs and ext_pillar, so far this works.
So far https:// is working for me.

I still do not know if this is a configuration error or salt/gitfs2/libssh2 error.
i no one else has this problem feel free to close this.

Thanks for the support.

graphicore · 2020-06-04T13:52:18Z

@rvvliet I basically have the same issue (but trying to fetch), using pygit2, I found this issue googling for the error message. I managed to use the GIT_CREDTYPE_SSH_KEY method with the underdocumented pygit2.KeypairFromAgent Keypair Constructor, after staring into pygit2 and libgit2 sources for a while. It uses ssh-agent deep down in libssh2 I think. Thus if your key is in the output of $ ssh-add -l you may have chances to use ssh authentication as well.

From your example, the relevant part would look like:

class MyRemoteCallbacks(pygit2.RemoteCallbacks):
    def credentials(self, url, username_from_url, allowed_types):
        if allowed_types & pygit2.credentials.GIT_CREDTYPE_USERNAME:
            return pygit2.Username("git")
        elif allowed_types & pygit2.credentials.GIT_CREDTYPE_SSH_KEY:
            return pygit2.KeypairFromAgent("git")
        else:
            return None

I'd be interested to hear feedback about this from you. It was a great waste of time for me so far so I'd like to see progress on the matter (and spread some hints in the web for those who follow). Using any Keypair from disk should work.

viq · 2021-09-17T20:52:30Z

FWIW, I just faced the same issue, and after fighting some with it, it went away when I switched from RSA to ED25519 key (in new format, header -----BEGIN OPENSSH PRIVATE KEY-----)

$ salt-call --local --versions-report
Salt Version:
          Salt: 3003.3
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.1
       libgit2: 1.1.1
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: 1.6.1
        Python: 3.9.6 (default, Jun 28 2021, 08:57:49)
  python-gnupg: 0.4.7
        PyYAML: 5.4.1
         PyZMQ: 22.2.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: nixos 21.11 porcupine
        locale: utf-8
       machine: x86_64
       release: 5.10.64
        system: Linux
       version: NixOS 21.11 porcupine

gvecchicert · 2022-04-08T15:01:17Z

HI there,

I had the same issue using gitfs as fileserver backend and github repos.
Starting from 15/03/2022, github.com is not accepting rsa keys anymore (see https://github.blog/2021-09-01-improving-git-protocol-security-github/); libssh2 can handle ECDSA and Ed25519 keys starting from 1.9.0 but ubuntu package is stuck to 1.8.0
My workaround was to upgrade libssh2 from sources:

sudo aptitude purge libssh2-1 -y
sudo aptitude install libssl-dev python3-pygit2 -y
wget https://www.libssh2.org/download/libssh2-1.10.0.tar.gz
tar xvzf libssh2-1.10.0.tar.gz
cd libssh2-1.10.0
sudo ./configure
sudo make
sudo make install
sudo ln -s /usr/local/lib/libssh2.so.1.0.1 /usr/lib/x86_64-linux-gnu/libssh2.so.1

An ECDSA key was succesfully used, Ed25519 seems broken

caseyrichins · 2022-05-12T19:42:41Z

An ECDSA key was successfully used, Ed25519 seems broken

Can confirm that works with ECDSA but not ed25519. I was bitten by this bug this week. Currently running on Debian 11 (Bullseye) with libssh2-1.9.0, seems to be an upstream issue related to libssh2 and ed25519 keys. Prior to changing keys to ECDSA i was unable to use gitfs or ext_pillar keyed authentication.

          Salt: 3004.1
 
Dependency Versions:
          cffi: 1.15.0
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: 3.1.14
        Jinja2: 2.11.3
       libgit2: 1.4.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: 1.9.1
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: 0.4.6
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-14-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

gvecchicert · 2022-05-13T06:59:12Z

Just a little update:
I was able to get everything working even with libssh2-1 ubuntu package, but pygit2 needs to be installed through pip in order to get later version (1.9.1 for me); I can confirm that ed25519 keys are still not working even in this scenario:

Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: 1.15.0
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: 1.4.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: 1.9.1
        Python: 3.8.10 (default, Mar 15 2022, 12:22:08)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.13.0-1017-aws
        system: Linux
       version: Ubuntu 20.04 focal

tvb · 2023-05-03T15:12:46Z

Using the following versions still generates the same error for me which is blocking as Github now kinda requires ed25519 deploy keys:

Salt Version:
          Salt: 3006.0
 
Python Version:
        Python: 3.10.11 (main, Apr 14 2023, 05:57:16) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.15.1
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.6.3
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.12.0
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.11.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: amzn 2 
        locale: utf-8
       machine: x86_64
       release: 5.15.49-linuxkit
        system: Linux
       version: Amazon Linux 2

rvvliet added the Bug broken, incorrect, or confusing behavior label May 7, 2020

waynew added the info-needed waiting for more info label May 7, 2020

waynew added this to the Blocked milestone May 7, 2020

graphicore mentioned this issue Jun 4, 2020

[git-clone-ssh recipe] Failed to retrieve list of SSH authentication methods: Failed getting response libgit2/pygit2#1013

Closed

tvb mentioned this issue May 4, 2023

clone_repository from github.com fails with ed25519 key from v1.7.2 libgit2/pygit2#1180

Closed

lkubb mentioned this issue Nov 2, 2023

[3006.x] salt-ssh pillar fixes #65484

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] salt-master unable to connect gitfs of ext_pillar #57121

[BUG] salt-master unable to connect gitfs of ext_pillar #57121

rvvliet commented May 7, 2020 •

edited

Loading

waynew commented May 7, 2020

rvvliet commented May 8, 2020

rvvliet commented May 13, 2020

graphicore commented Jun 4, 2020

viq commented Sep 17, 2021

gvecchicert commented Apr 8, 2022

caseyrichins commented May 12, 2022 •

edited

Loading

gvecchicert commented May 13, 2022

tvb commented May 3, 2023

[BUG] salt-master unable to connect gitfs of ext_pillar #57121

[BUG] salt-master unable to connect gitfs of ext_pillar #57121

Comments

rvvliet commented May 7, 2020 • edited Loading

waynew commented May 7, 2020

rvvliet commented May 8, 2020

rvvliet commented May 13, 2020

graphicore commented Jun 4, 2020

viq commented Sep 17, 2021

gvecchicert commented Apr 8, 2022

caseyrichins commented May 12, 2022 • edited Loading

gvecchicert commented May 13, 2022

tvb commented May 3, 2023

rvvliet commented May 7, 2020 •

edited

Loading

caseyrichins commented May 12, 2022 •

edited

Loading