asyncssh.misc.ProtocolError: Too many authentication failures

[asiron]

I have the following problem:
asyncssh.misc.ProtocolError: Too many authentication failures
when using SSH remote with a NAS, but:

• I can login with the following ssh_config (so ssh ws and ssh nas-access both work)

Host ws
  Hostname xx.xx.xx.xx
  User mzurad
  IdentityFile ~/.ssh/id_ws

Host nas-access
  Hostname xx.xx.xx.xx
  User mzurad
  IdentityFile /home/mzurad/.ssh/id_nas

• I have 3 remotes defined in .dvc/config, and ws works but nas or nas-access don't

['remote "ws"']
    url = ssh://ws/home/mzurad/test-storage
['remote "nas"']
    url = ssh://mzurad@xx.xx.xx.xx/share/dvc-storage
['remote "nas-access"']
    url = ssh://nas-access/share/dvc-storage
    max_sessions = 3

• I can create files with my user mzurad on both ws and nas
• I can copy files using scp (scp a nas-access:/share/dvc-storage/b works)
• I tried reducing max_sessions to 3, but didn't help
• I tried setting allow_agent to false, but didn't help
• I tried with gss_auth set to true, but didn't help

Here is my stacktrace, when running dvc push -r nas-access -v

2023-07-19 01:28:45,710 DEBUG: v3.5.1 (pip), CPython 3.10.6 on Linux-5.19.0-46-generic-x86_64-with-glibc2.35
2023-07-19 01:28:45,710 DEBUG: command: /home/mzurad/archived_code/datasets_dvc/.env/bin/dvc push -r nas-access -v
2023-07-19 01:28:46,458 DEBUG: Preparing to transfer data from '/home/mzurad/quick-test-dvc-cache/files/md5' to '/share/dvc-storage/files/md5'
2023-07-19 01:28:46,458 DEBUG: Preparing to collect status from '/share/dvc-storage/files/md5'
2023-07-19 01:28:46,500 DEBUG: Collecting status from '/share/dvc-storage/files/md5'
2023-07-19 01:28:46,501 DEBUG: Querying 2 oids via object_exists
2023-07-19 01:28:46,783 ERROR: unexpected error - Too many authentication failures
Traceback (most recent call last):
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc/cli/__init__.py", line 209, in main
    ret = cmd.do_run()
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc/commands/data_sync.py", line 60, in run
    processed_files_count = self.repo.push(
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc/repo/__init__.py", line 64, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc/repo/push.py", line 92, in push
    result = self.cloud.push(
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc/data_cloud.py", line 196, in push
    t, f = self._push(default_objs, jobs=jobs, odb=odb)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc/data_cloud.py", line 212, in _push
    return self.transfer(
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc/data_cloud.py", line 167, in transfer
    return transfer(src_odb, dest_odb, objs, **kwargs)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 209, in transfer
    status = compare_status(
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 178, in compare_status
    dest_exists, dest_missing = status(
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 134, in status
    exists = hashes.intersection(
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 55, in _indexed_dir_hashes
    dir_exists.update(
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/tqdm/std.py", line 1166, in __iter__
    for obj in iterable:
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc_objects/db.py", line 359, in list_oids_exists
    in_remote = self.fs.exists(paths, batch_size=jobs)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 352, in exists
    if self.fs.async_impl:
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/funcy/objects.py", line 47, in __get__
    return prop.__get__(instance, type)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/dvc_ssh/__init__.py", line 119, in fs
    return _SSHFileSystem(**self.fs_args)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/fsspec/spec.py", line 79, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/sshfs/spec.py", line 67, in __init__
    self._client, self._pool = self.connect(
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/fsspec/asyn.py", line 121, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/fsspec/asyn.py", line 106, in sync
    raise return_result
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/fsspec/asyn.py", line 61, in _runner
    result[0] = await coro
  File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/sshfs/spec.py", line 97, in _connect
    client = await self._stack.enter_async_context(_raw_client)
  File "/usr/lib/python3.10/contextlib.py", line 619, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/asyncssh/misc.py", line 274, in __aenter__
    self._coro_result = await self._coro
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/asyncssh/connection.py", line 8093, in connect
    return await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/home/mzurad/archived_code/datasets_dvc/.env/lib/python3.10/site-packages/asyncssh/connection.py", line 440, in _connect
    await options.waiter
asyncssh.misc.ProtocolError: Too many authentication failures

2023-07-19 01:28:46,804 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-07-19 01:28:46,804 DEBUG: Removing '/home/mzurad/archived_code/.aYoNGJYx5AT4o3M85XLAJt.tmp'
2023-07-19 01:28:46,804 DEBUG: Removing '/home/mzurad/archived_code/.aYoNGJYx5AT4o3M85XLAJt.tmp'
2023-07-19 01:28:46,804 DEBUG: Removing '/home/mzurad/archived_code/.aYoNGJYx5AT4o3M85XLAJt.tmp'
2023-07-19 01:28:46,804 DEBUG: Removing '/home/mzurad/quick-test-dvc-cache/files/md5/.GrNKxJ89A2CeGiMYdpiniT.tmp'
2023-07-19 01:28:46,808 DEBUG: Version info for developers:
DVC version: 3.5.1 (pip)
------------------------
Platform: Python 3.10.6 on Linux-5.19.0-46-generic-x86_64-with-glibc2.35
Subprojects:
	dvc_data = 2.5.0
	dvc_objects = 0.23.1
	dvc_render = 0.5.3
	dvc_task = 0.3.0
	scmrepo = 1.0.4
Supports:
	http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
	ssh (sshfs = 2023.7.0)
Config:
	Global: /home/mzurad/.config/dvc
	System: /etc/xdg/xdg-ubuntu/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/ubuntu--vg-root
Caches: local
Remotes: local, ssh, ssh, ssh
Workspace directory: ext4 on /dev/mapper/ubuntu--vg-root
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/1a391b49cb1404774e3f64185a97e608

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-07-19 01:28:46,810 DEBUG: Analytics is disabled.

The nas machine is a QNAP NAS:

[mzurad@NAS3CF084 ~]$ uname -a
Linux NAS3CF084 5.10.60-qnap #1 SMP Fri Apr 21 01:51:27 CST 2023 x86_64 GNU/Linux

[pmrowla]

Do you have any other entries in your .ssh/config (like a generic Host * IdentityFile)?

[asiron]

I have other entries, but nothing like a generic Host *, here is my complete config:

Host srcp-throughtunnel
        HostName localhost
        User srcp
        Port 2222

Host github.com
        HostName github.com
        User git
        IdentityFile ~/.ssh/id_github

Host ws
  Hostname 10.242.243.168
  User mzurad
  IdentityFile ~/.ssh/id_ws

Host nas-access
  Hostname 10.242.176.69
  User mzurad
  IdentityFile /home/mzurad/.ssh/id_nas

I saw a comment from @efiop on my email, but now it seems to be missing here. Anyway I tried -j 1 and that fails as well

[efiop]

I saw a comment from @efiop on my email, but now it seems to be missing here. Anyway I tried -j 1 and that fails as well

@asiron Sorry for that, I sometimes write faster than I think 😅

[pmrowla]

Can you try moving your nas-access entry to the top of your config?

There may be a bug somewhere in either dvc-ssh or asyncssh that is trying all of the identities in your file instead of matching the host properly, which could lead to exceeding the server's MaxAuthTries setting. Moving the nas-access identity to be first in the list may workaround it for now.

[asiron]

I tried that, and then also removed all identities from ~/.ssh/ except id_nas.pub and id_nas and all entries in the ~/.ssh/config except nas-access... and still no luck

[asiron]

For additional info, I am adding the sshd_config from the nas-access machine:

[mzurad@NAS3CF084 ~]$ cat /etc/ssh/sshd_config | grep '^[^#]'
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
UseDNS no
Subsystem       sftp    /usr/libexec/sftp-server
AllowUsers admin

Which is weird, because it seems that only admin is the allowed user ? However, I can ssh-in using mzurad account

[pmrowla]

I haven't used a QNAP NAS before, but a lot of consumer NAS's use their own SSH server and config whenever you enable SSH through the manufacturer's web-based config (so the SSH server is not actually using the default /etc/ssh/sshd_config).

I tried that, and then also removed all identities from ~/.ssh/ except id_nas.pub and id_nas and all entries in the ~/.ssh/config except nas-access... and still no luck

I would not expect to see "Too many authentication failures" if you have removed all of the other entries from your client config. Could you please try rebooting your NAS and then retry using DVC with only the nas-access entry in your client config?

[asiron]

Unfortunately, that didn't help, but I did find the actual sshd_config that's being used:

(.env) ➜  datasets_dvc git:(png-detailed) ✗ ssh nas-access                  
[mzurad@NAS3CF084 ~]$ ps aux | grep sshd  
22730 admin      9256 S   sshd: mzurad [priv]
22742 mzurad     6004 S   sshd: mzurad@pts/0
23046 admin      9000 S   /usr/sbin/sshd -f /etc/config/ssh/sshd_config -p 22
30519 admin      9224 S   sshd: mzurad [priv]
30529 mzurad     7572 S   sshd: mzurad@pts/1
30761 mzurad     1088 S   grep sshd

which is:

[~] # cat /etc/config/ssh/sshd_config
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
PermitRootLogin yes
UseDNS no
Subsystem sftp /usr/libexec/sftp-server
AllowTcpForwarding no
AllowUsers admin abc mzurad bvca

I also tried again explicit ssh url in the .dvc/config, which also does not work:

[core]
    remote = "nas-access"
    analytics = false
['remote "nas-access"']
    url = ssh://mzurad@xx.xx.xx.xx/share/dvc-storage
    ask_password = true
    max_sessions = 3

Is there a way to get more verbose output from asyncssh ?

[asiron]

I thought that maybe the problem was because /share/dvc-storage is not a physical location, but symlinked to /share/CE_CACHEDEV1_DATA/dvc-storage. However, specifying the actual location didn't help.
Currently, I am using sshfs to mount a /share/CE_CACHEDEV1_DATA/dvc-storage on another machine and access that machine with an ssh:// DVC remote and that works without problems.

[pmrowla]

You use an SSH key with passphrase to connect, and not username + password, correct?

['remote "nas-access"']
    url = ssh://mzurad@xx.xx.xx.xx/share/dvc-storage
    ask_password = true
    max_sessions = 3

The remote config should contain

    ask_passphrase = true

ask_password is only used for username + password authentication. ask_passphrase is used if you need to specify a passphrase for your SSH key.

[asiron]

yes I tried both, ask_password when using (username + password) and ask_passphrase when using a keypair with a passphrase (which was empty)

My current work-around is to sshfs-mount NAS volume on another server and use the following dvc remote: ssh://other-server/mnt/sshfs-mounted-nas-dvc-storage and that works without the problem.