asyncssh.sftp.SFTPBadMessage: Unable to decode name

[ttomasz]

I'm getting the following error when trying to list contents of a directory:

File "/home/xxx/venv/lib/python3.10/site-packages/sshfs/spec.py", line 204, in _ls
    file_attrs = await channel.readdir(path)
  File "/home/xxx/venv/lib/python3.10/site-packages/asyncssh/sftp.py", line 5098, in readdir
    return [entry async for entry in self.scandir(path)]
  File "/home/xxx/venv/lib/python3.10/site-packages/asyncssh/sftp.py", line 5098, in <listcomp>
    return [entry async for entry in self.scandir(path)]
  File "/home/xxx/venv/lib/python3.10/site-packages/asyncssh/sftp.py", line 5067, in scandir
    self.decode(cast(bytes, entry.filename))
  File "/home/xxx/venv/lib/python3.10/site-packages/asyncssh/sftp.py", line 3528, in decode
    raise SFTPBadMessage('Unable to decode name') from None
asyncssh.sftp.SFTPBadMessage: Unable to decode name

(I'm actually using sshfs library but this part of stack suggests error is coming from this library)

upon checking the file paths with lftp cli tool I see that some paths are garbage e.g.:

drwxr-xr-x    3 1002     1005     25210880 Nov 13 20:45 .
drwxr-xr-x    7 root     root         4096 Feb 22  2022 ..
drwx------    2 1002     1005         4096 Mar 26  2023 .cache
-rw-r--r--    1 root     root         7421 Nov 13 20:45 00000770_2303241948.##$$ã<8b>°+ót°+^D<89>°+ðæ^A^
-rw-r--r--    1 root     root        56697 Sep  9  2022 00001020_2209040017.xml
-rw-r--r--    1 root     root         1825 Sep  9  2022 00001020_2209040019.xml
-rw-r--r--    1 root     root         4029 Sep  9  2022 00001020_2209040121.xml
...
-rw-r--r--    1 root     root         9414 Dec 11  2022 00001113_2212111945.xml
-rw-r--r--    1 root     root          340 Jan 17  2023 00001113_2212112052.<8d><8d>ð2¿á(+8Ì(+¬á(+ÿ<^Gÿ
-rw-r--r--    1 root     root         6661 Dec 11  2022 00001113_2212112054.xml

It would be nice however if that could be gracefully handled. CLI tools like sftp and lftp deal with it somehow.

[ronf]

If the filenames aren't valid UTF-8 sequences, you can either specify the alternate encoding that the server is using in the path_encoding argument to conn.start_sftp_client(), or you can set path_encoding to None to get back filenames as bytes objects. This should avoid the "Unable to decode name" errors.

[ttomasz]

Thank you for quick response. I'll try with path_encoding set to None.

[ttomasz]

Interestingly there is also parameter:

           :param path_errors:
               The error handling strategy to apply on encode/decode

Though the problem for me seems to be that these parameters are not available in connect() method which is used by the upstream sshfs library that I use. I'll ask if they have any thoughts about this.

[ronf]

Setting path_errors is meant to be used in the case where you know the encoding and most of the characters decode correctly, but some small number of characters trigger a decode error. You can choose to replace or escape just those characters but still decode the rest and still return a string.

For your example, I don't think that's likely to work well here, so I suggested getting the raw bytes and then deciding yourself how to make sense out of them, if you need to.

Unfortunately, as you said, it doesn't look like sshfs provides a way to pass any parameters to start_sftp_cient() right now. It also doesn't look like you can pass in your own established SFTP sessions, so you'd probably need a code change in that package to make this work.

[ttomasz]

The FTP server has close to 500_000 files and only 6 of them have this broken path name so if we could just ignore those or automatically convert the characters in the name to something utf8 compatible that would work for us. We'll se what the sshfs guys say.

I guess we can close this issue as there is nothing to do with this library :) Thanks again for quick replies.

[ronf]

Happy to help - let me know if you need anything else.