[Suggestion] Add note about potential permissions issues

[FaySmash]

Since I've started using tubesync, everything worked but the removal of the files in the /downloads/cache directory. But I've noticed these lines in the log:

2025-01-11T10:26:03.119770319Z [MoveFiles] Moving file "/downloads/cache/test.mkv" to "/downloads/video/test.mkv"
ERROR: [Errno 1] Operation not permitted
Traceback (most recent call last):
  File "/usr/lib/python3.11/shutil.py", line 825, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/downloads/cache/test.mkv' -> '/downloads/video/test.mkv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/yt_dlp/YoutubeDL.py", line 1634, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/yt_dlp/YoutubeDL.py", line 1790, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/yt_dlp/YoutubeDL.py", line 1849, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/yt_dlp/YoutubeDL.py", line 3021, in process_video_result
    self.process_info(new_info)
  File "/usr/local/lib/python3.11/dist-packages/yt_dlp/YoutubeDL.py", line 177, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/yt_dlp/YoutubeDL.py", line 3567, in process_info
    replace_info_dict(self.post_process(dl_filename, info_dict, files_to_move))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/yt_dlp/YoutubeDL.py", line 3752, in post_process
    info = self.run_pp(MoveFilesAfterDownloadPP(self), info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/yt_dlp/YoutubeDL.py", line 3711, in run_pp
    files_to_delete, infodict = pp.run(infodict)
                                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/yt_dlp/postprocessor/common.py", line 22, in run
    ret = func(self, info, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/yt_dlp/postprocessor/movefilesafterdownload.py", line 47, in run
    shutil.move(oldfile, newfile)  # os.rename cannot move between volumes
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/shutil.py", line 845, in move
    copy_function(src, real_dst)
  File "/usr/lib/python3.11/shutil.py", line 437, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.11/shutil.py", line 376, in copystat
    lookup("utime")(dst, ns=(st.st_atime_ns, st.st_mtime_ns),
PermissionError: [Errno 1] Operation not permitted

The underlying problem was the shutil function move, which called the function copy2 which called the function copystat which called the function lookup which failed because of "Operation not permitted" and the script execution stopped, without calling os.unlink(src), leaving the file in the /cache folder.

Because this error was very annoying to track down, maybe there could be an added section in the documentation, with a note to set the UID/GID for the container so that the moving process can finish completely.

[tcely]

What is your setup that prevents the move from reading and/or preserving the stats on the copied file??

Ideally, /downloads/cache and /downloads/video should be the same filesystem so that the rename can happen as quickly as possible.

[FaySmash]

What is your setup that prevents the move from reading and/or preserving the stats on the copied file??

Great question! This is my docker-compose volume config

services:
    [...]
    tmpfs:
      - /downloads:mode=0777,uid=0,gid=0
    volumes:
      - config:/config
      - mnt_output:/downloads/video

volumes:
  config:
  mnt_output:
    driver: local
    driver_opts:
      device: //nas
      o: username=nas,password=nas,file_mode=0777,dir_mode=0777,nounix,vers=3.0,rw
      type: cifs

I didn't want to write the cache files onto my NAS with does snapshots every 10 minutes and wastes space on temp files. Write cache files into tmpfs also spares the drives from unnecessary wear. But because of this, move calls copy2 instead of os.rename

[tcely]

The copy appears to be partially completed, your NAS isn't letting the software set the times on the copied file.

Set the access and modified times of the file specified by path.
https://docs.python.org/3.11/library/os.html#os.utime

My best guess is either the mount is enforcing permissions on the client side or an ACL on your NAS needs to be adjusted.

Any idea why that might be happening?

FYI, I run TubeSync against a TrueNAS system with hourly snapshots, the files in the cache directory don't cause me any issues in my setup.

[FaySmash]

The copy appears to be partially completed, your NAS isn't letting the software set the times on the copied file.

Correct, the copy finishes but the script crashes when trying to change the file attributes.

My best guess is either the mount is enforcing permissions on the client side or an ACL on your NAS needs to be adjusted.

In the mount options it's specified with rwx-rwx-rwx and as rw, the user has modifying permissions on the share and mounted folder, so it should allowed to write files (which works) and setting the file attributes.

Any idea why that might be happening?

Because it works when I set the environment variables UID and GID to 0 (root) for the container, it seems that when the python process runs with a non-root user (default UID/GID =1000), it lacks the required permissions to set these file attributes on the mounted volume. As os.rename doesn't try to set the file attributes and just changes the path, the problem only appearers when copy2 is used instead.

[tcely]

Because it works when I set the environment variables UID and GID to 0 (root) for the container, it seems that when the python process runs with a non-root user (default UID/GID =1000), it lacks the required permissions to set these file attributes on the mounted volume.

Samba (cifs/smb3) is single user. By default, only root has access to the mounted files. It makes sense that trying to access them as another user won't work well. You can change the uid/gid for the mount and/or for the container, but you will need them to be matched.

The core CIFS protocol does not provide unix ownership information or mode for files and directories.
Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set,
and will have permissions set to the default file_mode and dir_mode for the mount.
Attempting to change these values via chmod/chown will return success but have no effect.

https://manpages.debian.org/unstable/cifs-utils/mount.smb3.8.en.html#uid=arg

[FaySmash]

Samba (cifs/smb3) is single user. By default, only root has access to the mounted files. It makes sense that trying to access them as another user won't work well. You can change the uid/gid for the mount and/or for the container, but you will need them to be matched.

I tried to switch back to the UID and GID 1000, I've set them as environment variables and adjusted the docker-compose:

    [...]
    tmpfs:
      - /downloads:mode=0777,uid=1000,gid=1000
    volumes:
      - config:/config
      - mnt_output:/downloads/video

volumes:
  config: null
  mnt_output:
    driver: local
    driver_opts:
      device: //nas
      o: username=nas,password=nas,file_mode=0777,dir_mode=0777,nounix,vers=3.0,rw,uid=1000,gid=1000
      type: cifs

but this didn't help. For now I'll let the container run as root.
If this issue is deemed noteworthy, I would suggest adding a hint about the "cache folder doesn't get deleted" issue to the FAQ section, with notice about the importance of the UID and GID when using mounts.

[tcely]

You didn't show the environment section for your compose service.

If I remember correctly, it would need to have variables like this:

environment:
      - PUID=1000
      - PGID=1000

Plus, the mount options you have already showed.
Oh, also be sure to inspect the containers to confirm that compose created a new volume and gave the proper mount options for your filesystems.

[FaySmash]

You didn't show the environment section for your compose service.

Yeah, but I mentioned that I changed them accordingly (even if I mislabeled them as UID and GID with the missing P, but there were correctly set)

Oh, also be sure to inspect the containers to confirm that compose created a new volume and gave the proper mount options for your filesystems.

This was the culprit! I forgot to remove the volume after changing the mount options

docker-compose:

    [...]
    env_file: stack.env
    [...]
    tmpfs:
      - /downloads:mode=0777,uid=1000,gid=1000
    volumes:
      - config:/config
      - mnt_output:/downloads/video

volumes:
  config: null
  mnt_output:
    driver: local
    driver_opts:
      device: //nas
      o: username=nas,password=nas,file_mode=0777,dir_mode=0777,nounix,vers=3.0,rw,uid=1000,gid=1000
      type: cifs

stack.env:

[...]
PUID=1000
PGID=1000

using this config, the problem doesn't appear, even when using a non-root user.
Maybe a little note about the uid / gid parameters on mounts could save other people the trouble.

[tcely]

I'm glad you have it working!

Write up the note you prefer in a pull request and I'm sure it will be reviewed. 😀

[FaySmash]

I'm glad you have it working!

Thanks for the help and the right pointers!

Write up the note you prefer in a pull request and I'm sure it will be reviewed. 😀

Done