Cannot move file error: invalid cross-device link

[camr0]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

I've come across this error when using the tempFirst option. The issue seems to be with the os.Rename function being used between different filesystems/disks. I'm working on a fix and am planning on submitting a PR soon 👍

Expected behaviour

Files are copied successfully and operation is completed without error.

Actual behaviour

File downloads to the tmp directory just fine. Error occurs when the program attempts to move the file from the tmp directory to the destination.

Steps to reproduce

1. Run ftpgrab w/ tempFirst enabled
2. Destination on a different drive than tmp directory

Version

• FTPGrab version (5.0.0):
• Platform (windows/linux/freebsd/macos): Linux (TrueNAS Scale & Arch Linux)
• System info (type uname -a): Linux Ali-Asahi 6.1.0-asahi-2-1-edge-ARCH error with awk #2 SMP PREEMPT_DYNAMIC Wed, 14 Dec 2022 14:33:17 +0000 aarch64 GNU/Linux
• Remote FTP (Pure-FTPd) or SFTP (OpenSSH) server: FTP

Configuration

# ./ftpgrab.yml

db:
  path: ftpgrab.db

server:
  ftp:
    host: {hostname}
    port: {#}
    username: {username}
    password: {password}
    sources:
      - /path/to/completed
    timeout: 5s

download:
  output: /home/ali/Documents/ftpgrab/downloads
  retry: 3
  hideSkipped: false
  tempFirst: true
  createBaseDir: false

Logs

Wed, 22 Feb 2023 01:37:54 EST WRN No notifier available
Wed, 22 Feb 2023 01:37:54 EST DBG 4 entries found in database
Wed, 22 Feb 2023 01:37:54 EST DBG Listing files source=path/to/completed
Wed, 22 Feb 2023 01:37:55 EST INF 1 file(s) found
Wed, 22 Feb 2023 01:37:55 EST DBG Move temp file destfile=/home/ali/Documents/ftpgrab/downloads/testfile.txt tempfile=/tmp/.ftpgrab.2063516771/testfile.txt544694374
Wed, 22 Feb 2023 01:37:55 EST ERR Cannot move file error="rename /tmp/.ftpgrab.2063516771/testfile.txt544694374 /home/ali/Documents/ftpgrab/downloads/testfile.txt: invalid cross-device link" dest=/home/ali/Documents/ftpgrab/downloads size=0B src=/path/to/completed/testfile.txt
Wed, 22 Feb 2023 01:37:55 EST INF Finished duration=1.359s

Additional info

No response

[screamjojo]

Same issue. @camr0 did you find a solution ?

[camr0]

@screamjojo Yes, turns out the issue was that the destination was on a different drive than the default tmp directory. ftpgrab uses the "os.rename" function internally to move the files from the tmp directory to the destination, which doesn't support renaming between filesystems. Initially I tried replacing the os.rename function with something else but ran into issues.

Ultimately, I decided to add an option for a custom temporary directory, that way I could make a tmp folder on the same drive. The flag is called "tempDir" and is working on my personal fork here. The code is a little rough so I'm not sure if it's ready for a PR yet lol. Give it a try and let me know if you have any issues!

[screamjojo]

Thank you for reply, how can I add it with docker ?

[crazy-max]

@camr0 Thanks for your investigation

is working on my personal fork here.

I'm not sure what are the actual changes looking at your fork. Can you point me to the commit?

[nothing2obvi]

I run into the same issue on v7.9.0 if I enable tempFirst. If I disable it, there is no problem. However, I would like to enable the tempFirst option.

[Thlb]

Hi ! Any update to fix this ? I've tried multiples configurations : none of them worked :(

[HBoardman]

Same issue here. If I could specify a temporary directory manually this should be rectified. Or possibly use copy then delete instead of rename?

[HBoardman]

Same issue here. If I could specify a temporary directory manually this should be rectified. Or possibly use copy then delete instead of rename?

I have a work-around for this. I now have ftpgrab download to a temporary location and wrote a bash script to move the files to the "final" directory (i.e where sonarr/radarr is expecting them) but only when they stop getting written to. Since Unraid itself is then doing the move it seems to be happier.

If anyone wants the script just let me know.

[Thlb]

Hi @HBoardman

I'm interested if you still ok to share the script :)

[com6056]

On linux, you can change the directory that is used as the tempdir by setting the TMPDIR environment variable: https://pkg.go.dev/os#TempDir

On Unix systems, it returns $TMPDIR if non-empty, else /tmp. On Windows, it uses GetTempPath, returning the first non-empty value from %TMP%, %TEMP%, %USERPROFILE%, or the Windows directory. On Plan 9, it returns /tmp.

[aclare92]

To jump in on this one, I assume most replies above are using Docker; to which Docker does weird things when mounting external directories to the container, in this case /download.

As @com6056 very helpfully mentioned, you can override the TMPDIR variable, setting this to something like;

TMPDIR="/download/.ftpgrab"

Meaning, internally to Docker, both /download and /download/.ftpgrab will be on the "same filesystem".

Workaround I know, and as per @camr0 I did also implement my own fix at one point in time but never got round to sorting a PR for it and discarded the changes in a re-fork.

@crazy-max, if you are looking for a fix, essentially rough notes what I did was change your internal/grabber/move.go#moveFile function from using os.Rename to the below;

import (
    ...
    "io"
    ...
)

inputFile, err := os.Open(oldpath)
// any error handling from os.Open
outputFile, err := os.Create(newpath)
// any error handling from os.Create and remember to inputFile.Close
defer outputFile.Close
_, err = io.Copy(outputFile, inputFile)
inputFile.Close()
// any error handling from io.Copy
// If the copy was successful, remove the oldpath
err = os.Remove(oldpath)
// any error handling from os.Remove