"cp" cannot read from GPFS file system

[eschnett]

Julia's cp function apparently cannot read from GPFS file systems. For example, this command

cp("/gpfs/eschnetter/_julia/packages/IJulia/fRegO/deps/logo-32x32.png", "/tmp/q9")

loops forever on my system. It reads from a GPFS file system and writes to a regular (local) file system. I verified that the shell cp command can copy the file just fine. Calling cp for files on other file systems also works fine.

Internally, Julia's cp calls sendfile, which calls jl_fs_sendfile:

function sendfile(dst::File, src::File, src_offset::Int64, bytes::Int)
    check_open(dst)
    check_open(src)
    while true
        result = ccall(:jl_fs_sendfile, Int32, (OS_HANDLE, OS_HANDLE, Int64, Csize_t),
                       src.handle, dst.handle, src_offset, bytes)
        @show "sendfile" result bytes
        uv_error("sendfile", result)
        nsent = result
        bytes -= nsent
        src_offset += nsent
        bytes <= 0 && break
    end
    nothing
end

When I insert a respective @show command (see above), I see that result=0 while bytes=1310 (its initial value).

This is Julia 1.2, on Ubuntu 16.04, kernel 4.15.0-47-generic.

I suspect that the Linux sendfile system call fails for GPFS and always returns 0 bytes read, but not indicating a true failure. libuv doesn't handle this case either, and Julia then loops forever. The destination file is created, but remains empty.

If this is correct, then Julia (or libuv) should detect this, and should abort with an error if no bytes were read and no EAGAIN is indicated. Ideally, Julia (or libuv) would then use a fallback mechanism to copy the file.

[Keno]

File a ticket with your kernel vendor? If the system call is not supported on gpfs it should properly indicate an error.

[eschnett]

I don't know where the actual bug is.

I just realize that the current Julia code doesn't handle the case where the file is shrinking while being read. In this case, sendfile will return 0 (no further bytes to copy), and nbytes will never reach 0. Other sample codes on the web exit the loop when nsent==0, not when bytes<=0.

[JonathanAnderson]

FWIW, I just tested this on centos cp("/gpfs/users/janders/data.csv", "/tmp/data.csv") and it worked just fine for me.

[StefanKarpinski]

Possibly a kernel/driver bug that got fixed. Can you two post your kernel versions?

[eschnett]

This is on Ubuntu 16.04, kernel 4.15.0-47-generic.
mmlsconfig says minReleaseLevel 4.2.3.0.

[Keno]

FWIW, I do think the shrinking file case is compelling. The behavior of sendfile isn't really well defined in that case, but infinite looping julia seems like a bad outcome.

[JonathanAnderson]

$ uname -r
3.10.0-862.14.4.el7.x86_64

$  cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)