saving >2GB file on Windows 7 gives ERROR: SystemError: truncate: Invalid Argument

[jaakkor2]

On Windows 7, Julia v0.6.0, FileIO v0.5.2, JLD2 0.0.4.

using FileIO, JLD2
N = 10^4
M = 10^4
a = rand(Float32, N, M)
b = rand(Float32, N, M)
c = rand(Float32, N, M)
d = rand(Float32, N, M)
e = rand(Float32, N, M)
f = rand(Float32, N, M)
save("test_foo.jld2", Dict("a" => a, "b" => b, "c" => c, "d" => d, "e" => e, "f" => f))

gives

Error encountered while saving "test_foo.jld2".
Fatal error:
ERROR: SystemError: truncate: Invalid argument
Stacktrace:
 [1] #systemerror#44 at .\error.jl:64 [inlined]
 [2] systemerror(::String, ::Bool) at .\error.jl:64
 [3] truncate(::IOStream, ::Int64) at .\iostream.jl:43
 [4] truncate_and_close(::JLD2.MmapIO, ::Int64) at C:\Users\jj\.julia\v0.6\
JLD2\src\mmapio.jl:164
 [5] close(::JLD2.JLDFile{JLD2.MmapIO}) at C:\Users\jj\.julia\v0.6\JLD2\src
\JLD2.jl:364
 [6] #jldopen#29(::Array{Any,1}, ::Function, ::JLD2.##35#36{Dict{String,Array{Fl
oat32,2}}}, ::String, ::Vararg{String,N} where N) at C:\Users\jj\.julia\v0.
6\JLD2\src\loadsave.jl:6
 [7] #save#34(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:JLD2}
}, ::Dict{String,Array{Float32,2}}) at C:\Users\jj\.julia\v0.6\JLD2\src\loa
dsave.jl:85
 [8] save(::FileIO.File{FileIO.DataFormat{:JLD2}}, ::Dict{String,Array{Float32,2
}}) at C:\Users\jj\.julia\v0.6\JLD2\src\loadsave.jl:85
 [9] eval(::Module, ::Any) at .\boot.jl:235
 [10] #save#21(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:JLD2
}}, ::Dict{String,Array{Float32,2}}, ::Vararg{Dict{String,Array{Float32,2}},N} w
here N) at C:\Users\jj\.julia\v0.6\FileIO\src\loadsave.jl:115
 [11] save(::FileIO.File{FileIO.DataFormat{:JLD2}}, ::Dict{String,Array{Float32,
2}}) at C:\Users\jj\.julia\v0.6\FileIO\src\loadsave.jl:106
 [12] #save#14(::Array{Any,1}, ::Function, ::String, ::Dict{String,Array{Float32
,2}}, ::Vararg{Dict{String,Array{Float32,2}},N} where N) at C:\Users\jj\.ju
lia\v0.6\FileIO\src\loadsave.jl:61
 [13] save(::String, ::Dict{String,Array{Float32,2}}) at C:\Users\jj\.julia
\v0.6\FileIO\src\loadsave.jl:61

If N and M are chosen smaller (<2GB), saving works.

[jaakkor2]

Looks similar to the (unreproduced and) closed #42 .

[simonster]

My guess is that Julia's truncate function does not support sizes >2GB on Windows, since ios_trunc calls _chsize, which takes a long and not a 64-bit integer. It should probably be modified to use _chsize_s. Can you confirm that:

open("some_file_name", "w") do f
    truncate(f, 2^31+1)
end

fails on Windows? (You'll need 4GB of free space, and you'll want to delete the file afterwards if it works.)

[jaakkor2]

Yes, it fails.

julia> open("some_file_name", "w") do f
           truncate(f, 2^31+1)
       end
ERROR: SystemError: truncate: Invalid argument
Stacktrace:
 [1] #systemerror#44 at .\error.jl:64 [inlined]
 [2] systemerror(::String, ::Bool) at .\error.jl:64
 [3] truncate(::IOStream, ::Int64) at .\iostream.jl:43
 [4] open(::##1#2, ::String, ::String) at .\iostream.jl:152

julia> open("some_file_name", "w") do f
           truncate(f, 2^30+1)
       end
IOStream(<file some_file_name>)

[simonster]

Filed JuliaLang/julia#24466. In principle this could be worked around by calling _chsize_s directly on Windows using ccall, but I'm not sure if I'll have time to get around to that in the near future.

[jaakkor2]

The upstream fix JuliaLang/julia#24616 still has the backport pending 0.6 label.

[jaakkor2]

With Julia v0.6.3 released containing the upstream fix, I now close this bug.