error with combination of types and select/drop options of CSV.File

[kragol]

An error occurs whenever the types option to CSV.File contains a pair corresponding to a dropped column with a type <: Union{Int16, Int32, Int8, UInt16, UInt32, UInt8} (aka SmallIntegers within CSV.jl).

MWE:
file test.csv:

"a","b","c"
1,"a",12
1,"a",12

julia> CSV.File("test.csv"),select=["a"],types=Dict("a"=>Int32,"b"=>String,"c"=>Int16))
ERROR: MethodError: Cannot `convert` an object of type Missing to an object of type Int16
Closest candidates are:
  convert(::Type{T}, ::T) where T<:Number at number.jl:6
  convert(::Type{T}, ::Number) where T<:Number at number.jl:7
  convert(::Type{T}, ::Ptr) where T<:Integer at pointer.jl:23
  ...
Stacktrace:
 [1] setindex!(::Array{Int16,1}, ::Missing, ::Int64) at ./array.jl:826
 [2] copyto!(::Array{Int16,1}, ::Array{Union{Missing, Int16},1}) at ./multidimensional.jl:962
 [3] Array at ./array.jl:541 [inlined]
 [4] convert(::Type{Array{Int16,1}}, ::Array{Union{Missing, Int16},1}) at ./array.jl:533
 [5] CSV.File(::CSV.Header{false,Parsers.Options{false,false,true,false,Missing,UInt8,Nothing},Array{UInt8,1}}; startingbyteposition::Nothing, endingbyteposition::Nothing, limit::Int64, threaded::Nothing, typemap::Dict{Type,Type}, tasks::Int64, debug::Bool) at /home/kragol/.julia/dev/CSV/src/file.jl:280
 [6] CSV.File(::String; header::Int64, normalizenames::Bool, datarow::Int64, skipto::Nothing, footerskip::Int64, transpose::Bool, comment::Nothing, use_mmap::Nothing, ignoreemptylines::Bool, select::Array{String,1}, drop::Nothing, missingstrings::Array{String,1}, missingstring::String, delim::Nothing, ignorerepeated::Bool, quotechar::Char, openquotechar::Nothing, closequotechar::Nothing, escapechar::Char, dateformat::Nothing, dateformats::Nothing, decimal::UInt8, truestrings::Array{String,1}, falsestrings::Array{String,1}, type::Nothing, types::Dict{String,DataType}, categorical::Nothing, pool::Float64, lazystrings::Bool, strict::Bool, silencewarnings::Bool, debug::Bool, parsingdebug::Bool, kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/kragol/.julia/dev/CSV/src/file.jl:215
 [7] top-level scope at REPL[37]:1

Note: the error may look a bit different when multi-threading is enabled but is essentially the same.

A simple workaround is to remove the pairs in the types Dict that are supposed to be dropped according to the select or drop options.

julia 1.4.2
CSV 0.7.2

[kragol]

The proposed fix appears to work only in non-threaded mode. (My fix doesn't work either in multi-threaded mode). I didn't realize the issue was that different in multi-threaded mode.

Here is a stack trace in multi-threaded mode for the same file as the issue report but with 20k rows.

CSV.File("test.csv",select=["a"],types=Dict("a"=>Int32,"b"=>String,"c"=>Int16))
ERROR: TaskFailedException:
MethodError: Cannot `convert` an object of type Missing to an object of type Int16
Closest candidates are:
  convert(::Type{T}, ::T) where T<:Number at number.jl:6
  convert(::Type{T}, ::Number) where T<:Number at number.jl:7
  convert(::Type{T}, ::Ptr) where T<:Integer at pointer.jl:23
  ...
Stacktrace:
 [1] setindex!(::Array{Int16,1}, ::Missing, ::Int64) at ./array.jl:826
 [2] copyto!(::Array{Int16,1}, ::Array{Union{Missing, Int16},1}) at ./multidimensional.jl:962
 [3] Array at ./array.jl:541 [inlined]
 [4] convert at ./array.jl:533 [inlined]
 [5] setindex! at ./array.jl:826 [inlined]
 [6] makechain(::Type{Array{Int16,1}}, ::Array{Union{Missing, Int16},1}, ::Int64, ::Int64, ::Array{Array{AbstractArray{T,1} where T,1},1}, ::Int64, ::Nothing) at /home/kragol/.julia/dev/CSV/src/file.jl:360
 [7] makechain(::Type{Array{Int16,1}}, ::Array{Union{Missing, Int16},1}, ::Int64, ::Int64, ::Array{Array{AbstractArray{T,1} where T,1},1}, ::Int64) at /home/kragol/.julia/dev/CSV/src/file.jl:354
 [8] macro expansion at /home/kragol/.julia/packages/CSV/W9RT2/src/file.jl:528 [inlined]
 [9] (::CSV.var"#36#41"{Array{Type,1},Array{UInt8,1},Array{UInt8,1},Parsers.Options{false,false,true,false,Missing,UInt8,Nothing},Nothing,Float64,Array{CSV.RefPool,1},Int64,Dict{Type,Type},Bool,DataType,Int64,Int64,Bool,Array{Int64,1},Array{Array{AbstractArray{T,1} where T,1},1},Array{Int64,1},Array{AbstractArray{T,1} where T,1},Int64})() at ./threadingconstructs.jl:126
Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:316
 [2] macro expansion at ./task.jl:335 [inlined]
 [3] multithreadparse(::Array{Type,1}, ::Array{UInt8,1}, ::Array{UInt8,1}, ::Int64, ::Int64, ::Parsers.Options{false,false,true,false,Missing,UInt8,Nothing}, ::Nothing, ::Int64, ::Int64, ::Float64, ::Array{CSV.RefPool,1}, ::Int64, ::Dict{Type,Type}, ::Bool, ::Type, ::Int64, ::Int64, ::Bool) at /home/kragol/.julia/dev/CSV/src/file.jl:459
 [4] CSV.File(::CSV.Header{false,Parsers.Options{false,false,true,false,Missing,UInt8,Nothing},Array{UInt8,1}}; startingbyteposition::Nothing, endingbyteposition::Nothing, limit::Int64, threaded::Nothing, typemap::Dict{Type,Type}, tasks::Int64, debug::Bool) at /home/kragol/.julia/dev/CSV/src/file.jl:254
 [5] CSV.File(::String; header::Int64, normalizenames::Bool, datarow::Int64, skipto::Nothing, footerskip::Int64, transpose::Bool, comment::Nothing, use_mmap::Nothing, ignoreemptylines::Bool, select::Array{String,1}, drop::Nothing, missingstrings::Array{String,1}, missingstring::String, delim::Nothing, ignorerepeated::Bool, quotechar::Char, openquotechar::Nothing, closequotechar::Nothing, escapechar::Char, dateformat::Nothing, dateformats::Nothing, decimal::UInt8, truestrings::Array{String,1}, falsestrings::Array{String,1}, type::Nothing, types::Dict{String,DataType}, typemap::Dict{Type,Type}, categorical::Nothing, pool::Float64, lazystrings::Bool, strict::Bool, silencewarnings::Bool, debug::Bool, parsingdebug::Bool, kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/kragol/.julia/dev/CSV/src/file.jl:217
 [6] top-level scope at REPL[11]:1

I am a bit lost with debugging multi-threaded code, so I won't try to fix this one. All I can say, is that adding the Missing type to the dropped columns solves the problem.

CSV.File("test.csv",select=["a"],types=Dict("a"=>Int32,"b"=>String,"c"=>Union{Missing,Int16}))

Notice though that it is not necessary for the "b" column, which I don't understand.

julia 1.4.2
CSV 0.7.3

[quinnj]

Thanks for the follow up; this is actually a separate issue and I've put up a fix here: #686.