error on filter

[simonbyrne]

I'm getting an error when trying to filter:

@time weekdayusd = filter((:pair => isusd, :timestamp => isweekday),
                          data) |> compute
ArgumentError: invalid index: Union{}[] of type Array{Union{},1}

Stacktrace:
 [1] to_index(::Array{Union{},1}) at ./indices.jl:262
 [2] to_index(::Array{JuliaDB.EmptySpace{NamedTuple{(:pair, :timestamp),Tuple{String,DateTime}}},1}, ::Array{Union{},1}) at ./indices.jl:247
 [3] to_indices at ./indices.jl:293 [inlined]
 [4] to_indices at ./indices.jl:290 [inlined]
 [5] getindex at ./abstractarray.jl:905 [inlined]
 [6] #fromchunks#104(::Array{JuliaDB.EmptySpace{NamedTuple{(:pair, :timestamp),Tuple{String,DateTime}}},1}, ::Tuple{DataType,DataType}, ::Function, ::Bool, ::Bool, ::Function, ::Type{NDSparse{NamedTuple{(:bid, :ask),Tuple{Float32,Float32}},Tuple{String,DateTime},Columns{NamedTuple{(:pair, :timestamp),Tuple{String,DateTime}},NamedTuple{(:pair, :timestamp),Tuple{WeakRefStrings.StringArray{String,1},Array{DateTime,1}}}},Columns{NamedTuple{(:bid, :ask),Tuple{Float32,Float32}},NamedTuple{(:bid, :ask),Tuple{Array{Float32,1},Array{Float32,1}}}}}}, ::Array{Dagger.Chunk{Any,MemPool.DRef},1}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/ndsparse.jl:389
 [7] fromchunks(::Type{NDSparse{NamedTuple{(:bid, :ask),Tuple{Float32,Float32}},Tuple{String,DateTime},Columns{NamedTuple{(:pair, :timestamp),Tuple{String,DateTime}},NamedTuple{(:pair, :timestamp),Tuple{WeakRefStrings.StringArray{String,1},Array{DateTime,1}}}},Columns{NamedTuple{(:bid, :ask),Tuple{Float32,Float32}},NamedTuple{(:bid, :ask),Tuple{Array{Float32,1},Array{Float32,1}}}}}}, ::Array{Dagger.Chunk{Any,MemPool.DRef},1}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/ndsparse.jl:388
 [8] #fromchunks#47(::Nothing, ::Int64, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Array{Dagger.Thunk,1}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/table.jl:160
 [9] fromchunks(::Array{Dagger.Thunk,1}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/table.jl:138
 [10] #filter#216(::Nothing, ::Function, ::Tuple{Pair{Symbol,typeof(isusd)},Pair{Symbol,typeof(isweekday)}}, ::JuliaDB.DNDSparse{NamedTuple{(:pair, :timestamp),Tuple{String,DateTime}},NamedTuple{(:bid, :ask),Tuple{Float32,Float32}}}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/selection.jl:22
 [11] filter(::Tuple{Pair{Symbol,typeof(isusd)},Pair{Symbol,typeof(isweekday)}}, ::JuliaDB.DNDSparse{NamedTuple{(:pair, :timestamp),Tuple{String,DateTime}},NamedTuple{(:bid, :ask),Tuple{Float32,Float32}}}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/selection.jl:21
 [12] top-level scope at util.jl:156
 [13] top-level scope at In[50]:1

See http://nbviewer.jupyter.org/github/JuliaComputing/TrueFX/blob/v1/JuliaDB%20with%20TrueFX%20dataset.ipynb#Filtering

[piever]

Somehow I think there are issues in filtering when one of the chunks becomes empty. In my particular case it got fixed on Julia 1.0.2. What version are you using?

[simonbyrne]

I had originally tried it with 1.0.1, but I just retried it with 1.0.2 and am seeing the same result.

[simonbyrne]

Ah, it's when the result of a distributed sparse array is empty:

julia> using JuliaDB

julia> d = ndsparse((a=collect(1:10),), rand(10),chunks=2);

julia> filter((:a=>x->x>9,), d)
1-d Distributed NDSparse with 1 values (Float64) in 1 chunks:
a  │
───┼─────────
10 │ 0.197287

julia> filter((:a=>x->x>10,), d)
ERROR: ArgumentError: invalid index: Union{}[] of type Array{Union{},1}
Stacktrace:
 [1] to_index(::Array{Union{},1}) at ./indices.jl:262
 [2] to_index(::Array{JuliaDB.EmptySpace{NamedTuple{(:a,),Tuple{Int64}}},1}, ::Array{Union{},1}) at ./indices.jl:247
 [3] to_indices at ./indices.jl:293 [inlined]
 [4] to_indices at ./indices.jl:290 [inlined]
 [5] getindex at ./abstractarray.jl:905 [inlined]
 [6] #fromchunks#104(::Array{JuliaDB.EmptySpace{NamedTuple{(:a,),Tuple{Int64}}},1}, ::Tuple{DataType,DataType}, ::Function, ::Bool, ::Bool, ::Function, ::Type{NDSparse{Float64,Tuple{Int64},Columns{NamedTuple{(:a,),Tuple{Int64}},NamedTuple{(:a,),Tuple{Array{Int64,1}}}},Array{Float64,1}}}, ::Array{Dagger.Chunk{Any,MemPool.DRef},1}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/ndsparse.jl:389
 [7] fromchunks(::Type{NDSparse{Float64,Tuple{Int64},Columns{NamedTuple{(:a,),Tuple{Int64}},NamedTuple{(:a,),Tuple{Array{Int64,1}}}},Array{Float64,1}}}, ::Array{Dagger.Chunk{Any,MemPool.DRef},1}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/ndsparse.jl:388
 [8] #fromchunks#47(::Nothing, ::Int64, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Array{Dagger.Thunk,1}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/table.jl:160
 [9] fromchunks(::Array{Dagger.Thunk,1}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/table.jl:138
 [10] #filter#216(::Nothing, ::Function, ::Tuple{Pair{Symbol,getfield(Main, Symbol("##5#6"))}}, ::JuliaDB.DNDSparse{NamedTuple{(:a,),Tuple{Int64}},Float64}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/selection.jl:22
 [11] filter(::Tuple{Pair{Symbol,getfield(Main, Symbol("##5#6"))}}, ::JuliaDB.DNDSparse{NamedTuple{(:a,),Tuple{Int64}},Float64}) at /home/simonbyrne/.julia/packages/JuliaDB/vRAcT/src/selection.jl:21
 [12] top-level scope at none:0

It's fine when not distributed (i.e. without the chunks argument)

[simonbyrne]

So the issue is here: https://github.com/JuliaComputing/JuliaDB.jl/blob/63b5afee5caca77b0c33715a0540ef354714cff5/src/ndsparse.jl#L388-L389
findall is returning Union{}[], which then causes problems with getindex.