Invalidations when loading CSV #3248

timholy · 2022-12-13T19:56:14Z

While DataFrames itself seems admirably unaffected by invalidations (more detail on this below), the combination CSV + DataFrames must surely be a common one and this combination does invalidate some of DataFrames' code. While "calling for a fix," this issue is primarily a tutorial on detecting and diagnosing such issues.

First, on what is currently the latest version of Julia (and ideally with JuliaLang/julia#47889), you need to use timholy/SnoopCompile.jl#300, as JET.jl is not currently working on Julia master.

The concept of the workflow is to record invalidations with @snoopr and inference with @snoopi_deep: by looking at the specializations that got inferred because of invalidations, we learn the minimal set of things that you need to do to get the DataFrames code running without invalidation. (This way you can ignore lots of invalidations that happen in Julia's own internal code or other packages, focusing just on what you need to fix.) The specific workload is copied from your own precompile workload so you can be optimistic it should have been precompiled (any inference is presumably a direct consequence of invalidation).

julia> using SnoopCompileCore

julia> invs = @snoopr begin using PooledArrays: PooledArrays, PooledArray; using DataFrames, Statistics; using CSV; end;

julia> tinf = @snoopi_deep begin
           df = DataFrame(a=[2, 5, 3, 1, 0], b=["a", "b", "c", "a", "b"], c=1:5,
                          p=PooledArray(["a", "b", "c", "a", "b"]),
                          q=[true, false, true, false, true],
                          f=Float64[2, 5, 3, 1, 0])
           describe(df)
           names(df[1, 1:2])
           sort(df, :a)
           combine(df, :c, [:c :f] .=> [sum, mean, std], :c => :d, [:a, :c] => cor)
           transform(df, :c, [:c :f] .=> [sum, mean, std], :c => :d, [:a, :c] => cor)
           groupby(df, :a)
           groupby(df, :q)
           groupby(df, :p)
           gdf = groupby(df, :b)
           combine(gdf, :c, [:c :f] .=> [sum, mean, std], :c => :d, [:a, :c] => cor)
           transform(gdf, :c, [:c :f] .=> [sum, mean, std], :c => :d, [:a, :c] => cor)
           innerjoin(df, df, on=:a, makeunique=true)
           innerjoin(df, df, on=:b, makeunique=true)
           innerjoin(df, df, on=:c, makeunique=true)
           outerjoin(df, df, on=:a, makeunique=true)
           outerjoin(df, df, on=:b, makeunique=true)
           outerjoin(df, df, on=:c, makeunique=true)
           semijoin(df, df, on=:a)
           semijoin(df, df, on=:b)
           semijoin(df, df, on=:c)
           leftjoin!(df, DataFrame(a=[2, 5, 3, 1, 0]), on=:a)
           leftjoin!(df, DataFrame(b=["a", "b", "c", "d", "e"]), on=:b)
           leftjoin!(df, DataFrame(c=1:5), on=:c)
           reduce(vcat, [df, df])
           show(IOBuffer(), df)
           subset(df, :q)
           @view df[1:3, :]
           @view df[:, 1:2]
           transform!(df, :c, [:c :f] .=> [sum, mean, std], :c => :d, [:a, :c] => cor)
       end;

julia> using SnoopCompile

julia> trees = invalidation_trees(invs);

julia> staletrees = precompile_blockers(trees, tinf)
8-element Vector{SnoopCompile.StaleTree}:
 inserting copyto!(dest::SentinelArrays.ChainedVector{T, A} where A<:AbstractVector{T}, doffs::Union{Signed, Unsigned}, src::AbstractVector, soffs::Union{Signed, Unsigned}, n::Union{Signed, Unsigned}) where T @ SentinelArrays ~/.julia/packages/SentinelArrays/XvFr7/src/chainedvector.jl:354 invalidated:
   backedges: 1: MethodInstance for copyto!(::AbstractVector, ::Int64, ::SubArray{String, 1, Vector{String}, Tuple{UnitRange{Int64}}, true}, ::Int64, ::Int64) at depth 0 with 1 children blocked InferenceTimingNode: 0.001038/0.001038 on DataFrames._noon_compose_helper!(::Vector{AbstractVector}, similar_missing::typeof(DataFrames.similar_missing), ::Int64, ::Vector{String}, ::Int64, ::Vector{Int64}, ::Int64, ::UnitRange{Int64}, ::Int64) with 0 direct children
              2: MethodInstance for copyto!(::AbstractVector, ::Int64, ::SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, ::Int64, ::Int64) at depth 0 with 1 children blocked InferenceTimingNode: 0.001042/0.001042 on DataFrames._noon_compose_helper!(::Vector{AbstractVector}, similar_missing::typeof(DataFrames.similar_missing), ::Int64, ::Vector{Int64}, ::Int64, ::Vector{Int64}, ::Int64, ::UnitRange{Int64}, ::Int64) with 0 direct children
              3: MethodInstance for copyto!(::AbstractVector, ::Int64, ::SubArray{Bool, 1, Vector{Bool}, Tuple{UnitRange{Int64}}, true}, ::Int64, ::Int64) at depth 0 with 1 children blocked InferenceTimingNode: 0.001050/0.001050 on DataFrames._noon_compose_helper!(::Vector{AbstractVector}, similar_missing::typeof(DataFrames.similar_missing), ::Int64, ::Vector{Bool}, ::Int64, ::Vector{Int64}, ::Int64, ::UnitRange{Int64}, ::Int64) with 0 direct children
              4: MethodInstance for copyto!(::AbstractVector, ::Int64, ::SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, ::Int64, ::Int64) at depth 0 with 1 children blocked InferenceTimingNode: 0.001057/0.001057 on DataFrames._noon_compose_helper!(::Vector{AbstractVector}, similar_missing::typeof(DataFrames.similar_missing), ::Int64, ::Vector{Float64}, ::Int64, ::Vector{Int64}, ::Int64, ::UnitRange{Int64}, ::Int64) with 0 direct children

 inserting extrema(x::SentinelArrays.ChainedVector) @ SentinelArrays ~/.julia/packages/SentinelArrays/XvFr7/src/chainedvector.jl:787 invalidated:
   backedges: 1: MethodInstance for extrema(::AbstractVector) at depth 0 with 1 children blocked InferenceTimingNode: 0.003433/0.005475 on DataFrames.get_stats(::Union{Base.SkipMissing, AbstractVector}, ::Vector{Symbol}) with 10 direct children
              2: MethodInstance for extrema(::AbstractArray) at depth 0 with 12 children blocked InferenceTimingNode: 0.000092/0.031425 on sort(::DataFrame, ::Symbol) with 2 direct children

 inserting string(a::Union{Char, SubString{String}, String}, b::InlineString) @ InlineStrings ~/.julia/packages/InlineStrings/HDs9F/src/InlineStrings.jl:600 invalidated:
   backedges: 1: MethodInstance for string(::String, ::Any) at depth 0 with 1 children blocked InferenceTimingNode: 0.002475/0.052416 on DataFrames.normalize_selection(::DataFrames.Index, ::Pair{<:Any, <:Union{Function, Type}}, ::Bool) with 11 direct children

 inserting all(f::Function, x::SentinelArrays.ChainedVector) @ SentinelArrays ~/.julia/packages/SentinelArrays/XvFr7/src/chainedvector.jl:775 invalidated:
   backedges: 1: MethodInstance for all(::DataFrames.var"#1#2", ::AbstractVector) at depth 0 with 4 children blocked 0.12726590999999998 inclusive time for 2 nodes

 inserting string(a::Union{Char, SubString{String}, String}, b::Union{Char, SubString{String}, String}, c::InlineString) @ InlineStrings ~/.julia/packages/InlineStrings/HDs9F/src/InlineStrings.jl:601 invalidated:
   backedges: 1: MethodInstance for string(::String, ::Vararg{Any}) at depth 0 with 4 children blocked InferenceTimingNode: 0.000146/0.000146 on Symbol(::String, ::Vararg{Any}) with 0 direct children
              2: MethodInstance for string(::String, ::Char, ::Any) at depth 0 with 2 children blocked InferenceTimingNode: 0.002475/0.052416 on DataFrames.normalize_selection(::DataFrames.Index, ::Pair{<:Any, <:Union{Function, Type}}, ::Bool) with 11 direct children
              3: MethodInstance for string(::String, ::String, ::Any) at depth 0 with 2 children blocked InferenceTimingNode: 0.002475/0.052416 on DataFrames.normalize_selection(::DataFrames.Index, ::Pair{<:Any, <:Union{Function, Type}}, ::Bool) with 11 direct children
              4: MethodInstance for string(::Any, ::String, ::Any) at depth 0 with 2 children blocked InferenceTimingNode: 0.001195/0.074849 on DataFrames.normalize_selection(::DataFrames.Index, ::Pair{<:Union{AbstractString, Signed, Symbol, Unsigned}, <:Union{Function, Type}}, ::Bool) with 9 direct children

 inserting in(x, A::SentinelArrays.ChainedVector) @ SentinelArrays ~/.julia/packages/SentinelArrays/XvFr7/src/chainedvector.jl:698 invalidated:
   backedges: 1: MethodInstance for in(::Symbol, ::AbstractVector{Symbol}) at depth 0 with 18 children blocked 0.42743727099999995 inclusive time for 6 nodes

 inserting convert(::Type{T}, x::SentinelArrays.ChainedVectorIndex) where T<:Union{Signed, Unsigned} @ SentinelArrays ~/.julia/packages/SentinelArrays/XvFr7/src/chainedvector.jl:211 invalidated:
   backedges: 1: MethodInstance for convert(::Type{UInt8}, ::Integer) at depth 0 with 26 children blocked 0.47230517399999994 inclusive time for 5 nodes

 inserting broadcasted(f::F, A::SentinelArrays.ChainedVector) where F @ SentinelArrays ~/.julia/packages/SentinelArrays/XvFr7/src/chainedvector.jl:929 invalidated:
   backedges: 1: MethodInstance for Base.Broadcast.broadcasted(::typeof(ismissing), ::AbstractVector) at depth 0 with 7 children blocked InferenceTimingNode: 0.000826/0.014839 on Core.kwcall(::NamedTuple{(:on,), Tuple{Symbol}}, leftjoin!::typeof(leftjoin!), ::DataFrame, ::DataFrame) with 1 direct children
              2: MethodInstance for Base.Broadcast.broadcasted(::typeof(eltype), ::Any) at depth 0 with 9 children blocked 0.08551727000000002 inclusive time for 5 nodes
              3: MethodInstance for Base.Broadcast.broadcasted(::typeof(last), ::AbstractVecOrMat{<:Pair}) at depth 0 with 1 children blocked InferenceTimingNode: 0.002322/0.086977 on DataFrames.broadcast_pair(::DataFrame, ::AbstractVecOrMat{<:Pair}) with 9 direct children
              4: MethodInstance for Base.Broadcast.broadcasted(::typeof(first), ::AbstractVecOrMat{<:Pair}) at depth 0 with 1 children blocked InferenceTimingNode: 0.002322/0.086977 on DataFrames.broadcast_pair(::DataFrame, ::AbstractVecOrMat{<:Pair}) with 9 direct children
              5: MethodInstance for Base.Broadcast.broadcasted(::typeof(string), ::Any) at depth 0 with 14 children blocked 0.12410217399999997 inclusive time for 2 nodes
              6: MethodInstance for Base.Broadcast.broadcasted(::typeof(first), ::Any) at depth 0 with 29 children blocked 0.19714425800000002 inclusive time for 8 nodes

It's worth noting that if you omit using CSV in that invs = @snoopr ... block, precompile_blockers returns an empty array (congratulations, invalidations do not affect "plain" DataFrames at all).

All of these invalidations come from just two packages, SentinelArrays and InlineStrings. If you want to resolve them, I think one easy approach would be to make both (or either, for a partial resolution) dependencies of DataFrames itself. If you want to try this and it doesn't work, let me know.

Otherwise, fixing these invalidations is almost identical to the standard ascend workflow except that data about the specific re-inferences also get recorded:

julia> tree = staletrees[end]
inserting broadcasted(f::F, A::SentinelArrays.ChainedVector) where F @ SentinelArrays ~/.julia/packages/SentinelArrays/XvFr7/src/chainedvector.jl:929 invalidated:
   backedges: 1: MethodInstance for Base.Broadcast.broadcasted(::typeof(ismissing), ::AbstractVector) at depth 0 with 7 children blocked InferenceTimingNode: 0.000826/0.014839 on Core.kwcall(::NamedTuple{(:on,), Tuple{Symbol}}, leftjoin!::typeof(leftjoin!), ::DataFrame, ::DataFrame) with 1 direct children
              2: MethodInstance for Base.Broadcast.broadcasted(::typeof(eltype), ::Any) at depth 0 with 9 children blocked 0.08551727000000002 inclusive time for 5 nodes
              3: MethodInstance for Base.Broadcast.broadcasted(::typeof(last), ::AbstractVecOrMat{<:Pair}) at depth 0 with 1 children blocked InferenceTimingNode: 0.002322/0.086977 on DataFrames.broadcast_pair(::DataFrame, ::AbstractVecOrMat{<:Pair}) with 9 direct children
              4: MethodInstance for Base.Broadcast.broadcasted(::typeof(first), ::AbstractVecOrMat{<:Pair}) at depth 0 with 1 children blocked InferenceTimingNode: 0.002322/0.086977 on DataFrames.broadcast_pair(::DataFrame, ::AbstractVecOrMat{<:Pair}) with 9 direct children
              5: MethodInstance for Base.Broadcast.broadcasted(::typeof(string), ::Any) at depth 0 with 14 children blocked 0.12410217399999997 inclusive time for 2 nodes
              6: MethodInstance for Base.Broadcast.broadcasted(::typeof(first), ::Any) at depth 0 with 29 children blocked 0.19714425800000002 inclusive time for 8 nodes


julia> root, newinfs = tree.backedges[end];   # newinfs is specific to `precompile_blockers` and lists the things that needed reinference due to invalidation

julia> root
MethodInstance for Base.Broadcast.broadcasted(::typeof(first), ::Any) at depth 0 with 29 children

julia> newinfs
8-element Vector{SnoopCompileCore.InferenceTimingNode}:
 InferenceTimingNode: 0.000154/0.000613 on map(::DataFrames.var"#548#549"{DataFrame}, ::Tuple{Colon, Symbol, Matrix{Pair{Symbol}}, Pair{Symbol, Symbol}, Pair{Vector{Symbol}, typeof(cor)}}) with 2 direct children
 InferenceTimingNode: 0.000166/0.000672 on map(::DataFrames.var"#744#745"{GroupedDataFrame{DataFrame}}, ::Tuple{Colon, Symbol, Matrix{Pair{Symbol}}, Pair{Symbol, Symbol}, Pair{Vector{Symbol}, typeof(cor)}}) with 2 direct children
 InferenceTimingNode: 0.001078/0.002028 on Core.kwcall(::NamedTuple{(:on, :makeunique), Tuple{Symbol, Bool}}, outerjoin::typeof(outerjoin), ::DataFrame, ::DataFrame) with 2 direct children
 InferenceTimingNode: 0.000185/0.000509 on map(::DataFrames.var"#554#555"{DataFrame}, ::Tuple{Symbol, Matrix{Pair{Symbol}}, Pair{Symbol, Symbol}, Pair{Vector{Symbol}, typeof(cor)}}) with 2 direct children
 InferenceTimingNode: 0.000190/0.000547 on map(::DataFrames.var"#740#741"{GroupedDataFrame{DataFrame}}, ::Tuple{Symbol, Matrix{Pair{Symbol}}, Pair{Symbol, Symbol}, Pair{Vector{Symbol}, typeof(cor)}}) with 2 direct children
 InferenceTimingNode: 0.000808/0.001443 on Core.kwcall(::NamedTuple{(:on,), Tuple{Symbol}}, semijoin::typeof(semijoin), ::DataFrame, ::DataFrame) with 2 direct children
 InferenceTimingNode: 0.000104/0.013342 on DataFrames.groupby(::DataFrame, ::Symbol) with 1 direct children
 InferenceTimingNode: 0.001051/0.177991 on Core.kwcall(::NamedTuple{(:on, :makeunique), Tuple{Symbol, Bool}}, innerjoin::typeof(innerjoin), ::DataFrame, ::DataFrame) with 3 direct children

julia> ascend(root)
Choose a call for analysis (q to quit):
     broadcasted(::typeof(first), ::Any)
       broadcast_pair(::DataFrame, ::AbstractVecOrMat{<:Pair})
 >       (::DataFrames.var"#548#549"{DataFrame})(::Matrix{Pair{Symbol}})
           map(::DataFrames.var"#548#549"{DataFrame}, ::Tuple{Matrix{Pair{Symbol}}, Pair{Symbol, Symbol}, Pair{Vector{Symbol}, typeof(cor)}})
...

You can't see that here, but the line pointed to with > has no red, but the line above it has red. That suggests broadcast_pair is being called with a non-inferrable second argument. But digging in quickly realizes this is a consequence of @nospecialize and so that may be what you want. Nevertheless, there are options, including:

check whether SentinelArrays really needs to create its own broadcasted or whether there's a more generic API they could specialize
check whether you really need broadcasting
check whether you can exclude the possibility of it being a SentinelArrays.ChainedVector. If you can restrict the subtypes of AbstractVecOrMat{<:Pair} that may be all that's required.

There's also

julia> root, newinfs = tree.backedges[end];

julia> root
MethodInstance for convert(::Type{UInt8}, ::Integer) at depth 0 with 26 children

julia> newinfs
5-element Vector{SnoopCompileCore.InferenceTimingNode}:
 InferenceTimingNode: 0.000061/0.391477 on DataAPI.describe(::DataFrame) with 2 direct children
 InferenceTimingNode: 0.000574/0.000574 on Core.kwcall(::NamedTuple{(:copycols, :keeprows, :renamecols), Tuple{Bool, Bool, Bool}}, manipulate::typeof(DataFrames.manipulate), ::DataFrame, ::Any) with 0 direct children
 InferenceTimingNode: 0.071626/0.071851 on DataFrames.var"#transform#747"(::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(transform), ::GroupedDataFrame{DataFrame}, ::Union{Regex, AbstractString, Function, Signed, Symbol, Unsigned, Pair, Type, All, Between, Cols, InvertedIndex, AbstractVecOrMat}, ::Vararg{Union{Regex, AbstractString, Function, Signed, Symbol, Unsigned, Pair, Type, All, Between, Cols, InvertedIndex, AbstractVecOrMat}}) with 4 direct children
 InferenceTimingNode: 0.000637/0.007018 on Core.kwcall(::NamedTuple{(:copycols, :keeprows, :renamecols), Tuple{Bool, Bool, Bool}}, ::typeof(DataFrames.manipulate), ::DataFrame, ::Any, ::Any, ::Any, ::Any) with 2 direct children
 InferenceTimingNode: 0.000533/0.001386 on Core.kwcall(::NamedTuple{(:renamecols, :threads), Tuple{Bool, Bool}}, ::typeof(select!), ::DataFrame, ::Any, ::Any, ::Any, ::Any, ::Any) with 1 direct children

julia> ascend(root)
Choose a call for analysis (q to quit):
     convert(::Type{UInt8}, ::Integer)
       fill!(::Union{Vector{Int8}, Vector{UInt8}}, ::Integer)
 >       select_transform!(::Base.RefValue{Any}, ::DataFrame, ::DataFrame, ::Set{Symbol}, ::Bool, ::Base.RefValue{Bool}, ::BitVector)
           _manipulate(::DataFrame, ::Vector{Any}, ::Bool, ::Bool)
             #manipulate#558(::Bool, ::Bool, ::Bool, ::typeof(DataFrames.manipulate), ::DataFrame, ::Any, ::Vararg{Any})
...
Choose possible caller of MethodInstance for fill!(::Union{Vector{Int8}, Vector{UInt8}}, ::Integer) or proceed to typed code:
   "/home/tim/.julia/packages/DataFrames/dgZn3/src/abstractdataframe/selection.jl", select_transform!: lines [873]
 > Browse typed code
select_transform!(::Ref{Any}, df::AbstractDataFrame, newdf::DataFrame, transformed_cols::Set{Symbol}, copycols::Bool, allow_resizing_newdf::Ref{Bool}, column_to_copy::BitVector) @ DataFrames ~/.julia/packages/DataFrames/dgZn3/src/abstractdataframe/selection.jl:773
Variables
  #self#::Core.Const(DataFrames.select_transform!)
  @_2::Base.RefValue{Any}
  df::DataFrame
  newdf::DataFrame
  transformed_cols::Set{Symbol}
  copycols::Bool
  allow_resizing_newdf::Base.RefValue{Bool}
  column_to_copy::BitVector
  @_9::Any
  @_10::Any
  #534::DataFrames.var"#534#539"
  #533::DataFrames.var"#533#538"
  #532::DataFrames.var"#532#537"
  #531::DataFrames.var"#531#536"
  @_15::Int64
  @_16::Int64
  @_17::Union{}
  nc::Any
  rows::Int64
  res_unwrap::Any
  newname::Any
  lr::Any
  startlen::Int64
  colnames::Any
  res::Any
  newres::DataFrame
  prepend::Any
  kp1::Any
  wfun::Base.RefValue{Any}
  fun::Union{Function, Type}
  col_idx::Any
  @_32::TypeVar
  @_33::TypeVar
  @_34::TypeVar
  @_35::TypeVar
  #535::DataFrames.var"#535#540"
  n::Any
  cn_multi::Any
  @_39::TypeVar
  @_40::Any
  @_41::Any
  @_42::Any
  @_43::Any
  @_44::Any
  @_45::Any
  @_46::Any
  @_47::Any
  @_48::Int64
  @_49::Any
  @_50::Any
  @_51::Any
  @_52::Any

Body::Union{Nothing, DataFrame}
      @ /home/tim/.julia/packages/DataFrames/dgZn3/src/abstractdataframe/selection.jl:777 within `select_transform!`
1 ───        Core.NewvarNode(:(@_9))
│            Core.NewvarNode(:(@_10))
│            Core.NewvarNode(:(#534))
│            Core.NewvarNode(:(#533))
│            Core.NewvarNode(:(#532))
│            Core.NewvarNode(:(#531))
│            Core.NewvarNode(:(@_15))
│            Core.NewvarNode(:(@_16))
│            Core.NewvarNode(:(@_17))
│            Core.NewvarNode(:(rows))
│            Core.NewvarNode(:(res_unwrap))
│            Core.NewvarNode(:(newname))
│            Core.NewvarNode(:(lr))
│            Core.NewvarNode(:(startlen))
│            Core.NewvarNode(:(colnames))
│            Core.NewvarNode(:(res))
│            Core.NewvarNode(:(newres))
│            Core.NewvarNode(:(prepend))
│            Core.NewvarNode(:(kp1))
│            Core.NewvarNode(:(wfun))
│            Core.NewvarNode(:(fun))
│            Core.NewvarNode(:(col_idx))
│     %23  = Base.indexed_iterate(@_2, 1)::Tuple{Any, Nothing}
│            (nc = Core.getfield(%23, 1))
│     %25  = nc::Any
...

and this one looks much more fixable; there are a lot of things that are probably Int that inference tags as ::Any.

The text was updated successfully, but these errors were encountered:

bkamins · 2022-12-27T13:55:18Z

Also in context of JuliaLang/julia#47184 the precompilation part of the package should be reviewed before the next release.

bkamins added this to the 1.5 milestone Dec 13, 2022

bkamins added the ecosystem Issues in DataFrames.jl ecosystem label Dec 13, 2022

timholy mentioned this issue Dec 14, 2022

Invalidations when loading DataFrames JuliaData/CSV.jl#1061

Open

bkamins mentioned this issue Feb 5, 2023

improve precompilation coverage #3285

Merged

2 tasks

bkamins closed this as completed in #3285 Feb 11, 2023

bkamins mentioned this issue Jul 2, 2023

Prepare for 1.6 release #3352

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Invalidations when loading CSV #3248

Invalidations when loading CSV #3248

timholy commented Dec 13, 2022 •

edited

Loading

bkamins commented Dec 27, 2022

Invalidations when loading CSV #3248

Invalidations when loading CSV #3248

Comments

timholy commented Dec 13, 2022 • edited Loading

bkamins commented Dec 27, 2022

timholy commented Dec 13, 2022 •

edited

Loading