Why isn't `df.col .= v` in-place?

[gustafsson]

This issue is a question on the most recent release notes

On Julia 1.7 or newer broadcasting assignment into an existing column of a data frame replaces it. Under Julia 1.6 or older it is an in place operation. (#3022)

I expected df.col .= v to broadcast and do in-place assignment. But I see that's no longer the case in Dataframes.jl 1.4.

The recent release broke some code of mine (I must have missed any deprecation warnings). A simple workaround was coltemp = df.col; coltemp .= v but I don't understand the reason for the new behaviour. To me this seems to make DataFrame inconsistent with other containers in Julia and left me wondering why this inconsistency would be a wanted one.

This issue equally applies to df[!, :x] .= v.

Compare:

julia> x = [1, 2, 3]
3-element Vector{Int64}:
 1
 2
 3

julia> x .= 1.5
ERROR: InexactError: Int64(1.5)

Whereas

julia> df = DataFrame(x=[1, 2, 3])
3×1 DataFrame
 Row │ x
     │ Int64
─────┼───────
   1 │     1
   2 │     2
   3 │     3

julia> x = df.x;

julia> df.x .= 1.5
3-element Vector{Float64}:
 1.5
 1.5
 1.5
 
julia> x === df.x
 false

As advertised df.x .= 1.5 does not work in-place but replaces the column, even with a new type.

If I put the vector in any other container, say, a Dict, NamedTuple or struct

julia> dt = Dict(:x=>[1, 2, 3])
Dict{Symbol, Vector{Int64}} with 1 entry:
  :x => [1, 2, 3]

julia> dt[:x] .= 1.5
ERROR: InexactError: Int64(1.5)

julia> nt = (x = [1, 2, 3],)
(x = [1, 2, 3],)

julia> nt.x .= 1.5
ERROR: InexactError: Int64(1.5)

julia> struct S
       x
       end

julia> s = S([1, 2, 3])
S([1, 2, 3])

julia> s.x .= 1.5
ERROR: InexactError: Int64(1.5)

They all behave the same. But a DataFrame behaves differently. Why is that?

---

The docs state "Since df[!, :col] does not make a copy" which to me makes it unexpected that it would create a new column rather than modifying the existing one.

---

For the use case of "create/replace column" we have df.x = v (akin to s.x = v or dict[:x] = v). Would there be any adverse side-effects of letting = broadcast scalars into new/replaced columns?

---

I understand there was a decision a year ago (#2804) to make df.x .= v work like d[!,:x] .= v but wouldn't a change to instead make df[!,:x] .= v work like df.x .= v have been more consistent with how containers in Julia typically work?

[bkamins]

There are three parts of the explanation of the current behavior (unfortunately this is a bit complicated).

Part 1. As opposed to other containers in Julia DataFrames.jl assumes it gets ownership of the columns stored in it. For example if you store a vector in a Dict this Dict does not claim ownership of the column. With DataFrame it is different. Columns stored in a DataFrame are owned by it. What the user can do is only to ask for them to be exposed in a copying or non-copying way. This decision was made to, in general, improve code safety.

In general if you say that

The recent release broke some code of mine

This is unfortunate, and sorry for this. As you noticed we have had this deprecation for a long time and in previous release announced that this change would be made.

However, this means that your code was unsafe. It does not say that it is not allowed to write unsafe code, but it is recommended not to write it unless you have some performance critical parts. The only scenario when this change would break code is when you store column col in a DataFrame but at the same keep it somewhere else (and rely on the fact that this stored elsewhere vector is identical to the vector in a DataFrame). Since DataFrame owns the column you should not do this. If you want a column from a DataFrame you should always query it to get it (as only the owner of some object gives you a single version of truth of the value of this object).

Part 2. The docs say "Since df[!, :col] does not make a copy" indeed, but this is for getting data from a data frame, not for setting data in a data frame. All the rules for indexing are described here. In general you can have three kinds of operations related to indexing:

• indexing access (getindex)
• indexing write (setindex!)
• broadcasted assignment (broadcast!)

Each of them requires a bit different treatment.

Would there be any adverse side-effects of letting = broadcast scalars into new/replaced columns?

It would be near impossible to implement it correctly + it would be inconsistent with the whole design of Julia that requires explicit broadcasting.

Part 3. Now let me discuss broadcasted assignment solely. We need two behaviors. One that updates the column in-place. It is currently:

df[:, :col] .= v

and the other that allocates new column. This is currently:

df[!, :col] .= v
df.col .= v

If we applied your recommendation there would be no way to do the second operation (all three operations would have the same effect.

However, users need both. I.e. the second behavior (allocating new column) is a must have and quite often needed. See e.g.:

julia> df = DataFrame(col=[-1, 1])
2×1 DataFrame
 Row │ col
     │ Int64
─────┼───────
   1 │    -1
   2 │     1

julia> df[:, :col] .+= 0.5
ERROR: InexactError: Int64(-0.5)

julia> df[!, :col] .+= 0.5
2-element Vector{Float64}:
 -0.5
  1.5

julia> df.col .+= 0.5
2-element Vector{Float64}:
 0.0
 2.0

Users just want such operations to work. And we had to give them a way to do it. Also, note (you probably saw this example already) that sometimes instead of an error you get just an unexpected result:

julia> df = DataFrame(a=[0])
1×1 DataFrame
 Row │ a
     │ Int64
─────┼───────
   1 │     0

julia> df[:, :a] .= 'a'
1-element view(::Vector{Int64}, :) with eltype Int64:
 97

julia> df
1×1 DataFrame
 Row │ a
     │ Int64
─────┼───────
   1 │    97

julia> df = DataFrame(a=[0])
1×1 DataFrame
 Row │ a
     │ Int64
─────┼───────
   1 │     0

julia> df.a .= 'a'
1-element Vector{Char}:
 'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)

julia> df
1×1 DataFrame
 Row │ a
     │ Char
─────┼──────
   1 │ a

(the second behavior is what people typically expect)

Now what is the difference at its core. In general if you write ! or use getproperty DataFrames.jl indexing for assignment always behaves as if the column to which you write were not present in a data frame yet. So you do not have any side effects independent if it is present or not present in a data frame. Most often this is something that users want (if you write generic code often you want to write some expression that has fully predictable effect). On the other hand if you use : in assignment DataFrames.jl behaves like Base Julia behaves if the column is present (i.e. it updates things in-place).

So in short, from the other perspective: now ! and getproperty always work the same way - which is also a desirable property.

Now you might ask why it was implemented only in DataFrames.jl 1.4. The fact is that we wanted to have these rules already in DataFrames.jl 1.0 release, but it was not possible, because of limitations of Base Julia. Making this change was only possible after Julia 1.7 was released. That is why we had a long deprecation period. We announced that this change would be made in earlier releases, e.g. if you check DataFrames.jl 1.3 release notes you can read:

Planned changes
In DataFrames.jl 1.4 release on Julia 1.7 or newer broadcasting assignment into an existing column of a data frame will replace it. Under Julia 1.6 or older it will be an in place operation.

[gustafsson]

Thanks a bunch for writing out this explanation for me @bkamins. Really appreciate it.

your code was unsafe.

No argument there. Using DataFrames in the first place inherently type-unstable.

It would be near impossible to implement it correctly

Is that a challenge? ;)
I would attempt it, but not unless it's wanted.

it would be inconsistent with the whole design of Julia that requires explicit broadcasting.

Maybe the "whole design of Julia" is subjective? In my mental model of Julia it's clearly breaking the "whole design of Julia" to deviate from how all other containers work and let .= allocate something new instead of doing in-place broadcasted assignment. In my mental model of DataFrames as a collection of named columns with equal length it doesn't take any stretch of imagination to (re)create a column by giving it some initial value that needs to be materialized (such as with df.col = 1:3).

I also recognize that there are individual differenes in mental models of what a language or library represents, and that mine is not necessarily a typical (or even "correct") one.

Users just want such operations to work. And we had to give them a way to do it
what people typically expect

Fair enough. Personally I would expect DataFrame to behave like other containers in Julia, and columns in a DataFrame to behave likte Vectors (or AbstractVectors in general) behave in Julia. But I guess what people typically expect is more formed from pandas or R dataframes rather than Julia. It seems to be that's what this is really about? As a long time user of Julia (and typed languages) I understand that I'm not a typical user and it's important to get more people on board.

---

Thanks again! DataFrames.jl is a great library that I use daily :)

[bkamins]

Using DataFrames in the first place inherently type-unstable.

The issue is not type stability, but object ownership. Julia does not have a well defined concept of ownership of a value (like Rust does), so it is more of a convention.

Let me start with explanation of the origin of the ownership concept related to a most common bug in code using DataFrames.jl.

Since you can extract a column as a vector from a data frame you can resize it. If you resize such a vector you invalidate a data frame (as every column of a data frame must have the same number of elements; note that this is a key difference from other containers - e.g. Dict of vectors does not have a restriction that every vector stored in it has the same number of rows).

It might seem an innocent issue and not very problematic, but in practice it is the opposite. Users, in the past, tended e.g. to create one data frame from another data frame without copying columns and then they were surprised that when they mutated one then the other was mutated. The most problematic case is when rows were added/removed as it invalidated the code. This was the most common problem in the past.

To resolve it we introduced a rule of column ownership and copycols kwarg. By default copycols=true in many functions so when you work on columns of a data frame a newly created object does not alias columns with the source.

However, we allow for copycols=false as in some performance critical code this is needed. However, users should be aware that this is an unsafe operation and should be super careful when working with such objects.

Similarly df.col and df[!, :col] operations are non-copying when getting data and therefore they are unsafe. If you extract a column from a data frame this way it is recommended that this is only done for short-lived objects and when you are sure that you will not break anything in the parent (a most common case when this is needed is when using function barriers).

Now, why all this is done? Most of the DataFrames.jl users are not Computer Science experts. As you have said - many of them are entry level programmers, not necessarily even understanding all the details how things work underneath. Therefore we design DataFrames.jl to be as simple to use as possible by them. For example the operation df.col .+= 0.5 when :col is an integer column was one of the common cases when entry-level users did not understand why it fails and it is common.

Is that a challenge? ;) I would attempt it, but not unless it's wanted.

DataFrame constructor, insertcols!, and combine and friends function define a concept of pseudo-broadcasting. This is an approximation of broadcasting used in some of the functions in DataFrames.jl (you can read about how it is defined in documentation). However, it is quite hard to cover every case of broadcasting fully correctly (especially that broadcasting assignment is defined in Base Julia for objects that exist and in DataFrames.jl broadcasting can be also used in the case of non-existing object when you create a new column). For example you write "scalar" and the concept of a scalar is not well defined in Julia. Users can define many strange types with complex expected behavior. For example what should be done if user passes an array of dimension higher than 1 would be unclear (should it error or should such object be treated as scalar). For these reasons, except for the cases that support for pseudo-broadcasting mentioned above, we try to just rely on broadcasting rules that Base Julia defines.

to deviate from how all other containers work and let .= allocate something new instead of doing in-place broadcasted assignment.

To some extent you are right.
However, if we wanted to be consistent how other containers work with .= we should error if you wanted to add a column using broadcasting. Base Julia does not allow broadcasting assignment into a non-existing object. However, this is something that we needed to define in DataFrames.jl as adding a column to a data frame is one of the most common operations users want.

As a long time user of Julia (and typed languages) I understand that I'm not a typical user and it's important to get more people on board.

I think this is a good summary of the problem here. As I have said above, we need to make DataFrames.jl relatively friendly to users that are not used to strongly-typed languages (and this is the majority - as you have commented - coming from R/Python and wanting things to mostly "just work").

In fact initially DataFrame object design was what you propose "a collection of columns", but over the years we needed to add and twist many features to make things newbie-users life easier.

[bkamins]

By the way - in some style guides of corporate users there is a rule is that using df.col and df[!, :col] is discouraged. The preference is given to df[:, :col] (which works fully consistently with Base Julia matrices, except for that it allows adding columns for convenience). The non-copying column access is recommended only for exceptional cases of performance-critical code.

[bkamins]

Finally - your feedback is much appreciated. I am planning to write a blog post on this change (as it was the most painful decisions we needed to make in DataFrames.jl design in general) and the discussion with you is an important in the preparation process.

Also note that historically df[:col] was supported (in times when data frame was considered to be a collection of columns) but this support was dropped. From the puristic perspective we ideally should drop df.col support also with the current design (as this is the last remaining element from the "collection of columns" thinking about DataFrame), but the df.col syntax is just too convenient for users + it has autocompletion support so we keep it.

[gustafsson]

Forgive me if I digress at times, but I find the conversation interesting:

The issue is not type stability, but object ownership.

Right. The issue I encountered arose from getting a different type under the new behaviour but the issue we're focusing on here is rather one of ownership.

I also think object ownership is a larger issue in Julia, not specific to DataFrames. Akin to how f!(x) is expected to modify x but f(x) might also change x, intentionally or not. I wouldn't mind having the option to specify some sort of constness in general in Julia. I often like to use type annotations to communicate intent (together with documentation and naming conventions). One could envision a ConstArray or FixedLengthArray that wraps a normal array but implements none or only a few of the mutable functions. I'm imagining a struct Column <: AbstractVector may similarily skip the functions that change the size but still allow setindex! and thus might help in making DataFrame code more safe?

copycols=true

I think this was a great decision. I appreciate a simple and clear option to be able to not get a copy when I don't want one.

in some style guides of corporate users there is a rule is that using df.col and df[!, :col] is discouraged

I'm personally an advocate for only using DataFrames while manipulating data and then to stop using the DataFrame as soon as you can. I.e to not let the DataFrame be some generic container with unknown columns that you pass around. This is related to the lack of type stability / lack of schema, not necessarily for performance but for correctness. (note I don't think DataFrames need a schema). Which brings me to:

function barriers

Apart from the simple case of f(d.col) I've also found it really useful to convert a whole DataFrame to a NamedTuple in order to introduce type stability before passing a DataFrame through a function barrier:

NamedTuple(d::AbstractDataFrame) = (;zip(Tables.columnnames(d), Tables.columns(d))...)

This let's me work with data from the table in a dynamic way while writing performant code. And I can compute gradients with Zygote or ForwardDiff. Exceptional? Performance-critical? Maybe both. But it can also be used to effectively ensure a table schema for the purpose of correctness rather than performance.

For example the operation df.col .+= 0.5 when :col is an integer column was one of the common cases when entry-level users did not understand why it fails and it is common.

v .+= 0.5 with v::Vector{Int} also fails. If this is an issue it looks to me like more of an issue with the Julia language rather than an issue related to DataFrames.jl.

However, if we wanted to be consistent how other containers work with .= we should error if you wanted to add a column using broadcasting.

Makes sense to me. That would be the expected behaviour of in-place assignment to something that doesn't exist. Like elementwise assignment to a Matrix outside of its dimensions (and I gather that consistency with matrices is expected and wanted?)

However, this is something that we needed to define in DataFrames.jl as adding a column to a data frame is one of the most common operations users want.

julia> df.col = 1
ERROR: ArgumentError: It is only allowed to pass a vector as a column of a DataFrame.
julia> df[!, :col] = 1
ERROR: MethodError: no method matching setindex!

I think users would also accept if this created a new/replaced column and filled it with the scalar (per the pseudo-broadcasting logic) instead of the error message even if they didn't use broadcast assignment. In much the same way the constructor accepts scalars (i.e _preprocess_column(v, nrow(df), copycols)). I even think there's a case for supporting assignment of scalars without broadcasting for conistency with the constructor alone.

we try to just rely on broadcasting rules that Base Julia defines.

Yes! That's really awesome.

Also note that historically df[:col] was supported (in times when data frame was considered to be a collection of columns) but this support was dropped.

Those were the days. I might still be stuck in some ancient mindset.

---

Looking forward to that blog post!

[bkamins]

I've also found it really useful to convert a whole DataFrame to a NamedTuple

There is a predefined function to do this: Tables.columntable. Indeed this is often useful if data frame is not super wide.

[CameronBieganek]

I haven't been following this change very closely, so take what I say with a grain of salt. However, I find this design choice a little befuddling.

Julia does not have a well defined concept of ownership of a value (like Rust does), so it is more of a convention.

Exactly. This was a design choice. It's quite natural to have variables aliased with DataFrame columns. To take one simple case:

x = [1, 2];
df = DataFrame(x=x; copycols=false)

Now x and df.x both refer to exactly the same vector: x === df.x. So it is quite odd that the same operation behaves differently depending on whether we reference the vector via x or df.x. I don't think this has anything to do with what containers should or should not do. The crux of the issue is that df.x is an AbstractVector. Every DataFrame column is an AbstractVector. So what you've done is change the behavior of in-place broadcasting on AbstractVector in one specific context. If you had a Column type you could do whatever you want, but don't change the broadcasting behavior for AbstractVector.

I think the fact that a change to Base Julia was needed in order to implement this design is a hint that this might not have been the best design choice.

[CameronBieganek]

So it is quite odd that the same operation behaves differently depending on whether we reference the vector via x or df.x.

To elaborate on this, foo(x) and foo(df.x) should behave identically for any function foo. Why should broadcasting be magical?