Standardizing where the target column goes in `transform`, `insertcols` and `constructors`

[pdeffebach]

This issue is for discussion about standardizing the following syntaxes for creating new columns in data frames. We currently have two approaches, the symbol goes first or it goes second.

Constructor: df = DataFrame(:a => [1, 2], :b => [5, 6]). This syntax is currently only possible as the Pair version of syntax for keyword arguments.

insertcols!: This woks like a constructor, insertcols!(df, 3, :c => ["a", "b"]). This is not implemented only through standard keyword argyment syntax but rather through an actual Pair{Symbol, <:Any} method.

Future transform. After discussions in #1975, we have settled on having transform(df, :b => f => :d). Notice here that :d is at the end of this command, indicating "at the end of this operation, we create it all goes into a new column :d".

There is an inconsistency here. With insertcols, :b => v isn't putting v in the column :b the same way the transform syntax works.

I think we should standardize around having a Symbol at the end. But this makes a lot of complications. For instance, we might need to get rid of keyword arguments in the constructor, otherwise :b => v would be valid syntax due to default Julia syntax.

Consequently I think the issue merits discussion.

[bkamins]

otherwise :b => v would be valid syntax due to default Julia syntax.

It would be valid only as DataFrame(; :b=>v) which would not be that bad, which means that:

get rid of keyword arguments in the constructor,

would not necessarily have to happen.

We should also bear in mind that with Dict we have Dict(:a => 1, :b => 2) and in a DataFrame constructor taking a Dict we take keys to be column names.

Thirdly :b => v is not a valid syntax in select/transform so this is less problematic than if it were the case that they would be contradictory.

The planned valid syntax in select/transform is :b => fun which gets expanded to :b => fun => :b_fun, as noted elsewhere, we could extend this rule to make :b => v get expanded to something like Col(:b) => (x -> v) => :b if v is an AbstractArray and :b => (x -> v) => :b if v is some other type so this could be consistent and even allow injecting external data in select/transform.

[bkamins]

I have thought a bit more on :b => v issue. We cannot allow it in select/transform as it would be ambiguous with renaming syntax (also used in rename) and shorthand transformation syntax.

In the end - as discussed earlier - I came to the conclusion that we might have to settle with two separate syntaxes (that unfortunately share Pair type):

• key-value pairs, as in DataFrame and insertcols!
• transform specifications source => [fun] => destination in transformation functions (select/transform/combine/rename)

This is not ideal, but maybe this is a reasonable compromise that is not breaking old code. Then the decision would have to be made what to do with describe. But if you feel differently please comment.

[nalimilan]

I'm not too worried about this small inconsistency. Constructors and insertcols! are quite different from select and combine, as they don't allow transformations. Anyway there will always be an inconsistency, either with the Dict constructor and with keyword arguments, or with select and combine.

We could almost allow either order in constructors (:a => v and v => :a) since there's no ambiguity. But for now I'd say it's OK.

I have thought a bit more on :b => v issue. We cannot allow it in select/transform as it would be ambiguous with renaming syntax (also used in rename) and shorthand transformation syntax.

Why would it be ambiguous? We could have a special case when v is an AbstracArray. But anyway it would be really confusing as the order would be reversed compared with transformations. We could allow something special like nothing => v => :b if we really needed to support that.

[bkamins]

Just to comment on the possible ambiguity. It comes out from implicit broadcasting we allow:

julia> using DataFrames

julia> DataFrame(:a => :b)
1×1 DataFrame
│ Row │ a      │
│     │ Symbol │
├─────┼────────┤
│ 1   │ b      │

julia> DataFrame(:b => :a)
1×1 DataFrame
│ Row │ b      │
│     │ Symbol │
├─────┼────────┤
│ 1   │ a      │

[nalimilan]

Good catch. Well I guess we would have to stop implicit broadcasting if we wanted to allow both orders.

[bkamins]

Yes, that is why I raised the point of disallowing it recently, but we came to the conclusion that it is better to keep it. So it is, as usual, the matter of limited syntax space into which we want to pack many things (like bidding conventions in Bridge 😄). That is why, given the syntax of Dict that we should just keep "key-value Pair" and "transformation Pair" as the discrepancy between them is not that problematic.

[pdeffebach]

Sounds good! I have no problem with it as long as you guys are aware and agree on this syntax.

Good to move forward with other PRs and on to 1.0