API: allow dependent assignment?

[chris-b1]

Since kwarg order will be guaranteed in python 3.6, we could allow this...though would only want to try with a 3.6+ version check, otherwise code could more or less randomly work/not work, depending on dict order.

In [44]: df = pd.DataFrame({'a': [1,2,3]})

In [45]: df.assign(b=1, c=lambda x: x['b'] * 2)

[TomAugspurger]

xref: #9777 (comment)

This would be a great, but I want to think a bit about exposing a feature that would so easily lead to subtle differences between 3.5 and 3.6. Not sure how best to handle it :/

Either way, we could probably deprecate the sorting of keys (documentation only; can't print a warning each time someone uses .assign).

[TomAugspurger]

FYI, here's the current docs (in a warning block):

Since the function signature of assign is **kwargs, a dictionary, the order of the new columns in the resulting DataFrame cannot be guaranteed to match the order you pass in. To make things predictable, items are inserted alphabetically (by key) at the end of the DataFrame.
All expressions are computed first, and then assigned. So you can't refer to another column being assigned in the same call to assign.

[chris-b1]

Yeah, the back-compat is awkward, maybe this could be opt-in from an option, so it would at least raise on older pythons, although that's ugly too.
pd.set_option('ordered_assign', True) # raises on py < 3.6

[TomAugspurger]

@mrocklin thoughts on this? IIRC you wanted this initially to make things easier for dask? How much would break for you / dask users if we just removed the sorting of keys?

[mrocklin]

I think that Pandas should make whatever decisions make sense for Pandas. Dask.dataframe will adapt as necessary. I suspect that this won't cause a problem. Dask.dataframe will just try the assign operation on an empty dataframe to determine the column ordering. Whatever Pandas does we'll probably do automatically.

[shoyer]

Ultimately, I think this we should switch to this behavior and stop sorting.

But I don't see much of a rush. It might be better to wait until Python 3.6 is more widely used, even until pandas 2.0.

[shoyer]

I would also be comfortable just changing this for Python 3.6+. My guess is that there aren't that many folks using Python 3.6 in production yet, so this will probably cause minimal issues.

[TomAugspurger]

I think we should sort this out for 0.21. I propose

• No change to 2.7 - 3.5; order is sorted by key
• For 3.6+, we use the original order the user passed in
• We don't allow dependent assignment yet

Maybe in the future, when there's less 3.5 and lower code around, we can allow for dependent assignment.

[jorisvandenbossche]

+1 on your summary

[jreback]

sgtm on @TomAugspurger summary as well.