Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate accepting None in pd.concat #57846

Open
twoertwein opened this issue Mar 14, 2024 · 3 comments
Open

Deprecate accepting None in pd.concat #57846

twoertwein opened this issue Mar 14, 2024 · 3 comments
Labels
Deprecate Functionality to remove in pandas Needs Discussion Requires discussion from core team before further action Reshaping Concat, Merge/Join, Stack/Unstack, Explode

Comments

@twoertwein
Copy link
Member

pd.concat accepts iterables that may contain Series, DataFrame and None, where None are simply ignored.

pd.concat([None, series, None, series, None]) # works, same as pd.concat([series, series])
pd.concat([None]) # raises

It would be great to deprecate/remove accepting Nones. This will help to better annotate concat pandas-dev/pandas-stubs#888

Pandas-stubs currently has to choose between either false-positives (accept concat([None]) ) or false-negatives (reject some concat(Iterable[Any/Unknown]) that could succeed).

@phofl
Copy link
Member

phofl commented Mar 16, 2024

I am -1 on simply deprecating this because of typing reasons. Let's think about use cases before we make a decision. We definitely can't do this without a deprecation

@twoertwein
Copy link
Member Author

I agree, it would be good first to understand which use case supporting None enables!

@Dr-Irv found a way to get it to type check without having to decide between false negatives/positives :)

@lithomas1 lithomas1 added Deprecate Functionality to remove in pandas Reshaping Concat, Merge/Join, Stack/Unstack, Explode Needs Discussion Requires discussion from core team before further action labels Mar 18, 2024
@Multihuntr
Copy link

If the function should accept None's in objs, then it seems most natural to me for it to return an empty DataFrame if objs contains only None's.

This is backwards compatible and would also resolve the typing issue.

A hypothetical use case for Iterable[DataFrame | None]: you have a list of files to load and parse and you want to put them into a single DataFrame, but not all of the filenames you've been given exist, or maybe parsing is allowed to fail. I could see someone implementing this as a function that returns DataFrame | None, and putting that into a list comprehension. So a situation where you just want to put everything you have into one table, and if everything you have is None, then your table is empty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Deprecate Functionality to remove in pandas Needs Discussion Requires discussion from core team before further action Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants