ENH: Enable using a boolean loc in a non-boolean index

[alonme]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

When using loc to access a boolean value on a DataFrame which has an index that contains both booleans and non-booleans (so the dtype of the index is not bool) an error is raised

import pandas as pd

df = pd.DataFrame({'a': [True, False, "Missing"]}).set_index("a")
df.loc[True]

raises

KeyError: 'True: boolean label can not be used without a boolean index'

I am unsure why this check is performed in _validate_key, but this was an unexpected behavior for me, especially as this works with other combinations of types (int and string for example)

Feature Description

I am not sure why this is currently not supported, so i am unsure if there is a need for a new api or we can use the current one for .loc

Alternative Solutions

df = pd.DataFrame({'a': [True, False, "Missing"]}).set_index("a")
df[df.index == True].loc[True]

This is my current solution, but i don't believe this is a good one

Additional Context

No response

[Rylie-W]

take

[phofl]

@jbrockmendel thoughts here? I am not really sure if this is a good idea since normally Boolean elements are a mask

[jbrockmendel]

I'd expect this to work. The desired behavior seems pretty unambiguous.

[phofl]

If you provide a list of booleans it gets a bit sketchy?

[jbrockmendel]

If you provide a list of booleans it gets a bit sketchy?

Yah. I think in principle we wouldn't allow a mask with .loc.

[Rylie-W]

If you provide a list of booleans it gets a bit sketchy?

Yah. I think in principle we wouldn't allow a mask with .loc.

So in the future, a list of bool elements could be considered as valid for loc only if the index column(s) has one or more bool elements, otherwise it would raise KeyError. And it doesn't check if the length of the list of booleans is the same as the row axis.

Do I understand it right?

[phofl]

Not allowing a mask would force chained indexing though? How would you filter the columns with a mask then?

[jbrockmendel]

Not allowing a mask would force chained indexing though?

Good point. That's a tough one.

Not allowing a mask would force chained indexing though? How would you filter the columns with a mask then?

wouldn't iloc work?

[phofl]

Fair but not ideal I guess for row selection

[alonme]

Any news on this?

[Rylie-W]

I have implemented the feature and am working on fixing and creating the corresponding unit tests.

As others have previously pointed out, there may be ambiguity when the input is a boolean array. One possible solution is to treat the boolean array as labels to search first. If there are no boolean elements in the index column, we can then check if its length matches that of the DataFrame object. If it does match, it will be treated as a mask, otherwise an error will be thrown. And I'll implement it if the maintainer agree with the solution or have a better idea.

[jbrockmendel]

Not allowing a mask would force chained indexing though? How would you filter the columns with a mask then?

If we eventually get laziness in the filtering (xref #52980) then the chained indexing wouldn't be so much of a problem (though i guess it still would for setitem)

[phofl]

Chained indexing and cow doesn’t work, so this would be tricky but could be worked around with laziness I guess

[Charlie-XIAO]

Hi @Rylie-W, as you have mentioned in #53267, the following should work (though you haven't tried it due to installation issues).

if isinstance(key, bool) and not (
    ax.dtype.kind in ["b", "O"]
    or isinstance(ax, MultiIndex) and ax._levels[0].dtype.kind in ["b", "O"]
):
    raise KeyError(
        f"{key}: boolean label can not be used without a boolean index"
    )

I have confirmed that with the change above, we can have the following (IMO is the desired behavior).

>>> import pandas as pd
>>> pd.__version__
'2.1.0.dev0+796.gb2bb68a88c.dirty'
>>> df = pd.DataFrame(
...     {"a": [0, 1, True, False, "Missing"], "b": [1, 2, 3, 4, 5]}
... ).set_index("a")
>>> df
         b
a
0        1
1        2
True     3
False    4
Missing  5
>>> df.loc[1]
      b
a
1     2
True  3
>>> df.loc[True]
      b
a
1     2
True  3

Are you making a PR for this? Otherwise I can help as well.

[Rylie-W]

Hi @Rylie-W, as you have mentioned in #53267, the following should work (though you haven't tried it due to installation issues).

if isinstance(key, bool) and not (
    ax.dtype.kind in ["b", "O"]
    or isinstance(ax, MultiIndex) and ax._levels[0].dtype.kind in ["b", "O"]
):
    raise KeyError(
        f"{key}: boolean label can not be used without a boolean index"
    )

I have confirmed that with the change above, we can have the following (IMO is the desired behavior).

>>> import pandas as pd
>>> pd.__version__
'2.1.0.dev0+796.gb2bb68a88c.dirty'
>>> df = pd.DataFrame(
...     {"a": [0, 1, True, False, "Missing"], "b": [1, 2, 3, 4, 5]}
... ).set_index("a")
>>> df
         b
a
0        1
1        2
True     3
False    4
Missing  5
>>> df.loc[1]
      b
a
1     2
True  3
>>> df.loc[True]
      b
a
1     2
True  3

Are you making a PR for this? Otherwise I can help as well.

Yes, I'm working on that and adding some tests for it. Thanks for the confirmation!