3.11.5 Regression: Flags with custom __new__ can't be iterated

[ziima]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.11.5 (main, Aug 25 2023, 23:47:33) [GCC 12.2.0]

A clear and concise description of the bug:

A behavior of Flag changed in python 3.11.5. When Flag with custom __new__ is used in combination with functional API, it's members can't be retrieved by iteration over the class.

MWE:

from enum import IntFlag

Perm = IntFlag('Perm', {'R': 4, 'W': 2, 'X': 1})
print(tuple(Perm))


class LabeledFlag(IntFlag):
    def __new__(cls, value: int, label: str):
        obj = super().__new__(cls, value)
        obj._value_ = value
        obj.label = label
        return obj


LabeledPerm = LabeledFlag('LabeledPerm', {'R': (4, 'Read'), 'W': (2, 'Write'), 'X': (1, 'Exec')})
print(tuple(LabeledPerm))

The output in python 3.11.4:

(<Perm.R: 4>, <Perm.W: 2>, <Perm.X: 1>)
(<LabeledPerm.R: 4>, <LabeledPerm.W: 2>, <LabeledPerm.X: 1>)

The output in python 3.11.5:

(<Perm.R: 4>, <Perm.W: 2>, <Perm.X: 1>)
()

I suspect this commit to introduce the regression: 59f009e

A workaround for 3.11.5:

class LabeledFlag(IntFlag):
    def __new__(cls, value: int, label: str):
        obj = super().__new__(cls, value)
        obj._value_ = value
        obj.label = label
        return obj

    @classmethod
    def _missing_(cls, value):
        pseudo_member = super(_DnskeyFlagBase, cls)._missing_(value)
        if value in cls._value2member_map_ and pseudo_member.name is None:
            cls._value2member_map_.pop(value)
        return pseudo_member

Linked PRs

• gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ #108704
• [3.12] gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ (GH-108704) #108733
• [3.11] gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ (GH-108704) #108739

[ethanfurman]

This ended up being a combination of user error, program error, and some not-detailed-enough documentation.

An enum's custom __new__ should not use super(...).__new__(), because the parent's __new__ is only for lookups; either use the data type directly:

def __new__(cls, value, label):
    obj = int.__new__(cls, value)

or, if using an already defined enum type (i.e. LabeledFlag(IntFlag) and not LabeledFlag(int, Flag)):

def __new__(cls, value, label):
    obj = super()._new_member_(cls, value)

The code has been updated to raise a TypeError if super().__new__ is called, and a warning added to the docs.

[ethanfurman]

@ziima: the correct fix for your code:

obj = int.__new__(cls, value)

which works across all Python versions.

[ziima]

Works like a charm, thanks @ethanfurman.

I think I actually tried that, but I may had had other error in there at that moment.