Working with attrs 22.1.0

[layday]

The 22.1.0 release of attrs introduced a typing-only protocol (not available at runtime) with one member, __attrs_attrs__, to identify attrs classes by (python-attrs/attrs#890). The protocol was used to annotate a number of core functions which take an attrs class as an argument. Previously, the attrs mypy plug-in was updated to "inject" the __attrs_attrs__ attribute to classes transformed by attrs' decorators (python/mypy#8578). For the discussion that motivated this change, see python-attrs/attrs#952.

Is there something that Pyright could do to accommodate attrs here without burdering the users of the library - perhaps by extending dataclass_transform with the option to register a distinguishing attribute or a protocol that the transformed class is expected to conform to? I suppose the proper way to express this relationship is with an intersection of the original class and the protocol; but typing support for intersections seems to be a long way away.

cc @Tinche

[erictraut]

We strictly adhere to the principle that pyright does not implement any special-case knowledge of third-party libraries. If you want to extend dataclass_transform to include additional functionality, that would need to be done through the normal PEP discussion and ratification process so it gets the appropriate scrutiny and other Python type checkers can implement it in turn.

Is the injected __attrs_attrs__ attribute something that a typical attrs users would ever access? Or is it primarily used internally to the attrs implementation?

Could you provide a code sample that demonstrates the current undesirable behavior for attrs users when they're using pyright / pylance?

[layday]

Is the injected attrs_attrs attribute something that a typical attrs users would ever access? Or is it primarily used internally to the attrs implementation?

It's primarily used internally, but see python-attrs/attrs#952 (comment) and the following reply.

Could you provide a code sample that demonstrates the current undesirable behavior for attrs users when they're using pyright / pylance?

from attr import asdict, astuple, define, fields, fields_dict, validate

@define
class A:
    foo: int


# All of the following will produce an error

asdict(A(1))
astuple(A(1))
fields(A)
fields_dict(A)
validate(A(1))

[erictraut]

Thanks, that helps clarify the problem.

Yeah, I don't think I would have recommended going down the route of creating a protocol that exposes an internal implementation detail like __attrs_attrs__, but it sounds like that damage is already done. I'm a bit frustrated because we did all of this work with dataclass_transform to make attrs work better with the type standards, and now the library is again introducing behaviors that deviate from the standards.

As you point out, this could be supported through the use of an intersection type, but that solution isn't available to us currently.

If attrs used a metaclass rather than a function decorator, that would have provided a potential solution.

We've talked about adding support for other general "class transforms" that support decorators that "inject" new methods and attributes into an existing class, but we never bottomed out on that discussion.

I don't see a good solution here with the existing tools available to us in the type system. I'll think about it more and let you know if I come up with any new ideas.

[Tinche]

I'm the person who implemented Mypy recognizing __attrs_attrs__. This is the start of an effort I have big plans for.

Mypy does more than just recognize the existence of this attribute, it knows it's a named tuple with an attribute for each class attribute, and the exact type of that attribute. Maybe this will make it clearer:

@define
class MyClass:
    a: int

reveal_type(MyClass.__attrs_attrs__.a)  # type is attr.Attribute[int]

The very next step is wrapping this attribute in a nice function so it's not an eyesore to use, so it'll be:

reveal_type(fields(MyClass).a)  # type is attr.Attribute[int]

I've actually posted to the typing list about this a few months ago and gotten crickets.

@erictraut I recognize your pain, but you need to see our side here too. For example, Python currently has no ORMs that can do statically-checked projected queries and if I had to go through the typing PEP committee, who knows when it would have happened. With these efforts it's a few Mypy PRs away, at least for attrs classes. We need to be able to innovate for the ecosystem.

That said, I'd be happy to discuss my plans over email, Zoom or on the typing meetup.

[erictraut]

Hi @Tinche, thanks for providing that perspective.

I think it's a good to experiment and innovate, and using mypy's plugin mechanism makes sense.

I'd like to offer a couple of suggestions for you to consider as you proceed with your experimentation.

1. Mypy users represent a small fraction of the overall Python developer base. There are other type checkers and language servers used by many Python developers. You'll serve the broader ecosystem better if your modifications don't degrade the experience for users of other tools. It sounds like changes were made to the attrs library types for version 22.1.0 that result in misleading or incorrect errors for developers who are using standards-compliant language servers and type checkers. That is, if you use anything other than mypy with your latest plugin logic, you will receive false positive errors. That's less than ideal.

2. As you explore ways to extend support for attrs, please think about how such functionality could be generalized and standardized within the Python static type system. In other words, don't design something that will work only for attrs. That will lead to long-term pain and delay the adoption to the broader ecosystem because we'll need to unravel it and then design a new mechanism that works for the more general case.

I think it would be great if you could present your latest thinking in a typing meetup. That would bring more visibility to the effort and give other members of the typing community an opportunity to weigh in with suggestions and assistance.

[Tinche]

Cool, I'll talk to the meetup organizers and see if I can get something for next month. Thanks!

[layday]

Hi @erictraut, would you be able to offer your input at python-attrs/attrs#999 if you've got time?

[erictraut]

This solution relies on a somewhat fragile assumption.

if MYPY:
    AttrsInstance_ = AttrsInstance
else:
    AttrsInstance_ = Any

As you know, type checkers don't evaluate runtime values, only types. So a type checker doesn't know which of these two branches will be taken at runtime. When pyright resolves an imported symbol that is defined conditionally (like the one above), it applies a number of heuristics to try to pick the "right" one. These heuristics take into account ordering (typically it uses the last assignment) and whether the assignments are within try or except clauses (it prefers the try over except). The behavior for type checkers is not defined in this case, and these heuristics will likely continue to evolve. That's why I say this is somewhat fragile.

Thinking about other ways you could address this... Let me first make sure I understand the motivation behind the use of the AttrsInstance protocol. I presume this is to provide type checking for functions that assume an input argument derives from a class has been decorated with @attrs. Is this motivated by empirical evidence that developers are often incorrectly passing non-attrs objects to these functions, and you want to provide a way to prevent this class of bugs? Or is this just a theoretical problem?

[layday]

I think @Tinche is in a better position to answer. But if the parameter does not interact with the mypy plug-in in some manner, I assume it's simply to provide the same typing guarantees that subclassing would provide.