datetime utcnow deprecation leads to type confusion

[bdarnell]

Bug report

"Naive" and "aware" datetimes are functionally two distinct types, although they are not represented as such in the typeshed annotations. Specifically, operations such as subtraction and inequality comparisons fail if one operand is naive and the other is aware.

Python 3.12b2 deprecates many datetime methods including utcnow and utcfromtimestamp, which return naive datetimes. The suggested alternative is to use e.g. datetime.datetime.fromtimestamp(datetime.UTC), which returns an aware datetime. Applying this suggestion without breaking existing code is tricky.

For example, consider a function that takes a datetime and compares it to the current time:

def is_expired(d: datetime) -> bool:
    now = datetime.datetime.utcnow()
    return (now - d) > datetime.timedelta(days=30)

This function only accepts naive datetimes, but even though it has type annotations this is not obvious, and indeed its author may not have been aware of this concern. Applying the suggestion from the deprecation warning turns it into a function that only accepts aware datetimes, breaking every existing caller.

This could be a problem if the function occurs in a library as part of a public interface: if the library author does a search-and-replace to convert the utc-prefixed naive methods with their aware counterparts, the library may no longer have any test coverage that uses naive datetimes to detect this breakage. And even if the author is aware of this as a potential issue, the error occurs at the subtraction operation, not at the utcnow() call, which may be some distance away. I have a situation like this in Tornado and I don't know how to move away from deprecated methods while being confident that I'm not breaking application code. (I gather that the recommendation is to use if d.tzinfo is None: d = d.replace(tzinfo=datetime.timezone.utc), but how can I be sure that I have this incantation everywhere it needs to be?)

(Other minor gripes about this transition are that datetime.UTC, which is mentioned in the deprecation warning, is relatively new (3.11) and I need to use datetime.timezone.utc for compatibility with older versions of Python. And the whole thing is just awkwardly verbose.)

I would like to ask that this deprecation be reconsidered. The recommended fix is not semantically equivalent to the deprecated code, and type checkers are not sufficient to detect incorrect usage. In adapting to this deprecation in Python 3.12, libraries may be tempted to make changes that will adversely affect users of all python versions.

I think my ideal solution would be to make "naive" and "aware" datetimes into real types (at the level of type annotations if not actual classes), so that mixing the two is a type-checkable error. (I think that's mostly doable with mypy's overload support, although you'd probably end up falling back to an ambiguous case sometimes, effectively a Union[AwareDateTime, NaiveDateTime]. The type checker would then force you to check the status of any datetimes you use before subtracting or comparing them).

[hugovk]

cc @pganssle

[pganssle]

We are aware of this issue and indeed it is something that we considered when trying to decide on a wording for the deprecation warning.

I suspect that this will indeed cause some confusion, but I believe that for the most part it will be the kind of error that loudly raises errors rather than the silent confusion that we're replacing where the result of a calculation is just silently wrong. Given the history here there's no painless solution.

I would like to ask that this deprecation be reconsidered. The recommended fix is not semantically equivalent to the deprecated code, and type checkers are not sufficient to detect incorrect usage. In adapting to this deprecation in Python 3.12, libraries may be tempted to make changes that will adversely affect users of all python versions.

Yes, the point of the deprecation is that people are doing something that is semantically incoherent, and there's no way for us to automatically fix that from our side. The recommended fix is to change the semantics of their interface, possibly with a deprecation of their own. If they need to maintain the old interface but don't want to trigger our deprecation warnings, they can do datetime.now(datetime.UTC).replace(tzinfo=None).

I think my ideal solution would be to make "naive" and "aware" datetimes into real types (at the level of type annotations if not actual classes), so that mixing the two is a type-checkable error. (I think that's mostly doable with mypy's overload support, although you'd probably end up falling back to an ambiguous case sometimes, effectively a Union[AwareDateTime, NaiveDateTime]. The type checker would then force you to check the status of any datetimes you use before subtracting or comparing them).

Yes, if we had the option to restructure datetime we would absolutely do it, but unfortunately we are hemmed in by backwards compatibility. In the meantime, the closest you can get is to use DateType. Unfortunately, after many conversations with glyph and the typeshed/mypy teams, it turns out we can't introduce this from the top down in a backwards-compatible way without higher-kinded types. The current plan is to add a recommendation for people to voluntarily adopt DateType into the datetime documentation.

I am going to close this ticket because we were aware of these issues and ultimately decided that the action we took was the least bad course, but please don't take that as a dismissal of your concerns. I really appreciate that you took the time to test against the beta version, that you identified a real problem and took the time to put together a clear and cogent explanation of the issue.