A question about import cycles

[bluetech]

I investigated cyclic imports and thought about adding a warning to mypy about them. I looked a bit and found that mypy logs SCCs with more than one module:

mypy -v --no-incremental src |& grep 'Processing SCC of size'

I noticed that mypy considered the following (common?) pattern in some packages to be cyclic:

[file package/_utils.py]
def some_helper(): pass

[file package/_a.py]
# Alternative 1
from . import _utils
# Alternative 2
# from ._utils import some_helper

class A: pass

[file package/__init__.py]
# Expose public interface
from ._a import A

Alternative 1 causes a cycle:

LOG:  Processing SCC of size 2 (package.a package) as inherently stale with stale deps (builtins)

Alternative 2 doesn't.

I wonder if, in the absence of some redefinition of _utils in __init__.py (e.g. _utils = 5), Alternative 1 can avoid creating a dependency of package._a on package?

#6312 is similar but there the two files actually import each other.

mypy version 0.720.

[JukkaL]

Mypy uses SCCs and import dependencies to decide which modules need to be type checked in incremental mode. In alternative 1, if package/__init__.py is changed to have _utils = 5, package/_a.py must also be re-processed, so the cyclic dependency is technically correct. In alternative 2 the contents of package/__init__.py have no significance to package/_a.py, so the dependency can be omitted.

If we'd only want to report 'real' cyclic imports, perhaps the spurious-looking dependency in alternative 1 could be tagged as a 'phantom' dependency. Phantom dependencies would then be ignored when determining if there are real cyclic imports.

[bluetech]

Thanks for the reply.

In alternative 1, if package/init.py is changed to have _utils = 5, package/_a.py must also be re-processed, so the cyclic dependency is technically correct.

Indeed -- the potential exists for __init__.py to affect _a.py, so the dependency is actually real. I did not think about it this way.

Phantom dependencies would then be ignored when determining if there are real cyclic imports.

Given the dependency is real, is this workable? If __init__.py is changed, _a.py needs to be rechecked. Can the "phantom" concept work around that?

Perhaps a flag can be added which will forbid "shadowing" modules in this way, and then it can be taken into consideration when building the dependency graph. But it probably is not worth it.

Conceptually, it seems to me like a failure in Python's relative import mechanism. I could imagine some syntax to alleviate it: from ._util import * as _util, or import ._util.

[bluetech]

BTW, my description missed a detail, which is that the order of the imports is important.

Consider:

[file package/_utils.py]
def some_helper(): pass

[file package/_a.py]
from . import _utils

class A: pass

print(_utils)
reveal_type(_utils)

[file package/__init__.py]
from ._a import A
_utils = 5

When imported, this prints

<module 'package._utils' from 'package/_utils.py'>

When type-checked, this gives

package/_a.py:6: note: Revealed type is '_importlib_modulespec.ModuleType'

However, when the order in __init__.py is switched:

[file package/__init__.py]
_utils = 5
from ._a import A

it now prints 5. Mypy still reveals ModuleType (this looks like a bug in mypy?).

[JukkaL]

Mypy still reveals ModuleType (this looks like a bug in mypy?).

It's a mypy design limitation. Mypy can't deal well when a thing changes type. In simple cases mypy uses heuristics that often work, but they can fail. At runtime, there may be no fixed type in a specific program location -- it may depend on runtime decisions. This is too dynamic for mypy to handle.

Can the "phantom" concept work around that?

Phantom dependencies would treated the same as normal dependencies when deciding which modules need to be processed. Some other analyses might ignore them. I'm not convinced that this concept is worth implementing, though.

Perhaps a flag can be added which will forbid "shadowing" modules in this way, and then it can be taken into consideration when building the dependency graph. But it probably is not worth it.

Agreed. This seems pretty marginal.

If you want to reject import cycles, a better approach for dealing with this might be to allow only whitelisted import cycles, assuming the tricky scenario doesn't happen too often.

[bluetech]

OK, this answers my question. Thanks!

It's a mypy design limitation. Mypy can't deal well when a thing changes type. [...]. This is too dynamic for mypy to handle.

I'd say that if mypy can't model the dynamic behavior, it should reject at the point the thing changes type, otherwise it is a soundness hole.

[JukkaL]

I'd say that if mypy can't model the dynamic behavior, it should reject at the point the thing changes type, otherwise it is a soundness hole.

The reason we don't do this right now is that mypy has accepted certain cases like this for a long time, and I believe that some libraries (SQLAlchemy?) depend on this feature. We need to compromise to avoid breaking common use cases. However, catching these would be a reasonable optional check. Would like to create an issue about the optional check?