Speed up "from X import A; from X import B; from X import C; ..."

[gvanrossum]

Over in Pyston-land someone remarked that this:

from X import A, B, C, ..., Z

is much faster than

from X import A
from X import B
from X import C
.
.
.
from X import Z

We can't easily rewrite the latter as the former, because in edge cases the semantics are different (though maybe Black could take the liberty to do it anyway...?). But we could try to make the latter execute faster -- in theory everything needed is just in sys.modules['X'], though in practice there appear to be some caches that are reset between from X import statements but that are kept around between A, B, ...., Z. Maybe there's some wiggle room or a quicker way to check for cache invalidation (which is probably rare).

Note that X could well be a package path like foo.bar.baz. Also, if X is a package, then A, B, ..., Z could be submodules; but if X is a module, A...Z must be variables (or functions, classes, etc.) defined in X.

[stuaxo]

Forgetting edge cases for a moment (I assume there are many).

My initial thought was a cache that holds only the previously loaded module, this is the dumbest implementation (avoiding the lookup in sys.modules even).

[gvanrossum]

That would seem like a good idea. I haven't actually looked at the code to try to understand what's slowing this down though, there may be a reason there explaining why this is difficult. (Another reason may not be obvious from the code: it's possible that there's other (non-import) code between the first and the second import that might invalidate the cache.)

[stuaxo]

I'm just going to note a 3rd style I've seen in some larger projects (it may well a variant on from X import A, B, C, ..., Z but just in case:

from X import (
  A,
  B,
  C,
  ..., 
   Z
)

[sthagen]

@stuaxo this variant is in my experience often used to make the linters happy ... and black called on a file with a from module import comma, separated, list line exceeding the set line length automatically rewrites that way (using the parentheses).

[gvanrossum]

There is no difference in execution speed between

from X import A, B, ..., Z

and

from X import (
    A,
    B,
    ...,
    Z
)

(at least assuming we're executing unmarshalled bytecode -- it is the same, as you can tell by experimenting with dis.py.)

[markshannon]

"someone remarked that X is much faster than Y" is a bit vague. Any actual numbers?

[davfsa]

Looks like this was kinda lost as other things were prioritized. I took the liberty to test this out and found very interesting results:

Script

import timeit
import platform


def dummy_work():
    # Some dummy work to churn up the CPU
    for j in range(1_000_000):
        if sum(j for j in range(20)) < 0:
            raise RuntimeError


def run_test(fn):
    dummy_work()
    result = timeit.timeit(fn, number=100_000)
    print(fn.__name__, result)


def import_single_line():
    from abc import abstractmethod, abstractclassmethod, abstractstaticmethod, abstractproperty, ABCMeta, ABC, get_cache_token


def import_multi_line():
    from abc import (
        abstractmethod,
        abstractclassmethod,
        abstractstaticmethod,
        abstractproperty,
        ABCMeta,
        ABC,
        get_cache_token,
    )


def import_multi_line_split():
    from abc import abstractmethod
    from abc import abstractclassmethod
    from abc import abstractstaticmethod
    from abc import abstractproperty
    from abc import ABCMeta
    from abc import ABC
    from abc import get_cache_token


if __name__ == "__main__":
    print("Using", platform.python_implementation(), platform.python_version())

    print()
    print("Results:")
    run_test(import_single_line)
    run_test(import_multi_line)
    run_test(import_multi_line_split)

Using CPython 3.10.4

Results:
import_single_line 0.06816064300073776
import_multi_line 0.06977313799961848
import_multi_line_split 0.3498124390007433

It is (on average, the above run is the best out of 10 consecutive runs) 20% slower to use split multi line imports than single line ones