Lazy Imports

[headtr1ck]

• Hopefully Closes Long import time #6726
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[max-sixty]

Very nice @headtr1ck !

[headtr1ck]

Not as much of an improvement as I thought, but shaving off another 20ms.

I can see if I can bring it down even further...

[max-sixty]

Do you think there's a risk something gets added back by mistake?

A galaxy brain idea is to have a lint in pre-commit which looks for ^import (foo|bar) for the libraries we should be lazy loading. Not fool proof. Or something to measure import time so we can see if it rises again.

[headtr1ck]

The simplest solution would be to add a simple unit test that checks that blacklisted packets are not in sys.modules.

For import timings we have asv, but that is not run so often.

[headtr1ck]

Now I have made every module lazy that is not too much work, this includes:
matplotlib, flox and all backends.

The only remaining large import is dask.array, here I have to check if that is possible to make lazy?
This alone is 500ms import time, since it also imports numba, scipy and sparse... Maybe we could even open a ticket at dask about this topic as well?

[headtr1ck]

Seems like it is but a good idea to mess with sys.modules in pytest. Anyone got an idea how to solve that?

Edit: solved :)

[headtr1ck]

I managed to get dask.array lazy as well, but dask.utils is still imported by backends.locks.

Someone else has to tackle this, I am not too familiar with this...

This got the import time quite low. I am happy with this now :)

[headtr1ck]

It seems that cfgrib has its own xarray_plugin now.
Should the internal version be deprecated?

[Illviljan]

Very nice! I've felt the long import times recently and this will help greatly!

Maybe next step in a future PR is to try making pandas lazy as well? I don't think it's used as much as the top import in every file makes it out to be.

[dcherian]

Thanks @headtr1ck great PR!

[hmaarrfk]

Exciting improvements on usability for the next version!

[dcherian]

FYI I'm getting this error when trying to update the version in pyodide: https://app.circleci.com/pipelines/github/pyodide/pyodide/5016/workflows/e2cc0865-a91d-4cdb-8dbc-dfaeeb522c5f/jobs/60731

JavascriptException message:  Traceback (most recent call last):
  File "/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/lib/python3.10/_pyodide/_base.py", line 531, in eval_code_async
    await CodeRunner(
  File "/lib/python3.10/_pyodide/_base.py", line 357, in run_async
    coroutine = eval(self.code, globals, locals)
  File "<exec>", line 1, in <module>
  File "/lib/python3.10/site-packages/xarray/__init__.py", line 1, in <module>
    from . import testing, tutorial
  File "/lib/python3.10/site-packages/xarray/tutorial.py", line 17, in <module>
    from .backends.api import open_dataset as _open_dataset
  File "/lib/python3.10/site-packages/xarray/backends/__init__.py", line 16, in <module>
    from .scipy_ import ScipyBackendEntrypoint, ScipyDataStore
  File "/lib/python3.10/site-packages/xarray/backends/scipy_.py", line 239, in <module>
    class ScipyBackendEntrypoint(BackendEntrypoint):
  File "/lib/python3.10/site-packages/xarray/backends/scipy_.py", line 260, in ScipyBackendEntrypoint
    available = module_available("scipy")
  File "/lib/python3.10/site-packages/xarray/core/utils.py", line 1009, in module_available
    return importlib.util.find_spec(module) is not None
  File "/lib/python3.10/importlib/util.py", line 103, in find_spec
    return _find_spec(fullname, parent_path)
  File "/lib/python3.10/_pyodide/_importhook.py", line 226, in find_spec
    raise ModuleNotFoundError(
ModuleNotFoundError: The module 'scipy' is included in the Pyodide distribution, but it is not installed. You can install it by calling: await micropip.install("scipy") in Python or await pyodide.loadPackage("scipy") in JavaScript. See https://pyodide.org/en/stable/usage/loading-packages.html for more details.

this seems like a bug in importlib

[headtr1ck]

I don't think this is a bug in importlib but rather pyodide. They seem to add a hook to importlib (which is allowed, but probably causing this issue?)

Edit: The error message from pyodide seems more like this is intentional? Is scipy really installed correctly?

[dcherian]

Turns out it is a pyodide bug with how they hook in to importlib