`xarray/datatree_` missing in 2024.2.0 sdist

[mgorny]

What happened?

Apparently xarray-2024.2.0 requires xarray.datatree_ module but this module isn't included in sdist tarball.

What did you expect to happen?

No response

Minimal Complete Verifiable Example

$ tar -tf /tmp/dist/xarray-2024.2.0.tar.gz  | grep datatree_
(empty)

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

n/a

[keewis]

can you explain why you think we require it? Is it that the tests fail for the released version (meaning we need a importorskip for the relevant tests)? Or does anything else fail?

For context, xarray/datatree_ is a temporary directory that contains the code of the datatree repository while we move forward with the migration. This means that open_datatree intentionally does not work for now (but we also don't expose it as public API / document it).

[mgorny]

Yes, tests failing to import was the issue I've encountered. I've grepped for it and noticed it being imported in installed modules, so I've presumed it was supposed to be included. I'm sorry for being presumptious.

[keewis]

no worries, we apparently forgot to skip the tests that need it, so this is a good hint.

[bnavigator]

Besides the supposedly ignorable open_datatree in

xarray/xarray/backends/common.py

Lines 134 to 141 in e47eb92

	def _open_datatree_netcdf(
	ncDataset: ncDataset | ncDatasetLegacyH5,
	filename_or_obj: str | os.PathLike[Any] | BufferedIOBase | AbstractDataStore,
	**kwargs,
	) -> DataTree:
	from xarray.backends.api import open_dataset
	from xarray.datatree_.datatree import DataTree
	from xarray.datatree_.datatree.treenode import NodePath

there is also still

xarray/xarray/backends/h5netcdf_.py

Lines 37 to 42 in e47eb92

	if TYPE_CHECKING:
	from io import BufferedIOBase
	from xarray.backends.common import AbstractDataStore
	from xarray.core.dataset import Dataset
	from xarray.datatree_.datatree import DataTree

,

xarray/xarray/backends/netCDF4_.py

Line 48 in e47eb92

from xarray.datatree_.datatree import DataTree

,

xarray/xarray/backends/api.py

Line 72 in e47eb92

from xarray.datatree_.datatree import DataTree

and so on in the sdist without shipping xarray/datatree_.

[keewis]

same as above: does this cause anything to fail? If it does, we can try to fix these.

(For reference, one of the examples above is a helper function for open_datatree which should never get called, while the remaining instances are within TYPE_CHECKING guards, so at most I'd expect typing to be affected)

[bnavigator]

Same as above: Not at regular production runtime, but when running (typechecking) tests.

[bnavigator]

Note that many developers like to have type checking enabled when they write code. If that code involves xarray, I supposed those developers might encounter failures.

[shadchin]

In xarray 24.3.0 too

>>> import xarray
>>> xarray.__version__
'2024.3.0'
>>> import xarray.core.datatree
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shadchin/arc/arcadia/contrib/python/xarray/xarray/core/datatree.py", line 34, in <module>
    from xarray.datatree_.datatree.common import TreeAttrAccessMixin
ModuleNotFoundError: No module named 'xarray.datatree_'

[shadchin]

$ tar -tf xarray-2024.3.0.tar.gz  | grep datatree_
(empty)
$ unzip -l xarray-2024.3.0-py3-none-any.whl | grep datatree_
(empty)

[max-sixty]

Is there a case where someone is writing a library that consumes xarray, type-checks their library, and their type check then fails?

If so, I think we should prioritize fixing this ASAP — we're somewhat violating the contract of having py.typed in our library.

Otherwise, it seems like a more theoretical problem (unless there are other cases which I'm missing), which we should consider solving but less of an immediate rush...