Repo layout proposal

[max-sixty]

What is your issue?

As part of the efforts described in #10039, I added #10088, and noticed the repo layout has arguably not kept up with the code growth over the past decade. This isn't the most pressing issue, but it does make the returns to refactors lower, since we're moving lines from 11K LOC files to 1K LOC files, rather than anything smaller.

(Even if you think LLMs aren't that useful / aren't going to get better / etc; these changes would still make the repo easier for people to navigate...)

In particular, 2/3 of our code is in xarray/core — 66873 LOC vs 97118 LOC in xarray

I can imagine splitting this up into a few categories:

• compat — dask_array_*, npcompat, pdcompat, array_api_compat
• compute / computation — computation, arithmetic, nanops, weighted, the curvefit that's currently in dataset, rolling, rolling_exp, maybe missing
• reshape / align / merge (need a better name) — merge, alignment, concat

I'd propose having each of those be paths within xarray/. Then there's more freedom to make new files within those paths relative to the current state, where a new file means adding onto a very long list of files in xarray/core.

I'm not confident on how much disruption that can cause to existing PRs. I think if we land them as commits which mostly just move the files, then git will mostly handle merges well. We can start slowly and see how it goes...

[dcherian]

I agree. One specific suggestion: breaking apply_ufunc.py out of computation.py is easy, and shouldn't be disruptive.

[TomNicholas]

I also support this.

reshape / align / merge (need a better name)

manipulation? restructuring?

There's another opportunity for refactoring that could split up large files in #9203.

Related is the general issue of scope creep within the main repository. I think at some point we should revisit the idea of splitting out as much non-core functionality as possible into a separate package (there was a very old issue about this that I'm struggling to find right now - proposing "xr-scipy"). Distinguishing between crucial things such as apply_ufunc and other things that currently live in computation.py such as curvefit would be important step towards that.

[benbovy]

(Even if you think LLMs aren't that useful / aren't going to get better / etc; these changes would still make the repo easier for people to navigate...)

This would also improve the performance of real-time developer tools like lsp servers, linters, auto-formatters, tree-sitter, etc. (EDIT: not reworking the repo layout but reducing LOC files).

One small suggestion: xarray/core/indexes.py may probably be moved within xarray/indexes which already exists.

[max-sixty]

To what extent do we need to maintain backward-compat? (I would obv prefer to just move files and not build stub layers, but if there are other libs relying on these location of files, rather than just importing from the top level, then we can add them...)

[TomNicholas]

Any library not importing from the top level is using private API, which has no backwards compatibility guarantees. That's what our docs say.

We have actually made significant changes to the locations of code multiple times in the past couple of years (especially as part of the NamedArray refactor, DataTree integration, and adding the ChunkManager entrypoint), with no real complaints.

[max-sixty]

So far, I've:

• Moved a bunch of files to xarray/computation
• Moved a bunch of files to xarray/structure
• Moved a bunch of files to xarray/compat
• Moved apply_ufunc to its own file in xarray/computation

I'll probably let that sit for a while and see how it goes.

Some things we could do next:

• Take some computation code out of dataset.py; likely make a computation/curvefit.py
• Refine some of the groupings — for example, there's a unify_chunks function in computation/computation.py, but also a core/chunks.py. Maybe we want those in the same place... More generally I'm sure there are other things, either caused by me or previously existing, which could be improved.
• Start breaking up the test files — this is fairly easy, just need to make an xarray/tests/{dataset,dataarray} path and make some groupings within files there

---

The more difficult thing is going to be breaking up dataset.py; currently 10K lines long, and is basically just Dataset now. We can probably thin 2K lines by extracting some of the code in methods. But to make real progress, given classes can't split over multiple files in python, we would need to do something like DatasetSel / DatasetReduce, and then have Dataset inherit from them. (Unless there's a cleverer approach?)

The alternative is to do nothing, wait for LLMs to get better with big files (Claude Code can already grab segments of files, for example). Many of the annoyances of us working with huge files, like black taking a few seconds on every save, have gone away with tools like ruff, so maybe waiting is OK.

[TomNicholas]

That all sounds great @max-sixty . I think you can be relatively aggressive about this, and I'm happy to review things.

The more difficult thing is going to be breaking up dataset.py; currently 10K lines long, and is basically just Dataset now.
...
(Unless there's a cleverer approach?)

I asked Claude for ideas but several of them require doing things dynamically upon import.

[benbovy]

Results from a quick search of from xarray.core through all github, just in case.