[aladinor]

Closes #481. Follow-up to #335.

#335 made the deprecated open_*_datatree functions and xd.open_datatree call each backend's open_datatree directly. That skipped xarray's dataset wrapping (chunks, caching, default indexes) and dropped most backend_kwargs. Both now go through xr.open_datatree, with one shared helper (_open_legacy_datatree in common.py) for the 12 deprecated wrappers.

Fixed (each checked against 77534e4, main before #335)

	Before #335	#335	This PR
open_odim_datatree(f, chunks={}) (also gamic, iris, rainbow, datamet, cfradial1, cfradial2), xd.open_datatree(..., chunks={})	dask	TypeError	dask
backend_kwargs=dict(first_dim="time") (8 engines)	honoured	silently ignored	honoured
legacy optional=False (iris, hpl, metek)	works	TypeError	works
range index on iris, rainbow, furuno, datamet sweeps	yes	no	yes
optional_groups=True: station vars in /radar_parameters	no	yes	no

Also:

• The reindex_angle FutureWarning is emitted once and points at the caller. Before, it fired once per sweep (14 times for one ODIM volume) and pointed inside xradar. The NEXRAD "Dropped incomplete sweeps" warnings and the CfRadial2 warnings also point at the caller.
• open_uf_datatree has its signature and docstring back.
• Docstrings no longer list parameters an engine rejects (fix_second_angle on rainbow and datamet, optional on imd), and IRIS drops an unused group parameter.
• backend_kwargs given to the NEXRAD and UF wrappers may contain decoders such as decode_times again.

Behaviour changes worth knowing

• CfRadial1 and CfRadial2: the backend option for the netCDF reader is now netcdf_engine. engine collided with xarray's own engine argument, so xr.open_datatree(f, engine="cfradial1", backend_kwargs={"engine": ...}) could not work. The legacy open_cfradial1_datatree(engine=...) and open_cfradial2_datatree(engine=...) still accept engine. ENH: add xarray-native open_datatree with engine= parameter #335 is not released yet, so no released API changes.
• As in xr.open_datatree, a keyword given both directly and in backend_kwargs takes the backend_kwargs value. Passing both site_coords and site_as_coords raises TypeError, as IMD already did.
• With chunks={}, root variables are chunked too, because xarray chunks the whole tree.
• NEXRAD sweeps get a range index like the other engines.
• CfRadial2 now reads the file in one pass and closes it before returning, so the data are in memory rather than lazy. Going through xr.open_datatree made xarray reopen the closed file for lazy reads, and with several handles on the same file netCDF4/HDF5 segfaulted intermittently, locally and once in CI. A CfRadial2 tree now holds no open file handle.

Tests

New tests in tests/io/test_backend_datatree.py compare legacy output with xd.open_datatree for every deprecated function. They also cover:

• chunks;
• backend_kwargs, including decoders and Optional;
• optional=False, optional_groups and site_as_coords;
• the range index and xd vs xr equality;
• which sweep is returned;
• the reindex_angle warning;
• the UF signature, docstrings, netcdf_engine and CfRadial2 close().

Mutation checks (reverting each hunk on its own) confirmed a test fails for each fix.

• pytest -n 4 --dist loadfile tests: 1085 passed, 18 skipped
• black --check ., ruff check .
• pytest docs/notebooks: 25 passed. Two failed only because ffmpeg (multiple-sweeps-into-volume-scan) and pyart (Gridding) are not installed locally.
• Comparison harness: 13 engines × 25 keyword cases against 77534e4. The only differences are the intended ones above, or cases that raised before and now work.

Not in this PR (listed in #481)

• DataTree.close() does not close files for the other engines (same before ENH: add xarray-native open_datatree with engine= parameter #335).
• Loose sweep-string parsing ("sweep_1_0").
• site_coords has no effect on most engines.
• _ENGINE_REGISTRY duplicates the entry points.

#472 touches common.py too; whichever merges second resolves the conflict.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.69%. Comparing base (6c07893) to head (147ba3b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #484      +/-   ##
==========================================
+ Coverage   95.54%   95.69%   +0.14%     
==========================================
  Files          29       29              
  Lines        6783     6759      -24     
==========================================
- Hits         6481     6468      -13     
+ Misses        302      291      -11     

Flag	Coverage Δ
notebooktests	0.00% <0.00%> (ø)
unittests	95.69% <100.00%> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kmuehlbauer]

• CfRadial2 now reads the file in one pass and closes it before returning, so the data are in memory rather than lazy. Going through xr.open_datatree made xarray reopen the closed file for lazy reads, and with several handles on the same file netCDF4/HDF5 segfaulted intermittently, locally and once in CI. A CfRadial2 tree now holds no open file handle.

@aladinor Just to get this right, when reading CfRadial2 we now materialize everything in memory? Is there no way to have this data lazy? Can this be eventually fixed on xarray side?

There where some changes in xarray recently (pydata/xarray#11692 and pydata/xarray#11629) which might resolve this (not checked).

Another question wrt CfRadial2, what's the difference in using xr.open_datatree(..., engine="netcdf4") compared to our engine="cfradial2"?

[syedhamidali]

Thanks @aladinor, I tested this independently. The full suite passes locally (1084 passed). Compared with the commit before #335, I ran 10 engines × 10 legacy keyword cases: all four regressions from #481 are fixed, and reindex_angle now warns once at the caller (14 → 1).

Findings:

1. IMD regression: open_imd_datatree(f, backend_kwargs=dict(Optional=False)) worked before #335 and on main (IMD has no optional variables and ignored it), but raises TypeError here, because the helper maps it to optional= and the engine rejects it. The helper could drop optional for engines that don't take it; a test for it would be good too.
2. Not in the description: NEXRAD and UF now honour first_dim in backend_kwargs (dims time instead of azimuth), and UF also gets a range index. Both are consistent with the other engines, just worth listing.

On @kmuehlbauer's CfRadial2 question: with a copy of a CfRadial2 file that I delete before reading, engine="cfradial2" on main is lazy but fails with FileNotFoundError (it reopens the closed file), while here the read works and no handle stays open. engine="netcdf4" stays lazy with one open handle. With chunks={} you get dask arrays, but over already-loaded data. Same values as engine="netcdf4", plus platform_is_mobile and wmo__cf_profile in the root and a length-1 frequency variable in each sweep.

[kmuehlbauer]

@aladinor CI is finally green. Please feel free to use any of that or remove if you are not satisfied or have another solution in mind. I also did not want to fiddle around with the history.md entry. It's already quite long. But here is what I did in the last commits:

• Test Suite Refactoring: Wrapped xr.open_datatree invocations inside test_xr_open_datatree_cfradial2 using a explicit context manager (with statement) to enforce deterministic resource cleanup per test execution loop.
• Backend Refactoring: Removed eager memory loading (.load()) in xradar/io/backends/cfradial2.py. Preserved target file handle reference at tree root via dtree["/"].set_close(tree._close) and routed through datatree_from_dict_with_io_cleanup to enable lazy loading and eliminate repetitive backend disk reopening requests.
• Parallel Execution Alignment: Synchronized shared session fixture setup across pytest-xdist runners by wrapping file generation hooks with a FileLock mechanism (only cfradial1_file and cfradial2_file).
• Environment Maintenance: Enforced Python >= 3.12 constraints and dropped legacy pip dependency definitions for upstream xarray builds in favor of conda-forge packages.

[aladinor]

Thanks @kmuehlbauer for making CfRadial2 lazy and CI green, and @syedhamidali for the thorough testing! CI changes stay here. Next: single cache layer + close on errors, check the IMD Optional case vs main, update changelog/description, then review.

[aladinor]

@kmuehlbauer I tried to move forward with the suggested additions, but CI kept failing on a flaky segfault. It comes from pydata/xarray#11692 (unreleased): on d23bab1, test_backend_datatree.py run serially crashed 8/10 times with xarray 2026.9.0 and 0/10 with xarray main. We strongly suggest waiting for the next xarray release or testing against main until then.

[kmuehlbauer]

@aladinor Just for testing, can you run this for xarray main? I'll catch up tomorrow.

[aladinor]

@kmuehlbauer done: with xarray main, CI is green on 3.12–3.14, no segfault (6d05c94).

[aladinor]

@syedhamidali IMD Optional raised the same TypeError before #335, so it is not a #484 regression. @kmuehlbauer cfradial2 gives the same values as netcdf4, plus sweep and FM301 normalization. Follow-ups: #495, #496, #497.