Skip to content

Comments

ADD: Support reading NEXRAD Level 2 chunks from real-time S3 bucket#332

Open
aladinor wants to merge 11 commits intoopenradar:mainfrom
aladinor:add-nexrad-chunks-notebook
Open

ADD: Support reading NEXRAD Level 2 chunks from real-time S3 bucket#332
aladinor wants to merge 11 commits intoopenradar:mainfrom
aladinor:add-nexrad-chunks-notebook

Conversation

@aladinor
Copy link
Member

Summary

  • Add support for list/tuple of chunk sources (bytes, file-like, or paths) as input to open_nexradlevel2_datatree for streaming NEXRAD Level 2 data from the unidata-nexrad-level2-chunks S3 bucket
  • Add incomplete_sweep parameter ("drop"/"pad") to handle partial volumes where not all chunks have arrived yet
  • Add notebook example demonstrating real-time chunk streaming from S3
  • Add comprehensive offline test suite for chunk list-input and incomplete sweep handling

Closes #330

Changes

Core implementation (xradar/io/backends/nexrad_level2.py)

  • _concatenate_chunks(): Assembles a list of chunk byte objects, file-like objects, or file paths into a single byte stream, with automatic volume header detection and skipping for non-start chunks
  • incomplete_sweeps property on NEXRADLevel2File: Tracks which sweeps were force-closed (incomplete) during parsing
  • incomplete_sweep parameter in open_nexradlevel2_datatree:
    • "drop" (default): Excludes incomplete sweeps and emits a warning
    • "pad": Keeps incomplete sweeps and reindexes to a full azimuth grid (360 or 720 azimuths), filling missing rays with NaN

Fix (xradar/util.py)

  • Use .values.copy() instead of .values to prevent modifying read-only arrays during angle reindexing

Notebook (examples/notebooks/nexrad_read_chunks.ipynb)

  • Full walkthrough: connecting to S3, downloading chunks, opening full/partial volumes, drop vs pad mode, early streaming with few chunks

Documentation (docs/importers.md, docs/usage.md, docs/history.md)

  • User-facing docs for chunk input and incomplete_sweep parameter with code examples
  • Notebook registered in docs toctree
  • Changelog entries

Tests (tests/io/test_nexrad_level2.py)

  • 21 new tests across 5 test classes covering:
    • _concatenate_chunks edge cases (empty list, single element, volume header, short chunks, file-like objects, nonexistent paths, tuple input)
    • Sweep completeness detection (multiple incomplete, all incomplete, missing key defaults)
    • incomplete_sweep parameter behavior (drop warns, pad passes incomplete set, late-detection path)
    • Pad-mode reindexing with real data (full azimuth grid, data preservation, NaN percentage)
    • List/tuple input integration with drop/pad modes

Test plan

  • All 75 tests pass (pytest tests/io/test_nexrad_level2.py)
  • Notebook executes successfully against live S3 data
  • Docs build successfully (cd docs && make html)

Demonstrates listing/downloading chunk files from the
unidata-nexrad-level2-chunks bucket and using the list input
with incomplete_sweep drop/pad modes.
…weep

21 new tests across 5 classes covering:
- _concatenate_chunks edge cases (empty list, single element, short
  chunks, nonexistent paths, file-like at nonzero position)
- Tuple input to open_nexradlevel2_datatree
- incomplete_sweeps property with multiple/all incomplete sweeps
- Drop mode warning with specific sweep indices
- Pad mode data flow (incomplete set passed to open_sweeps_as_dict)
- Pad mode late-detection for user-specified sweep lists
- Pad-mode reindex pipeline using real NEXRAD data with removed rays
- List/tuple input combined with drop and pad modes
- Add Development section to history.md with list-input, incomplete_sweep,
  notebook, and test suite entries
- Document chunk file / list input and incomplete_sweep parameter in
  importers.md with usage examples
- Add nexrad_read_chunks notebook to usage.md toctree
…weep handling

Add _concatenate_chunks() for assembling chunk byte objects, volume header
detection, incomplete_sweeps property on NEXRADLevel2File, and the
incomplete_sweep parameter (drop/pad) to open_nexradlevel2_datatree.
Fix read-only array issue in util.py with .values.copy().
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@aladinor aladinor requested a review from kmuehlbauer February 20, 2026 15:46
Fix E712 (equality comparisons to True/False), I001 (unsorted imports),
F401 (unused import), F841 (unused variables). Run nbstripout on notebook.
is_compressed returns np.True_/np.False_, not Python bool, so
`is True`/`is False` identity checks fail.
@codecov
Copy link

codecov bot commented Feb 20, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.69%. Comparing base (b00e01f) to head (a2dc470).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #332      +/-   ##
==========================================
+ Coverage   93.59%   93.69%   +0.10%     
==========================================
  Files          27       27              
  Lines        5654     5730      +76     
==========================================
+ Hits         5292     5369      +77     
+ Misses        362      361       -1     
Flag Coverage Δ
notebooktests 0.00% <0.00%> (ø)
unittests 93.69% <100.00%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Collaborator

@kmuehlbauer kmuehlbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two quick comments after skimming through this neat PR. I'll have a closer look over the weekend. Thanks @aladinor!

aladinor and others added 3 commits February 20, 2026 12:00
- Add tests covering 12 previously uncovered lines from PR openradar#332:
  is_compressed with tiny chunks, force-close sweep on premature end,
  open_sweeps_as_dict default incomplete_sweeps, and pad-mode reindex
- Validate that chunk list input includes the S file (volume header);
  raise ValueError with clear message when missing
- Patch coverage should now be ~100%

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Chunks must be passed in chronological order (S file first, then
I/E chunks in sequence).  Out-of-order chunks silently corrupt
uncompressed data or raise BZ2 errors for compressed files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Address review comments from kmuehlbauer:
- Use {doc} cross-reference for nexrad_read_chunks notebook
- Add ({pull}`332`) links to all changelog entries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@aladinor aladinor requested a review from kmuehlbauer February 20, 2026 18:15
Verify that incomplete_sweep='pad' on a complete VCP file produces
the same sweeps as the default ('drop'), confirming no interference.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Support reading NEXRAD Level 2 chunks from real-time S3 bucket

2 participants