ADD: Support reading NEXRAD Level 2 chunks from real-time S3 bucket#332
Open
aladinor wants to merge 11 commits intoopenradar:mainfrom
Open
ADD: Support reading NEXRAD Level 2 chunks from real-time S3 bucket#332aladinor wants to merge 11 commits intoopenradar:mainfrom
aladinor wants to merge 11 commits intoopenradar:mainfrom
Conversation
Demonstrates listing/downloading chunk files from the unidata-nexrad-level2-chunks bucket and using the list input with incomplete_sweep drop/pad modes.
…weep 21 new tests across 5 classes covering: - _concatenate_chunks edge cases (empty list, single element, short chunks, nonexistent paths, file-like at nonzero position) - Tuple input to open_nexradlevel2_datatree - incomplete_sweeps property with multiple/all incomplete sweeps - Drop mode warning with specific sweep indices - Pad mode data flow (incomplete set passed to open_sweeps_as_dict) - Pad mode late-detection for user-specified sweep lists - Pad-mode reindex pipeline using real NEXRAD data with removed rays - List/tuple input combined with drop and pad modes
- Add Development section to history.md with list-input, incomplete_sweep, notebook, and test suite entries - Document chunk file / list input and incomplete_sweep parameter in importers.md with usage examples - Add nexrad_read_chunks notebook to usage.md toctree
…weep handling Add _concatenate_chunks() for assembling chunk byte objects, volume header detection, incomplete_sweeps property on NEXRADLevel2File, and the incomplete_sweep parameter (drop/pad) to open_nexradlevel2_datatree. Fix read-only array issue in util.py with .values.copy().
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Fix E712 (equality comparisons to True/False), I001 (unsorted imports), F401 (unused import), F841 (unused variables). Run nbstripout on notebook.
is_compressed returns np.True_/np.False_, not Python bool, so `is True`/`is False` identity checks fail.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #332 +/- ##
==========================================
+ Coverage 93.59% 93.69% +0.10%
==========================================
Files 27 27
Lines 5654 5730 +76
==========================================
+ Hits 5292 5369 +77
+ Misses 362 361 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
kmuehlbauer
reviewed
Feb 20, 2026
Collaborator
kmuehlbauer
left a comment
There was a problem hiding this comment.
Two quick comments after skimming through this neat PR. I'll have a closer look over the weekend. Thanks @aladinor!
- Add tests covering 12 previously uncovered lines from PR openradar#332: is_compressed with tiny chunks, force-close sweep on premature end, open_sweeps_as_dict default incomplete_sweeps, and pad-mode reindex - Validate that chunk list input includes the S file (volume header); raise ValueError with clear message when missing - Patch coverage should now be ~100% Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Chunks must be passed in chronological order (S file first, then I/E chunks in sequence). Out-of-order chunks silently corrupt uncompressed data or raise BZ2 errors for compressed files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Address review comments from kmuehlbauer:
- Use {doc} cross-reference for nexrad_read_chunks notebook
- Add ({pull}`332`) links to all changelog entries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Verify that incomplete_sweep='pad' on a complete VCP file produces
the same sweeps as the default ('drop'), confirming no interference.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
open_nexradlevel2_datatreefor streaming NEXRAD Level 2 data from theunidata-nexrad-level2-chunksS3 bucketincomplete_sweepparameter ("drop"/"pad") to handle partial volumes where not all chunks have arrived yetCloses #330
Changes
Core implementation (
xradar/io/backends/nexrad_level2.py)_concatenate_chunks(): Assembles a list of chunk byte objects, file-like objects, or file paths into a single byte stream, with automatic volume header detection and skipping for non-start chunksincomplete_sweepsproperty onNEXRADLevel2File: Tracks which sweeps were force-closed (incomplete) during parsingincomplete_sweepparameter inopen_nexradlevel2_datatree:"drop"(default): Excludes incomplete sweeps and emits a warning"pad": Keeps incomplete sweeps and reindexes to a full azimuth grid (360 or 720 azimuths), filling missing rays with NaNFix (
xradar/util.py).values.copy()instead of.valuesto prevent modifying read-only arrays during angle reindexingNotebook (
examples/notebooks/nexrad_read_chunks.ipynb)Documentation (
docs/importers.md,docs/usage.md,docs/history.md)Tests (
tests/io/test_nexrad_level2.py)_concatenate_chunksedge cases (empty list, single element, volume header, short chunks, file-like objects, nonexistent paths, tuple input)incomplete_sweepparameter behavior (drop warns, pad passes incomplete set, late-detection path)Test plan
pytest tests/io/test_nexrad_level2.py)cd docs && make html)