Skip to content

perf: parallelise cmip7repack calls across split output files#549

Merged
rbeucher merged 2 commits into
mainfrom
perf/parallel-cmip7-repack
Jul 23, 2026
Merged

perf: parallelise cmip7repack calls across split output files#549
rbeucher merged 2 commits into
mainfrom
perf/parallel-cmip7-repack

Conversation

@rbeucher

Copy link
Copy Markdown
Member

Summary

When split_years produces N output chunks, cmip7repack was previously called sequentially (blocking subprocess) after each chunk was written — one at a time.

This PR dispatches all repack calls concurrently using ThreadPoolExecutor once all chunk files have been written.

Changes

  • _write_single(): removed the inline _repack_cmip7_output + validate_cmip7_output calls.
  • write()multi-chunk path: writes all chunks sequentially, then repacks in parallel via ThreadPoolExecutor.map(), then validates sequentially (validation must follow repack).
  • write()single-file path (split off / fx variables): repack + validate called synchronously as before — no behaviour change.

Why threads are safe here

  • Repack threads are only launched after all dask computations have been flushed and the netCDF4.Dataset file handles are closed, so there is no scheduler contention with dask.
  • Each thread runs an independent subprocess.run(["cmip7repack", ...]) on a separate file — no shared state.

Impact

For a 165-year monthly run (17 × 10-year chunks), all 17 cmip7repack subprocesses now run concurrently instead of serially.

@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.8%. Comparing base (d0815eb) to head (74d2cc1).

Files with missing lines Patch % Lines
src/access_moppy/base.py 70.0% 1 Missing and 2 partials ⚠️

❌ Your patch check has failed because the patch coverage (70.0%) is below the target coverage (90.0%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@           Coverage Diff           @@
##            main    #549     +/-   ##
=======================================
- Coverage   74.9%   74.8%   -0.0%     
=======================================
  Files         36      36             
  Lines       7156    7163      +7     
  Branches    1333    1336      +3     
=======================================
+ Hits        5357    5361      +4     
- Misses      1519    1520      +1     
- Partials     280     282      +2     
Flag Coverage Δ
unit 74.8% <70.0%> (-<0.1%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rbeucher
rbeucher force-pushed the perf/parallel-cmip7-repack branch from 02c10d1 to 0a0295d Compare July 23, 2026 07:19
rbeucher added 2 commits July 23, 2026 17:19
When split_years produces N output chunks, cmip7repack was previously
called sequentially after each _write_single(), one blocking subprocess
at a time.

Now all chunk files are written first, then their repacks are dispatched
concurrently via ThreadPoolExecutor. CMIP7 validation still runs
sequentially after all repacks finish (it must follow repack).

The single-file path (split off or fx variables) is unchanged: repack
and validate are called synchronously as before.

No Dask interference: repack threads start only after all dask
computations are flushed and the netCDF file handles are closed.
@rbeucher
rbeucher merged commit 69d481f into main Jul 23, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant