CLI Only select specific variables

[rbeucher]

This pull request adds support for running a targeted subset of variables in batch processing using a new --variable command-line option. This allows users to limit a run to specific variables from the config, which is useful for debugging or partial reruns. The implementation ensures that only the specified variables are added to the tracking database and handled by the monitor and job submission scripts. Comprehensive tests are included to verify correct behavior and error handling.

The most important changes are:

Feature: Subset Variable Selection

• Added a --variable command-line argument to moppy-cmorise that allows users to specify a subset of variables to process, instead of always running all variables in the config. Only these variables are inserted into the tracking database and processed for the current run (src/access_moppy/batch_cmoriser.py, docs/source/batch_processing.rst, [1] [2].
• The monitor script and job submission logic now respect the filtered variable list, using an environment variable (MOPPY_VARIABLE_FILTER) to ensure only the selected variables are monitored and submitted (src/access_moppy/batch_cmoriser.py, src/access_moppy/templates/cmor_monitor_script.j2, [1] [2].

Validation and Error Handling

• The code validates that all variables specified with --variable exist in the config file, and exits with a clear error message if any are missing, before making any changes to the database (src/access_moppy/batch_cmoriser.py, src/access_moppy/batch_cmoriser.pyL999-R1030).

Integration with Existing Options

• The --variable option integrates cleanly with existing options like --force and --rerun-variable, so that resets and re-runs only affect the filtered subset, not all variables in the config (src/access_moppy/batch_cmoriser.py, [1] [2].

Testing

• Added thorough unit tests to verify that the subset selection works as intended, including that only the specified variables are inserted into the database, the monitor script/environment is correctly configured, and error handling is robust (tests/unit/test_batch_cmoriser.py, [1] [2].

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.9%. Comparing base (d0815eb) to head (f16f8ba).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #552   +/-   ##
=====================================
  Coverage   74.9%   74.9%           
=====================================
  Files         36      36           
  Lines       7156    7204   +48     
  Branches    1333    1344   +11     
=====================================
+ Hits        5357    5394   +37     
- Misses      1519    1525    +6     
- Partials     280     285    +5     

Flag	Coverage Δ
unit	74.9% <100.0%> (+<0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.