Transport improvements for 2025-W44 #447

khaeru · 2025-10-30T09:32:15Z

#430 ← previous PR | following PR → (none)

iiasa/message_data#626
Implement runs using ExogenousEmissionPrice.
Add a SLURM batch script for running mix-models transport on UniCC.
- Small adjustments and clean-ups to other code to work in this environment.
New utility class .sdmx.StructureFactory.

How to review

TBA

PR checklist

Continuous integration checks all ✅
Add or expand tests; coverage checks both ✅
Add, expand, or update documentation.
Update doc/whatsnew.

codecov · 2025-10-30T10:23:57Z

Codecov Report

❌ Patch coverage is 78.21522% with 83 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.9%. Comparing base (cb47229) to head (eaa3bc2).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
message_ix_models/report/operator.py	44.3%	44 Missing ⚠️
message_ix_models/model/transport/report.py	25.0%	21 Missing ⚠️
message_ix_models/model/transport/plot.py	81.8%	6 Missing ⚠️
message_ix_models/model/transport/build.py	78.5%	3 Missing ⚠️
message_ix_models/cli.py	66.6%	2 Missing ⚠️
message_ix_models/model/transport/operator.py	33.3%	2 Missing ⚠️
message_ix_models/model/workflow.py	60.0%	2 Missing ⚠️
message_ix_models/util/sdmx.py	88.8%	2 Missing ⚠️
message_ix_models/model/buildings/rc_afofi.py	0.0%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #447     +/-   ##
=======================================
- Coverage   75.6%   74.9%   -0.8%     
=======================================
  Files        278     279      +1     
  Lines      22740   22892    +152     
=======================================
- Hits       17201   17150     -51     
- Misses      5539    5742    +203

Files with missing lines	Coverage Δ
message_ix_models/__init__.py	`100.0% <100.0%> (ø)`
message_ix_models/model/emissions.py	`98.7% <100.0%> (ø)`
message_ix_models/model/transport/__init__.py	`100.0% <100.0%> (ø)`
message_ix_models/model/transport/config.py	`99.4% <100.0%> (+0.5%)`	⬆️
message_ix_models/model/transport/data.py	`98.9% <ø> (ø)`
message_ix_models/model/transport/demand.py	`100.0% <ø> (ø)`
message_ix_models/model/transport/key.py	`100.0% <100.0%> (ø)`
message_ix_models/model/transport/passenger.py	`96.6% <100.0%> (ø)`
message_ix_models/model/transport/policy.py	`100.0% <100.0%> (ø)`
message_ix_models/model/transport/slurm.py	`100.0% <100.0%> (ø)`
... and 29 more

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Add --remote option for using SSH via fabric. - New Template class for templating Slurm `sbatch` scripts. - Add --module/-m option for locating templates in submodules of message-ix-models. - Add fabric to packages without type hints. - Update tests.

- Use a better name for 'needs_log_file' local in .cli.main().

- add_tasks() methods to set up the build Computer to prepare data to represent these policies. - prepare_computer() to invoke these methods according to the Policy instances present in .transport.Config.policy. - Add .transport.policy to default .transport.Config.modules.

- Adjust test.

- Expand CHECKS for data created by .transport.policy.

- Rely instead on .transport.policy.prepare_computer() to chain to distinct Policy subclasses. - Call .model.workflow.step_0() directly; don't remove data for tax_emission added in the transport build step.

- Move from .transport.report and generalize.

- Rely on .report.operator.latest_reporting(). - Use .iamc.to_quantity(). - Structure operations internally using genno.Computer.

- Add .plot.prepare_computer(…, kind=…, target=…) keyword arguments. - Filter plots to add using `kind`.

- Drop similar/parallel code for adding plots.

Use features of Key directly.

- Incorporate features from .transport.plot.Plot. - New attributes strage, single. - Remove url attribute. - Add PlotFromIAMC, PlotTimeSeries subclasses; adjust existing plots. - New COMMON global collection of plotnine elements. - Move items from .transport.plot: LabelFirst, change level of matplotlib logger. - New functions collect() and prepare_computer(). - Use prepare_computer() in callback(); remove PLOTS global.

- Use in .transport.build instead of hard-coded string.

khaeru added this to the 2025-12 milestone Oct 30, 2025

khaeru self-assigned this Oct 30, 2025

khaeru added enh New features or functionality transport MESSAGEix-Transport variant or transport in the base model labels Oct 30, 2025

khaeru force-pushed the transport/2025-w44 branch from 3c6c62d to bd49756 Compare October 30, 2025 09:33

khaeru force-pushed the transport/2025-w44 branch from bd49756 to 1be671a Compare October 31, 2025 10:29

khaeru mentioned this pull request Nov 6, 2025

Transport improvements for 2025-W39 #430

Merged

7 tasks

khaeru force-pushed the transport/2025-w44 branch 3 times, most recently from 7a6303d to bc4ca5e Compare November 12, 2025 14:21

khaeru added a commit that referenced this pull request Nov 18, 2025

TEMPORARY Invoke message_data workflow using #447

7c58887

khaeru force-pushed the transport/2025-w44 branch 2 times, most recently from b4bd5ae to 2e4fe45 Compare November 19, 2025 15:52

khaeru added a commit that referenced this pull request Nov 21, 2025

TEMPORARY Invoke message_data workflow using #447

8a5ba72

khaeru force-pushed the transport/2025-w44 branch 7 times, most recently from b133d02 to ecde703 Compare November 25, 2025 14:29

khaeru mentioned this pull request Nov 25, 2025

CircEUlar runs for 2025-W48 #462

Draft

khaeru added 7 commits November 25, 2025 17:03

Update "transport" CI workflow labels

bab825a

Handle policy codes in LoadFactorLDV

c4fc44f

Drop "sha" from message_data workflow dispatch

f2fb751

Add "slurm_remote_args" config setting

6bb47d0

Improve .util.slurm and CLI

e9e75aa

- Add --remote option for using SSH via fabric. - New Template class for templating Slurm `sbatch` scripts. - Add --module/-m option for locating templates in submodules of message-ix-models. - Add fabric to packages without type hints. - Update tests.

Skip logging for mix-models sbatch command

20c4c39

- Use a better name for 'needs_log_file' local in .cli.main().

Remove plotnine from packages without type hints

fb97962

khaeru added 16 commits November 25, 2025 17:03

Add IIASA_ECE:CL_TRANSPORT_SCENARIO(1.2.1)

786a342

Add USD / t to transport units

36f9a0b

Assign units in PRICE_EMISSION.add_tasks

a413a6d

- Adjust test.

Add policy cases to transport…test_debug

c3a348a

- Expand CHECKS for data created by .transport.policy.

Add remove_emission_bounds(…, parameters=…) arg

fb739b8

Add step_0(…, remove_emission_parameters=…) arg

53d6f65

Remove .transport.workflow.tax_emission()

6a5fa91

- Rely instead on .transport.policy.prepare_computer() to chain to distinct Policy subclasses. - Call .model.workflow.step_0() directly; don't remove data for tax_emission added in the transport build step.

Ignore warnings due to iiasa/ixmp4#209

0dbedbe

Add .operator.latest_reporting_from_{file,platform}

df7a0c2

- Move from .transport.report and generalize.

Add .report.operator.latest_reporting()

408dac7

Add .transport.plot.MultiStock

91e0835

Allow .iamc.to_quantity(…, query=None)

0a3e448

Refactor .transport.report.multi()

6ea5102

- Rely on .report.operator.latest_reporting(). - Use .iamc.to_quantity(). - Structure operations internally using genno.Computer.

Add .util.genno.append()

fbde5be

Add .transport.plot.Kind enumeration

4128f96

- Add .plot.prepare_computer(…, kind=…, target=…) keyword arguments. - Filter plots to add using `kind`.

khaeru force-pushed the transport/2025-w44 branch from ecde703 to eaa3bc2 Compare November 25, 2025 16:03

khaeru added 6 commits November 26, 2025 15:44

Use .plot.prepare_computer() in {build, report}

b90bd5a

- Drop similar/parallel code for adding plots.

Update usage of genno.KeySeq

0ac08a0

Use features of Key directly.

Add .model.workflow.STAGE enumeration

26d96c5

Add .types.PlotAddable

55df6ff

Add make_output_path(…, config_key=…) argument

9c72c42

Handle 0 args to .operator.summarize()

2869d4d

khaeru force-pushed the transport/2025-w44 branch 2 times, most recently from 10f2fc3 to dcab36b Compare November 26, 2025 16:34

khaeru added 4 commits November 26, 2025 21:45

Use improved .report.plot features in .transport

ef76cdb

Add .transport.plot.MultiFE

89ec434

Add .transport.key.debug

81fee17

- Use in .transport.build instead of hard-coded string.

khaeru force-pushed the transport/2025-w44 branch from dcab36b to 81fee17 Compare November 26, 2025 21:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Transport improvements for 2025-W44 #447

Transport improvements for 2025-W44 #447

Uh oh!

khaeru commented Oct 30, 2025 •

edited

Loading

Uh oh!

codecov bot commented Oct 30, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Transport improvements for 2025-W44 #447

Are you sure you want to change the base?

Transport improvements for 2025-W44 #447

Uh oh!

Conversation

khaeru commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to review

PR checklist

Uh oh!

codecov bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

khaeru commented Oct 30, 2025 •

edited

Loading

codecov bot commented Oct 30, 2025 •

edited

Loading