Add reporting of resource extraction to IAMC structure

[khaeru]

Per discussion 2025-11-03, this PR gives a proof-of-concept for implementation of one reporting 'table' (measure) using the Reporter class. See in particular the added module .report.extraction and the function .report.extraction.callback().

Some things to observe about the implementation:

• Reporting for this measure starts from specific quantities that are already present in the Reporter, thanks to the default behaviour of message_ix and other code.
  • If other reporting tasks also use the same inputs, they can be supplied from memory/cache and not recomputed.
• For aggregation, lists of codes (in this case, for the c/commodity dimension) are derived from the same files (in this case, commodity.yaml) that describe the model structure.
  • This ensures a single point of reference/truth for both model-building and -reporting.
  • The function itself should require minimal adjustment in the future; its behaviour will follow changes to commodity.yaml (or use of different commodity code lists for particular model variants).
  • The same data structures from the same keys can be used in any other module.
• The callback() function:
  • First adds some tasks specific to this measure, to get all the right values ready in genno.Quantity format and 'full' dimensionality.
  • Then uses a utility class (IAMCConversion) to set up more tasks that 'finalize' the IAMC-structured data, perform some partial sums, do renaming, assign units, etc.

Some things to observe about the PR:

• The new tasks are added in a 5-line function. The function is in a submodule of .report; a 'flat'/wide arrangement of similar reporting code will make for easier maintenance and readability than 1 or a few files with >1000 lines.
• The new function/module is added to .report.config._default_callbacks.
  This ensures it is called every time prepare_reporter() is called (unless user code deliberately removes it).
• The variables .report.util.NOT_IMPLEMENTED_{IAMC,MEASURE} and the expectation of .test_report.test_compare() are adjusted to reflect the additions.

Future PRs could make similar changes.

How to review

• Read the function linked above.
• Comment on whether documentation allows to (a) understand its behaviour and (b) follow it as a pattern for expanding reporting of other quantities.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update doc/whatsnew.

[codecov]

Codecov Report

❌ Patch coverage is 97.14286% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.0%. Comparing base (ccb9393) to head (dc81640).
⚠️ Report is 26 commits behind head on main.

Files with missing lines	Patch %	Lines
message_ix_models/tools/iamc.py	0.0%	2 Missing ⚠️
message_ix_models/report/util.py	97.1%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #451     +/-   ##
=======================================
- Coverage   76.2%   75.0%   -1.3%     
=======================================
  Files        278     278             
  Lines      23552   22751    -801     
=======================================
- Hits       17970   17065    -905     
- Misses      5582    5686    +104     

Files with missing lines	Coverage Δ
message_ix_models/model/bare.py	100.0% <100.0%> (ø)
message_ix_models/model/transport/structure.py	91.9% <100.0%> (-3.5%)	⬇️
message_ix_models/report/__init__.py	92.5% <100.0%> (-1.3%)	⬇️
message_ix_models/report/config.py	100.0% <100.0%> (ø)
message_ix_models/report/extraction.py	100.0% <100.0%> (ø)
message_ix_models/report/key.py	100.0% <100.0%> (ø)
message_ix_models/report/operator.py	85.2% <100.0%> (-2.5%)	⬇️
message_ix_models/report/sim.py	98.5% <100.0%> (ø)
message_ix_models/tests/model/test_bare.py	100.0% <ø> (ø)
message_ix_models/tests/test_report.py	100.0% <100.0%> (ø)
... and 3 more

... and 13 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[khaeru]

Hi @macflo8 @Wegatriespython @Tyler-lc @junukitashepard @yiyi1991 —another PR following on our discussion on Monday.

There, we discussed the idea of transferring some of the genno-based reporting of quantities like Final Energy or GDP that Florian had developed for the materials variant, into the general-purpose message_ix_models.report module. We talked about the strategy of tackling 'simpler' cases (such as GDP) first, to gain familiarity and identify what tools/utilities are still needed.

For the same reasons, I tried here to do one of the smaller/simpler functions:

message-ix-models/message_ix_models/report/legacy/default_tables.py

Lines 411 to 442 in ccb9393

	@_register
	def retr_extraction(units):
	"""Resources: Extraction (Fossil + Uranium).
	Annual extraction of fossil and uranium resources.
	Parameters
	----------
	units : str
	Units to which variables should be converted.
	"""
	vars = {}
	Coal = pp.extr("coal", units)
	Lignite = pp.extr("lignite", units)
	Conv_oil = pp.extr(["crude_1", "crude_2", "crude_3"], units)
	Unconv_oil = pp.extr(["crude_4", "crude_5", "crude_6", "crude_7", "crude_8"], units)
	Conv_gas = pp.extr(["gas_1", "gas_2", "gas_3", "gas_4"], units)
	Unconv_gas = pp.extr(["gas_5", "gas_6", "gas_7", "gas_8"], units)
	Uranium = pp.extr("uranium", units)
	vars["Coal"] = Coal + Lignite
	vars["Gas|Conventional"] = Conv_gas
	vars["Gas|Unconventional"] = Unconv_gas
	vars["Oil|Conventional"] = Conv_oil
	vars["Oil|Unconventional"] = Unconv_oil
	vars["Uranium"] = Uranium
	df = pp_utils.make_outputdf(vars, units)
	return df

Most of the changes on the branch are adjustments/housekeeping/improved utilities/improved docs, in order to make the one function linked in the PR branch as simple and clean as possible. I hope that this, in turn, will make it just as simple to report other quantities.

Please ask any questions by comments on the PR, on particular lines of the diff, or in Slack—I will try to add to the docs until the answers are all in there.

[yiyi1991]

Thank you so much Paul! I was able to run ctx.report.key = "all::iamc" and generated one extraction variable output file.

This is a good example to follow (e.g., for adding one variable category, and for adding building-related variables reporting). Please feel free to merge this PR.

This is not the best place but let me list two things I immediately sensed difficulties if we want to reproduce more complex power/material reporting under the same framework.

The first byproduct issue becomes ok after I took a scecond look.

message-ix-models/message_ix_models/model/material/report/reporting.py

Lines 1709 to 1712 in dc81640

	exclude = [
	"in|final|*|cokeoven_steel|*",
	"in|final|bf_gas|*",
	"in|final|co_gas|*",

When dealing with Final Energy variables, after retrieving all in flows by df = rep.get("in:nl-t-ya-m-c-l") with the explicit naming flow\|level\|commodity\|tech\|mode, we filtered them by level final (as only energy commodities use that level). Then the byproduct gases need to be excluded or even treated as negative energy input in specific scopes (e.g., bf_gas are generated by blast furnace but mostly immediately used by basic oxygen furnace, similarly those coke oven gas, but they should be counted under emission accounting.). But we can work with an, e.g., excluding_comm list to the key c right?

The second one is level-specific variable assignment.

message-ix-models/message_ix_models/data/material/reporting/fe.yaml

Lines 373 to 389 in dc81640

	Other Sector|Electricity|Thermal|Heat Pumps:
	filter:
	{ technology: hp_el_i, mode: M1, commodity: electr, level: final }
	short:
	fe_pe_other_el_th_hp
	Other Sector|Electricity|Specific:
	filter:
	{ technology: [sp_el_I, sp_el_I_RT], mode: M1, commodity: electr, level: [final, final_RT] }
	short:
	fe_pe_other_el_sp
	Other Sector|Electricity|Specific|Rooftop:
	filter:
	{ technology: sp_el_I_RT, mode: M1, commodity: electr, level: final_RT }
	short:
	fe_pe_other_el_sp_rt

To handle this the material reporting simply uses pandas groupby after genno. After 10 minutes digging, I think I need additional help to get genno handle that flexibly too.

In short, I think as a maintainer I get the idea of what to do with the commodity.yaml. My understanding is, the commodities with the same commodity names in gdx but different levels, they should be treated as different commodities here in this yaml? E.g., the commodity steel has 9 levels, including product and new_scrap, which go to different iamc variables. More level names in the upcoming bilateral trade...

message-ix-models/message_ix_models/data/commodity.yaml

Lines 61 to 67 in dc81640

	crude_8:
	description: Crude oil resource.
	Oil|Unconventional:
	description: Reporting aggregate for oil resources.
	report-only: true
	child: [crude_4, crude_5, crude_6, crude_7, crude_8]

[khaeru]

Thanks @yiyi1991 for the detailed comments! I will note these two points for specific response (maybe in tomorrow's weekly meeting), but in the meantime per:

Please feel free to merge this PR.

…could you please give a ✅ so I can do that?