Add memory usage diff to SPMI reports by EgorBo · Pull Request #124943 · dotnet/runtime

EgorBo · 2026-02-27T05:34:51Z

Example:

Add a new 'memorydiff' command to the SuperPMI tooling alongside 'asmdiffs' and 'tpdiff'. This command measures JIT arena allocation differences (BytesAllocated metric) between base and diff JITs for all contexts and reports aggregate statistics including P90 per-context allocation ratios. Key changes: - superpmi.py: Add SuperPMIReplayMemoryDiff class, aggregate_memory_diff_metrics(), write_memorydiff_markdown_summary(), subparser, setup_args, main dispatch, and summarize support for 'memorydiff' - superpmi_diffs.py: Add do_memorydiff() method to Diff class, wire into dispatch - superpmi_diffs_setup.py: Add 'memorydiff' to valid types, use checked JITs - superpmi_diffs_summarize.py: Add 'memorydiff' consolidation support The command leverages the existing BytesAllocated JITMETADATAMETRIC that is already automatically written to the SPMI details CSV. No C++ changes needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

dotnet-policy-service · 2026-02-27T05:35:46Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Copilot

Pull request overview

Adds a new “memorydiff” mode to the CoreCLR SuperPMI diff infrastructure, enabling CI to report JIT memory allocation deltas alongside existing asm diffs and throughput diffs.

Changes:

Extend SuperPMI scripts (superpmi.py, superpmi_diffs*.py) to run and summarize a new memorydiff diff type.
Update JIT metrics reporting so BytesAllocated is available (and reported) for memory-diffing scenarios.
Wire memorydiff through Helix project + Azure Pipelines templates and produce consolidated markdown reports.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/coreclr/scripts/superpmi_diffs_summarize.py	Recognize and consolidate `memorydiff` summaries into the overall markdown report.
src/coreclr/scripts/superpmi_diffs_setup.py	Allow `memorydiff` as a diff type and ensure release assets are required for it.
src/coreclr/scripts/superpmi_diffs.py	Add `memorydiff` execution path and capture its JSON summary output.
src/coreclr/scripts/superpmi.py	Add `memorydiff` subcommand, implement replay+aggregation logic, and add JSON summarization support.
src/coreclr/scripts/superpmi-diffs.proj	Download Helix result artifacts for `memorydiff` runs.
src/coreclr/jit/jitmetadata.cpp	Make `JitMetrics::report()` available outside `DEBUG` to support metric reporting in release builds.
src/coreclr/jit/fginline.cpp	Always merge inlinee metrics into root metrics (not `DEBUG`-only).
src/coreclr/jit/compiler.cpp	Ensure `BytesAllocated` is set and `Metrics.report()` is called (outside `DEBUG`) so the metric is surfaced.
eng/pipelines/coreclr/templates/superpmi-diffs-job.yml	Add `memorydiff` handling to job dependencies/artifact usage.
eng/pipelines/coreclr/templates/run-superpmi-diffs-job.yml	Pass correct setup args for `memorydiff` jobs.
eng/pipelines/coreclr/superpmi-diffs.yml	Add CI matrix entries to run `memorydiff` alongside existing diff types.

Copilot · 2026-02-27T05:43:11Z

src/coreclr/scripts/superpmi_diffs.py

+        # This is the summary file name and location written by superpmi.py. If the file exists, remove it to ensure superpmi.py doesn't created a numbered version.
+        overall_json_memorydiff_summary_file = os.path.join(self.spmi_location, "memorydiff_summary.json")
+        if os.path.isfile(overall_json_memorydiff_summary_file):
+            os.remove(overall_json_memorydiff_summary_file)
+
+        overall_json_memorydiff_summary_file_target = os.path.join(self.log_directory, "superpmi_memorydiff_summary_{}.json".format(self.target))
+        self.summary_json_files.append((overall_json_memorydiff_summary_file, overall_json_memorydiff_summary_file_target))


Diff.summarize() writes a JSON placeholder string when the expected summary JSON doesn’t exist. For the new memorydiff flow, superpmi_diffs_summarize.py will pass these files to superpmi.py summarize, which expects a specific JSON tuple shape for memorydiff and will throw if it receives a placeholder string. Consider updating the memorydiff path to ensure a valid (possibly empty) memorydiff_summary.json is always produced, or adjust the placeholder generation to emit the correct JSON structure for memorydiff.

Copilot · 2026-02-27T05:43:12Z

src/coreclr/scripts/superpmi.py

+
+                print_superpmi_error_result(return_code, self.coreclr_args)
+
+                (base_metrics, diff_metrics) = aggregate_memory_diff_metrics(details_info_file)


aggregate_memory_diff_metrics(details_info_file) will throw if the SuperPMI run fails to produce the -details CSV (e.g., early failure/abort), which can mask the real SuperPMI error and prevent summary generation. Consider checking os.path.isfile(details_info_file) (or catching OSError) and treating missing/invalid CSV as (None, None) so the run can continue and emit a usable summary/logs.

Suggested change

(base_metrics, diff_metrics) = aggregate_memory_diff_metrics(details_info_file)

base_metrics = None

diff_metrics = None

try:

if os.path.isfile(details_info_file):

(base_metrics, diff_metrics) = aggregate_memory_diff_metrics(details_info_file)

else:

logging.warning("Details info file '%s' not found; skipping memory diff aggregation", details_info_file)

except Exception as ex:

logging.warning("Failed to aggregate memory diff metrics from '%s': %s", details_info_file, ex)

Copilot · 2026-02-27T05:43:12Z

src/coreclr/scripts/superpmi.py

+        # Construct an overall Markdown summary file.
+
+        if len(memory_diffs) > 0:
+            if not os.path.isdir(self.coreclr_args.spmi_location):
+                os.makedirs(self.coreclr_args.spmi_location)
+
+            (base_jit_options, diff_jit_options) = get_base_diff_jit_options(self.coreclr_args)
+
+            if self.coreclr_args.summary_as_json:
+                overall_json_summary_file = create_unique_file_name(self.coreclr_args.spmi_location, "memorydiff_summary", "json")
+                if os.path.isfile(overall_json_summary_file):
+                    os.remove(overall_json_summary_file)
+
+                with open(overall_json_summary_file, "w") as write_fh:
+                    # Strip Per-context ratios lists to avoid bloating the JSON file
+                    json_memory_diffs = []
+                    for (mch_file, base_m, diff_m) in memory_diffs:
+                        json_base = {k: {mk: mv for mk, mv in v.items() if mk != "Per-context ratios"} for k, v in base_m.items()}
+                        json_diff = {k: {mk: mv for mk, mv in v.items() if mk != "Per-context ratios"} for k, v in diff_m.items()}
+                        json_memory_diffs.append((mch_file, json_base, json_diff))
+                    json.dump((base_jit_options, diff_jit_options, json_memory_diffs), write_fh)
+                    logging.info("  Summary JSON file: %s", overall_json_summary_file)
+            else:
+                overall_md_summary_file = create_unique_file_name(self.coreclr_args.spmi_location, "memorydiff_summary", "md")
+
+                if os.path.isfile(overall_md_summary_file):
+                    os.remove(overall_md_summary_file)
+
+                with open(overall_md_summary_file, "w") as write_fh:
+                    write_memorydiff_markdown_summary(write_fh, base_jit_options, diff_jit_options, memory_diffs, True)
+                    logging.info("  Summary Markdown file: %s", overall_md_summary_file)
+
+                short_md_summary_file = create_unique_file_name(self.coreclr_args.spmi_location, "memorydiff_short_summary", "md")
+
+                if os.path.isfile(short_md_summary_file):
+                    os.remove(short_md_summary_file)
+
+                with open(short_md_summary_file, "w") as write_fh:
+                    write_memorydiff_markdown_summary(write_fh, base_jit_options, diff_jit_options, memory_diffs, False)
+                    logging.info("  Short Summary Markdown file: %s", short_md_summary_file)
+


This code only writes memorydiff_summary.json when len(memory_diffs) > 0. If the run yields no diffed contexts (or fails before producing metrics), superpmi_diffs.py will end up uploading a placeholder JSON, but superpmi.py summarize expects a structured tuple for memorydiff and will fail to parse a non-conforming placeholder. To keep the pipeline summary robust, consider always emitting a valid JSON summary file (even with an empty memory_diffs list) whenever --summary_as_json is requested.

Suggested change

# Construct an overall Markdown summary file.

if len(memory_diffs) > 0:

if not os.path.isdir(self.coreclr_args.spmi_location):

os.makedirs(self.coreclr_args.spmi_location)

(base_jit_options, diff_jit_options) = get_base_diff_jit_options(self.coreclr_args)

if self.coreclr_args.summary_as_json:

overall_json_summary_file = create_unique_file_name(self.coreclr_args.spmi_location, "memorydiff_summary", "json")

if os.path.isfile(overall_json_summary_file):

os.remove(overall_json_summary_file)

with open(overall_json_summary_file, "w") as write_fh:

# Strip Per-context ratios lists to avoid bloating the JSON file

json_memory_diffs = []

for (mch_file, base_m, diff_m) in memory_diffs:

json_base = {k: {mk: mv for mk, mv in v.items() if mk != "Per-context ratios"} for k, v in base_m.items()}

json_diff = {k: {mk: mv for mk, mv in v.items() if mk != "Per-context ratios"} for k, v in diff_m.items()}

json_memory_diffs.append((mch_file, json_base, json_diff))

json.dump((base_jit_options, diff_jit_options, json_memory_diffs), write_fh)

logging.info(" Summary JSON file: %s", overall_json_summary_file)

else:

overall_md_summary_file = create_unique_file_name(self.coreclr_args.spmi_location, "memorydiff_summary", "md")

if os.path.isfile(overall_md_summary_file):

os.remove(overall_md_summary_file)

with open(overall_md_summary_file, "w") as write_fh:

write_memorydiff_markdown_summary(write_fh, base_jit_options, diff_jit_options, memory_diffs, True)

logging.info(" Summary Markdown file: %s", overall_md_summary_file)

short_md_summary_file = create_unique_file_name(self.coreclr_args.spmi_location, "memorydiff_short_summary", "md")

if os.path.isfile(short_md_summary_file):

os.remove(short_md_summary_file)

with open(short_md_summary_file, "w") as write_fh:

write_memorydiff_markdown_summary(write_fh, base_jit_options, diff_jit_options, memory_diffs, False)

logging.info(" Short Summary Markdown file: %s", short_md_summary_file)

# Construct overall summary files.

if self.coreclr_args.summary_as_json:

if not os.path.isdir(self.coreclr_args.spmi_location):

os.makedirs(self.coreclr_args.spmi_location)

(base_jit_options, diff_jit_options) = get_base_diff_jit_options(self.coreclr_args)

overall_json_summary_file = create_unique_file_name(self.coreclr_args.spmi_location, "memorydiff_summary", "json")

if os.path.isfile(overall_json_summary_file):

os.remove(overall_json_summary_file)

with open(overall_json_summary_file, "w") as write_fh:

# Strip Per-context ratios lists to avoid bloating the JSON file

json_memory_diffs = []

for (mch_file, base_m, diff_m) in memory_diffs:

json_base = {k: {mk: mv for mk, mv in v.items() if mk != "Per-context ratios"} for k, v in base_m.items()}

json_diff = {k: {mk: mv for mk, mv in v.items() if mk != "Per-context ratios"} for k, v in diff_m.items()}

json_memory_diffs.append((mch_file, json_base, json_diff))

json.dump((base_jit_options, diff_jit_options, json_memory_diffs), write_fh)

logging.info(" Summary JSON file: %s", overall_json_summary_file)

elif len(memory_diffs) > 0:

if not os.path.isdir(self.coreclr_args.spmi_location):

os.makedirs(self.coreclr_args.spmi_location)

(base_jit_options, diff_jit_options) = get_base_diff_jit_options(self.coreclr_args)

overall_md_summary_file = create_unique_file_name(self.coreclr_args.spmi_location, "memorydiff_summary", "md")

if os.path.isfile(overall_md_summary_file):

os.remove(overall_md_summary_file)

with open(overall_md_summary_file, "w") as write_fh:

write_memorydiff_markdown_summary(write_fh, base_jit_options, diff_jit_options, memory_diffs, True)

logging.info(" Summary Markdown file: %s", overall_md_summary_file)

short_md_summary_file = create_unique_file_name(self.coreclr_args.spmi_location, "memorydiff_short_summary", "md")

if os.path.isfile(short_md_summary_file):

os.remove(short_md_summary_file)

with open(short_md_summary_file, "w") as write_fh:

write_memorydiff_markdown_summary(write_fh, base_jit_options, diff_jit_options, memory_diffs, False)

logging.info(" Short Summary Markdown file: %s", short_md_summary_file)

jakobbotsch · 2026-02-27T09:10:14Z

src/coreclr/jit/compiler.cpp

    }
 #endif // DEBUG
+
+    Metrics.report(this);


This will make a ton of JIT-EE calls. I would suggest adding DOTNET_JitReportMetrics and setting it unconditionally from SuperPMI so that we only report in cases where the metrics will actually be used.

Good point, I'll move the JIT changes to a separate PR so this PR can work with already built jitrollingbuilds once that one is merged

egorbot and others added 2 commits February 27, 2026 03:59

add memorydiff

797d16b

Copilot AI review requested due to automatic review settings February 27, 2026 05:34

github-actions bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Feb 27, 2026

dotnet-policy-service bot assigned EgorBo Feb 27, 2026

Copilot started reviewing on behalf of EgorBo February 27, 2026 05:35 View session

Copilot AI reviewed Feb 27, 2026

View reviewed changes

This was referenced Feb 27, 2026

[FAIL] System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandlerTest.LargeUriAndHeaders_Works #124487

Open

[android][clr] No peer certificates when executing System.Net.Http.Functional.Tests on Android emulator #124526

Open

jakobbotsch reviewed Feb 27, 2026

View reviewed changes

Copilot AI mentioned this pull request Feb 27, 2026

Introduce DOTNET_JitReportMetrics #124975

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add memory usage diff to SPMI reports #124943

Add memory usage diff to SPMI reports #124943
EgorBo wants to merge 2 commits intodotnet:mainfrom
EgorBo:memorydiff-spmi

EgorBo commented Feb 27, 2026

Uh oh!

dotnet-policy-service bot commented Feb 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

jakobbotsch Feb 27, 2026

Uh oh!

EgorBo Feb 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		print_superpmi_error_result(return_code, self.coreclr_args)

		(base_metrics, diff_metrics) = aggregate_memory_diff_metrics(details_info_file)

-                (base_metrics, diff_metrics) = aggregate_memory_diff_metrics(details_info_file)
+                base_metrics = None
+                diff_metrics = None
+                try:
+                    if os.path.isfile(details_info_file):
+                        (base_metrics, diff_metrics) = aggregate_memory_diff_metrics(details_info_file)
+                    else:
+                        logging.warning("Details info file '%s' not found; skipping memory diff aggregation", details_info_file)
+                except Exception as ex:
+                    logging.warning("Failed to aggregate memory diff metrics from '%s': %s", details_info_file, ex)

Conversation

EgorBo commented Feb 27, 2026

Uh oh!

dotnet-policy-service bot commented Feb 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

jakobbotsch Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

EgorBo Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants