8390550: C2 hit MemLimit when run with -Xcomp -XX:VerifyIterativeGVN=1110

[vnkozlov]

The cause of the issue is that during SubTypeCheck node expansion control and memory are affected and a lot of temporary MergeMem nodes created as result. Even if temporary nodes are removed from graph after processing they still hold space in nodes arena and other arenas. The more IGVN optimizations are called the more such temporary nodes are created.

In the failed compilation case C2 created about 1M temporary MergeMem nodes (note the method $Proxy132::<clinit> is big: 5967 byte codes). Which alone requires about 200Mb of memory in arena.

This is ironically consequences of JDK-8239072: "subtype check macro node causes node budget to be exhausted" changes which added whole graph IGVN optimization after EACH macro node expansion.

Calling _igvn.optimize() after elimination of Opaque and LoopLimit nodes and before expanding nodes solved the JDK-8239072 issue alone. So calling it after each macro expansion is excessive.

The simplest solution is to adjust JDK-8239072 changes to call _igvn.optimize() inside iteration only when we reach some node count limit or reasonable number of macro nodes expansion happened.

Tested: tier1,tier2,tier3,hs-aot-stress

---

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed (2 reviews required, with at least 1 Reviewer, 1 Author)

Issue

• JDK-8390550: C2 hit MemLimit when run with -Xcomp -XX:VerifyIterativeGVN=1110 (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/32421/head:pull/32421
$ git checkout pull/32421

Update a local copy of the PR:
$ git checkout pull/32421
$ git pull https://git.openjdk.org/jdk.git pull/32421/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 32421

View PR using the GUI difftool:
$ git pr show -t 32421

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/32421.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back kvn! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@vnkozlov The following label will be automatically applied to this pull request:

• hotspot-compiler

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[openjdk]

The total number of required reviews for this PR has been set to 2 based on the presence of this label: hotspot-compiler. This can be overridden with the /reviewers command.

[mlbridge]

Webrevs

• 00: Full (59556e56)