Define baseline target environment for AOT compilations

[dsouzai]

Currently, the target environment of AOT code is essentially the host environment. However, the need for defining some baseline target environment has risen with the arrival of the multi-layer SCC feature. The point of having multi-layer SCCs is to facilitate having SCCs in different layers in Docker images; images that are, by design, portable. While the AOT Header validation [1] does validate that the processors and other features are compatible, there are some problems - problems which have always been there; it's just that generally in a single SCC world, the SCC does not move from the machine on which it was generated.

The problems generally stem from the processor validations. Specifically:

1. If the SCC is generated on machine A which is newer than machine B, it is possible that a JVM on machine B will not be able to use the SCC. While this certainly the right thing to do, it does mean that whoever wishes to build these SCC layers needs to find some old machine to populate the SCCs to ensure the widest coverage.
2. If the SCC is generated on a machine B which is older than machine A, then while the AOT code will likely be loaded on machine A, it is also possible that the JVM will add new AOT code to the SCC using features that were available on machine A but not on machine B. Now, if we were to use the updated SCC on machine B, we would just load the code because when we validated the SCC on machine A, we only check whether the features of machine A are a superset of machine B and we never update the AOT Header with this super set. As such we might run code that isn't compatible on machine B.

The way to deal with both problems is to do what all static compilers do. Define some baseline target environment with the reasonably widest possible coverage. This depends on eclipse-omr/omr#4518 (and associated OpenJ9 changes that will have to follow).

Obviously there are some subtleties that have to be considered. For example, because the generated AOT code is also run on the machine on which it is generated in a normal cold run, the features of host have to be a superset of a baseline target environment; otherwise the host becomes the baseline target environment. Another example is, in the second run, if the code in the SCC was generated on a machine who's features are a subset of the baseline target environment, then the AOT code should again only be targeted to use only the features specified in the SCC.

This is also relevant for JITServer, as the target of the compilation could be a machine that is different from where the compilation occurs.

---

[1] https://github.com/eclipse/openj9/blob/dc7b78cbd099a1188d018d9b43c2d4e51933d8a3/runtime/compiler/runtime/RelocationRuntime.cpp#L1022

---

1 - By "optional dynamic relocatable compilation", I mean the decision to compile some method is made at runtime (dynamic) and the decision to generate position independent code for a particular compilation (relocatable compilation) is also made at runtime (optional).

[dsouzai]

fyi @mstoodle @ymanton @vijaysun-omr @mpirvu

[vijaysun-omr]

Thanks, is the next step to discuss the OMR issue at an OMR architecture meeting ?

[dsouzai]

Yeah I asked that eclipse-omr/omr#4518 be added to the architecture meeting.

That said, after an offline discussion with @mpirvu, maybe going down that road isn't absolutely necessary. Since it isn't particularly likely that a portable SCC is not going to be in a container of some sort, Marius suggested that using an option to inform the JVM that the current run is the "populate SCC" run should be sufficient.

Such an option would cause the global target environment to be set to some base level, which would result in even the JIT'd code targeting that arch level; however, because this run would be part of the container build step whose main purpose is to populate the SCC, it doesn't matter too much if the JIT'd code's quality isn't the best.

I think the subtleties still remain in the multi-SCC world.

1. If an TR_AOTHeader already exists, then we need to make the base level whatever the level of the TR_AOTHeader is in the first run.
2. We would need to prevent new AOT code from being added to the SCC in the second run unless we were certain it didn't use features that were not available when the previous SCCs were generated.

What do you think @mstoodle @ymanton ?

[dsouzai]

I thought about this some more and talked to @mpirvu offline more about his idea about the option, which I misunderstood; he actually suggested specifying the target environment. After that discussion, I'm in favour of going down that road as well. We would need something like:

1. A JIT/AOT option along the lines of lastPopulateRun
2. A JIT/AOT option to specify the target processor as well as the vendor
3. A bit in the TR_AOTHeader that's stored in the SCC

The general semantics would be:

1. When lastPopulateRun is specified, if the bit isn't already set in the TR_AOTHeader, it gets set.
2. If lastPopulateRun is set, new AOT code can be added to the SCC.
3. If lastPopulateRun is not set, then if the bit in the TR_AOTHeader is set, no new AOT code can be added to the SCC.
4. If the target processor is not specified, then the host processor is targeted; however, if an SCC already exists, then the processor specified in the SCC is targeted. Obviously there has to be sanity checks regarding the specified target, the target in the SCC, and the host.

---

The process would be as follows:

Scenario 1: Traditional: User runs with -Xshareclasses:..
In the cold run SCC is populated with AOT. In the warm run, AOT code is loaded from SCC, but other AOT code can still be added to the SCC. In other words, no change to current behaviour. This is because the bit in TR_AOTHeader is never set.

Scenario 2: Last Populate Run: User runs with -Xshareclasses:.. -Xaot:lastPopulateRun,processor=...
In the code run the SCC is populated with AOT. Additionally, a bit is set in the TR_AOTHeader. In the warm run, so long as the user has the lastPopulateRun option, new AOT code will be added to the SCC. Once the user removes the lastPopulateRun option, we prevent new AOT code from being added to the SCC.

The processor=... option would be used to tell the compiler what processor to target for the generated code. Because this is global, when used in conjunction with lastPopulateRun, this would effectively result in sub-par JIT'd code but that wouldn't matter because populating the SCC is the main concern.

---

Because in #7630 we learned that there is only one TR_AOTHeader for a given layer chain, this means that with with the lastPopulateRun proposal, anyone who takes an existing stack of multiSCCs will need to specify lastPopulateRun in order to populate the newest layer with AOT code. However I'm fine with this requirement because it makes explicit that the populate run is generating code for some targeted architecture and so should be thought of as exactly that: the "populate run".

[dsouzai]

Closing this in favour of the discussion in #7966