Open
Description
We consistently run and collect performance data for MMTk. It is inevitable that we will see performance change due to changes in the CI environment changes, including:
- Testing hardware may change. We update our testing machines every few years.
- Testing software may change. We update our testing machines regularly for new security patches, kernel changes, dependencies etc.
- Benchmarks used may change. We will update our benchmarks when there is a new version of the benchmarks available.
We should be aware of the environment changes and know the performance impact from the changes. There are a few things we should do:
- Add notes on the graph to note those changes.
- Run a binary (such as an OpenJDK release binary) for a fixed workload to detect any performance change due to environment change.
- Run the performance CI before or after we update the testing machines so we can see the diff between the environment with the same MMTk version.