Skip to content

Deal with CI environment changes #999

Open
@qinsoon

Description

@qinsoon

We consistently run and collect performance data for MMTk. It is inevitable that we will see performance change due to changes in the CI environment changes, including:

  • Testing hardware may change. We update our testing machines every few years.
  • Testing software may change. We update our testing machines regularly for new security patches, kernel changes, dependencies etc.
  • Benchmarks used may change. We will update our benchmarks when there is a new version of the benchmarks available.

We should be aware of the environment changes and know the performance impact from the changes. There are a few things we should do:

  • Add notes on the graph to note those changes.
  • Run a binary (such as an OpenJDK release binary) for a fixed workload to detect any performance change due to environment change.
  • Run the performance CI before or after we update the testing machines so we can see the diff between the environment with the same MMTk version.

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-opsArea: Operations (e.g. CI)P-highPriority: High. A high-priority issue should be fixed as soon as possible.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions