[SR-1609] Support recording baseline for performance regression tests

[briancroom]

Previous ID	SR-1609
Radar	None
Original Reporter	@briancroom
Type	Bug

Additional Detail from JIRA

Votes	0
Component/s	XCTest
Labels	Bug
Assignee	None
Priority	Medium

md5: 89bb688c4da749aeb43653263685237d

Issue Description:

We added basic support for XCTest's performance testing APIs in https://bugs.swift.org/browse/SR-1355, however one crucial component of that is still missing, namely recording baseline results of performance tests and triggering test failures when significant regressions are detected relative to those baselines.

I see a few pieces to this:

• Persisting baseline data to disk. Xcode uses a Plist format for this. It would probably make sense to use a compatible format here.

• Comparing test results to a previously recorded baseline and triggering a failure if the performance has regressed. Xcode's supposedly triggers a failure if Average - BaselineAverage is greater than 0.1 seconds and is more than 10% higher than BaselineAverage itself.

• Providing a mechanism (probably command-line flags?) instructing the runner where to find and create baseline data, and for causing the baseline data to be updated.

[doozMen]

How can this be so long open?