Description
Description
We've recently ported one of our packages to the new testing infrastructure, and it has been overall smooth - we can see one significant performance regression on the actual runtime of the test and have narrowed it down to the use of #expect in two places - I will provide the test code and timing results and sample of the runtime here, but we can work on providing a reduced case if this does not give enough hints on what the underlying problem can be.
The timing measurements we see are:
Release mode (swift test -c release):
XCTest:
Test Case '-[PricingTests.ArbitrageTests testBlackScholesForArbitrage]' passed (3.371 seconds).
New Testing:
Test blackScholesForArbitrage() passed after 37.032 seconds.
Debug (swift test):
XCTest:
Test Case '-[PricingTests.ArbitrageTests testBlackScholesForArbitrage]' passed (30.888 seconds).
New Testing:
Test blackScholesForArbitrage() passed after 164.550 seconds.
New Testing (with commenting out two #expect in the loops in the sample code provided below):
Test blackScholesForArbitrage() passed after 98.767 seconds.
So basically we see a 10x runtime difference in release mode with XCtest winning over the new test framework - and a factor 6 in debug mode. Commenting out the two #expects in the sample below (best guess based on the sample) gave a significant improvement from 164 ---> 98 seconds runtime.
Attaching sample:
Here is the test code in question for the new testing framework (no changes in the test itself, just using the new framework):
@Test func blackScholesForArbitrage() throws {
let allowedDeviation = 5.0e-11
let sampleSize = 1_000_000
for _ in 0...sampleSize {
let S = Double.random(in: 0...1_000)
let X = Double.random(in: 0...1_000)
let σ = Double.random(in: 0...2.0)
let r = Double.random(in: 0...2.0)
let q = Double.random(in: 0...2.0)
let t = Double.random(in: 0...10.0)
let putCallParity = S * .exp(-q * t) - .exp(-r * t) * X
let sampleDividends: [Pricing.Dividend] = []
var bsValues: [Pricing.Results] = []
let commonParams = Pricing.Parameters.Common(
exerciseType: .european,
spotPrice: Pricing.SideValues(bid: S, ask: S),
financingRate: r,
yield: q,
timeToMaturity: t,
dividends: sampleDividends
)
let specifics = [
Pricing.Parameters.Specific(
marketPrice: Pricing.SideValues(bid: 5.0, ask: 21.0), volatility: σ
),
Pricing.Parameters.Specific(
marketPrice: Pricing.SideValues(bid: 5.0, ask: 21.0), volatility: σ
),
]
let referenceData = [
Pricing.Parameters.InstrumentReferenceData(instrumentType: .call, strikePrice: X),
Pricing.Parameters.InstrumentReferenceData(instrumentType: .put, strikePrice: X),
]
let pricingContext = Pricing.Context(
commonParameters: commonParams,
reference: referenceData,
parameters: specifics,
theoreticalValues: [.fair, .delta, .vega, .rho, .vanna, .gamma]
)
BlackScholes.calculate(context: pricingContext, results: &bsValues)
let call = bsValues[0].fair ?? 0.0
let put = bsValues[1].fair ?? 0.0
for (index, result) in bsValues.enumerated() {
let boundaries = ArbitrageBoundaries.validFairValue(
commonParameters: commonParams,
reference: referenceData[index]
)
if let fairResult = result.fair {
// #expect(
// (boundaries?.contains(fairResult)) != false,
// "fair value \(fairResult) is not within \(String(describing: boundaries))"
// )
} else {
#expect(result.fair != nil, "No fair result returned")
}
}
#expect(abs(call - put - putCallParity) < allowedDeviation, "\(call) \(put)")
}
for _ in 0...sampleSize {
let S = Double.random(in: 0...1_000)
let X = Double.random(in: 0...1_000)
let σ = Double.random(in: 0...2.0)
let r = Double.random(in: 0...2.0)
let q = Double.random(in: 0...2.0)
let t = Double.random(in: 0...10.0)
let timeToDiv = Double.random(in: 0...t)
let dividendSize = Double.random(in: 0...S)
let pvOfDiv = dividendSize * .exp(-r * timeToDiv)
let putCallParity = (S - pvOfDiv) * .exp(-q * t) - .exp(-r * t) * X
let sampleDividends: [Pricing.Dividend] = [
Pricing.Dividend(amount: dividendSize, timeTo: timeToDiv)
]
var bsValues: [Pricing.Results] = []
let commonParams = Pricing.Parameters.Common(
exerciseType: .european,
spotPrice: Pricing.SideValues(bid: S, ask: S),
financingRate: r,
yield: q,
timeToMaturity: t,
dividends: sampleDividends
)
let specifics = [
Pricing.Parameters.Specific(
marketPrice: Pricing.SideValues(bid: 5.0, ask: 21.0), volatility: σ
),
Pricing.Parameters.Specific(
marketPrice: Pricing.SideValues(bid: 5.0, ask: 21.0), volatility: σ
),
]
let referenceData = [
Pricing.Parameters.InstrumentReferenceData(instrumentType: .call, strikePrice: X),
Pricing.Parameters.InstrumentReferenceData(instrumentType: .put, strikePrice: X),
]
let pricingContext = Pricing.Context(
commonParameters: commonParams,
reference: referenceData,
parameters: specifics,
theoreticalValues: [.fair]
)
BlackScholes.calculate(context: pricingContext, results: &bsValues)
let call = bsValues[0].fair ?? 0.0
let put = bsValues[1].fair ?? 0.0
for (index, result) in bsValues.enumerated() {
let boundaries = ArbitrageBoundaries.validFairValue(
commonParameters: commonParams,
reference: referenceData[index]
)
if let fairResult = result.fair {
// #expect(
// (boundaries?.contains(fairResult)) != false,
// "fair value \(fairResult) is not within \(String(describing: boundaries))"
// )
} else {
#expect(result.fair != nil, "No fair result returned")
}
}
#expect(abs(call - put - putCallParity) < allowedDeviation, "\(call) \(put)")
}
}Expected behavior
We expected approximately the same runtime with the new testing framework.
Actual behavior
We saw a 5-10x test runtime regression for certain tests.
Steps to reproduce
No response
swift-testing version/commit hash
Built in with 6.0 release toolchain with Xcode 16
Swift & OS version (output of swift --version ; uname -a)
swift-driver version: 1.115 Apple Swift version 6.0 (swiftlang-6.0.0.9.10 clang-1600.0.26.2)
Target: arm64-apple-macosx15.0
Darwin ice.local 24.0.0 Darwin Kernel Version 24.0.0: Mon Aug 12 20:51:54 PDT 2024; root:xnu-11215.1.10~2/RELEASE_ARM64_T6000 arm64