-
Notifications
You must be signed in to change notification settings - Fork 6
Description
The old performance testing setup also tests for I/O performance. For large input files, this drives up the required experiment time, as I/O tends to be by far the slowest component in every benchmark. This issue proposes two fixes:
-
every smoke test should read inputs only once. To cope with variability under differing allocations (esp. detected on ARM), the file should be read to an in-memory storage (e.g., an
std::vectorthat stores a nonzero struct for each nonzero, or multiplestd::vectors). The remainder smoke tests then remain unchanged, except every experiment now reads from this in-memory storage, but otherwise still performs new allocations for matrix and vector data; -
a new performance test should be introduced that tests I/O only -- i.e., reads the input multiple times, and reporting all the regular timing statistics (average, min, max, unbiased sample standard deviation).
Thus, the new performance test shall form the only exception to the first item, retains I/O speed benchmarking capabilities, and speeds up all other performance testing in future.