Skip to content

Repeated performance tests should skip repeated file I/O #264

@anyzelman

Description

@anyzelman

The old performance testing setup also tests for I/O performance. For large input files, this drives up the required experiment time, as I/O tends to be by far the slowest component in every benchmark. This issue proposes two fixes:

  • every smoke test should read inputs only once. To cope with variability under differing allocations (esp. detected on ARM), the file should be read to an in-memory storage (e.g., an std::vector that stores a nonzero struct for each nonzero, or multiple std::vectors). The remainder smoke tests then remain unchanged, except every experiment now reads from this in-memory storage, but otherwise still performs new allocations for matrix and vector data;

  • a new performance test should be introduced that tests I/O only -- i.e., reads the input multiple times, and reporting all the regular timing statistics (average, min, max, unbiased sample standard deviation).

Thus, the new performance test shall form the only exception to the first item, retains I/O speed benchmarking capabilities, and speeds up all other performance testing in future.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions