So pytest-benchmark is already very good at comparing different variations of a test function, and also comparing different versions of code thanks to --benchmark-save and --benchmark-compare. Hurray!
I would love to also compare an operation's performance across different input sizes. Say I have an algorithm with two dimensions of input, n and m, and I assume the algorithm to have runtime O(n). To verify correct implementation I could choose a matrix of parameters for n and m, and divide all timings by n before printing to the comparison table, essentially measuring "runtime per element", not "runtime per operation".
I would assume to see a little overhead on small values for parameter n, but overall, the "runtime per element" would be approaching a constant factor for larger values of n. If this is not the case, or seemling different values for m cause a nontrivial amount of variation in the runtime per element, it would seem I have not truly achieved runtime O(n) and need to improve my implementation.
At the moment I'm eyeballing this from the results, approximating the division by input size in my head. This kind of works for linear time complexity, but if my algorithm is, say, logarithmic in time, I can forget about that and will need to create a spreadsheet. Of course, nobody wants that :P
It would be good if I could supply a timing normalization function per test case that operates on the parameters and some measured duration to compute a normalized duration. I could divide by log(n) in that function too, if I assume logarithmic time complexity.
I have achieved this doing some monkey patching, but of course that is ugly:
from pytest_benchmark.stats import Metadata
original = Metadata.update
def monkeypatched_metadata_update(self, duration):
self.stats.update(duration / self.iterations / n) # <----------- normalizing here!
try:
Metadata.update = monkeypatched_metadata_update
benchmark(my_test)
finally:
Metadata.update = originalThe result (I've edited the table a bit for clarity) shows nicely how my OPS or Mean appear to follow O(n) for large n and seem not to depend on m too much:
------------------------------------------------------ benchmark: 16 tests --------------------------
Name n m Min Max Mean OPS (Mops/s)
-----------------------------------------------------------------------------------------------------
my_test 10.000 100 1.0851 (1.0) 1.6805 (1.50) 1.0983 (1.0) 910.4756 (1.00)
my_test 100.000 100 1.0879 (1.00) 1.2075 (1.08) 1.1031 (1.00) 906.5639 (1.00)
my_test 10.000 1.000 1.0968 (1.01) 1.8141 (1.62) 1.1113 (1.01) 899.8729 (0.99)
my_test 10.000 100.000 1.0977 (1.01) 1.8788 (1.68) 1.1255 (1.02) 888.4684 (0.98)
my_test 10.000 10.000 1.0982 (1.01) 1.5467 (1.38) 1.1231 (1.02) 890.4150 (0.98)
my_test 100.000 1.000 1.0997 (1.01) 1.3460 (1.20) 1.1340 (1.03) 881.8282 (0.97)
my_test 100.000 10.000 1.0997 (1.01) 1.2193 (1.09) 1.1234 (1.02) 890.1755 (0.98)
my_test 100.000 100.000 1.1004 (1.01) 1.2076 (1.08) 1.1236 (1.02) 889.9990 (0.98)
my_test 1.000.000 100 1.1068 (1.02) 1.3093 (1.17) 1.1334 (1.03) 882.3242 (0.97)
my_test 10.000.000 100 1.1083 (1.02) 1.1192 (1.0) 1.1133 (1.01) 898.2257 (0.99)
my_test 1.000.000 1.000 1.1180 (1.03) 1.2152 (1.09) 1.1327 (1.03) 882.8179 (0.97)
my_test 1.000.000 10.000 1.1184 (1.03) 1.3050 (1.17) 1.1318 (1.03) 883.5809 (0.97)
my_test 10.000.000 1.000 1.1185 (1.03) 1.1379 (1.02) 1.1256 (1.02) 888.3814 (0.98)
my_test 1.000.000 100.000 1.1187 (1.03) 1.3280 (1.19) 1.1330 (1.03) 882.6440 (0.97)
my_test 10.000.000 10.000 1.1532 (1.06) 1.2086 (1.08) 1.1646 (1.06) 858.6855 (0.94)
my_test 10.000.000 100.000 1.1543 (1.06) 1.2005 (1.07) 1.1675 (1.06) 856.5528 (0.94)
-----------------------------------------------------------------------------------------------------
It would be really good to have this as a quick and supported way, such as:
@pytest.mark.parametrize("n", [1, 10, 100, 1000, 1e5, 1e6, 1e7, 1e8])
@pytest.mark.parametrize("m", [1, 10, 100, 1000, 1e5, 1e6, 1e7, 1e8])
def test_my_test(benchmark, n, m):
inputs = create_input_arrays(n, m)
my_test = partial(my_implementation, inputs)
# overwrite the normalization behaviour
benchmark.normalize_duration = lambda duration: duration / n
benchmark(my_test)I'm very willing to implement this and create a PR.
So pytest-benchmark is already very good at comparing different variations of a test function, and also comparing different versions of code thanks to
--benchmark-saveand--benchmark-compare. Hurray!I would love to also compare an operation's performance across different input sizes. Say I have an algorithm with two dimensions of input, n and m, and I assume the algorithm to have runtime
O(n). To verify correct implementation I could choose a matrix of parameters for n and m, and divide all timings by n before printing to the comparison table, essentially measuring "runtime per element", not "runtime per operation".I would assume to see a little overhead on small values for parameter n, but overall, the "runtime per element" would be approaching a constant factor for larger values of n. If this is not the case, or seemling different values for m cause a nontrivial amount of variation in the runtime per element, it would seem I have not truly achieved runtime
O(n)and need to improve my implementation.At the moment I'm eyeballing this from the results, approximating the division by input size in my head. This kind of works for linear time complexity, but if my algorithm is, say, logarithmic in time, I can forget about that and will need to create a spreadsheet. Of course, nobody wants that :P
It would be good if I could supply a timing normalization function per test case that operates on the parameters and some measured duration to compute a normalized duration. I could divide by
log(n)in that function too, if I assume logarithmic time complexity.I have achieved this doing some monkey patching, but of course that is ugly:
The result (I've edited the table a bit for clarity) shows nicely how my OPS or Mean appear to follow
O(n)for largenand seem not to depend onmtoo much:It would be really good to have this as a quick and supported way, such as:
I'm very willing to implement this and create a PR.