Skip to content

benchmark: http/simple is unreliable #8139

Closed
@bnoordhuis

Description

@bnoordhuis

I was working on some HTTP parser improvements and I noticed the numbers from the http/simple benchmark fluctuate wildly, even when the system is otherwise quiescent. This is with master, without my changes applied.

Results from 10 runs. Columns are mean(x) median(x) std(x) std(x)/median(x)*100.

http/simple.js c=50 chunks=0 length=4 type="bytes":         6041.3  6201.2  625.4  10.1%
http/simple.js c=500 chunks=0 length=4 type="bytes":        4706.7  4731.9  264.0   5.6%
http/simple.js c=50 chunks=1 length=4 type="bytes":         5683.0  5600.1  289.0   5.2%
http/simple.js c=500 chunks=1 length=4 type="bytes":        4435.2  4444.8  164.2   3.7%
http/simple.js c=50 chunks=4 length=4 type="bytes":         1157.7  1161.3   13.1   1.1%
http/simple.js c=500 chunks=4 length=4 type="bytes":        2797.2  2835.9  145.1   5.1%
http/simple.js c=50 chunks=0 length=1024 type="bytes":      5950.9  6013.6  318.8   5.3%
http/simple.js c=500 chunks=0 length=1024 type="bytes":     4648.4  4601.7  346.3   7.5%
http/simple.js c=50 chunks=1 length=1024 type="bytes":      5494.6  5365.3  297.9   5.6%
http/simple.js c=500 chunks=1 length=1024 type="bytes":     4367.5  4403.7  375.8   8.5%
http/simple.js c=50 chunks=4 length=1024 type="bytes":      1159.9  1161.8    6.8   0.6%
http/simple.js c=500 chunks=4 length=1024 type="bytes":     2772.3  2744.5   90.9   3.3%
http/simple.js c=50 chunks=0 length=102400 type="bytes":    1587.4  1582.8   28.9   1.8%
http/simple.js c=500 chunks=0 length=102400 type="bytes":   1458.2  1450.0   36.9   2.5%
http/simple.js c=50 chunks=1 length=102400 type="bytes":     952.4   958.8   27.1   2.8%
http/simple.js c=500 chunks=1 length=102400 type="bytes":    864.7   877.1   43.3   4.9%
http/simple.js c=50 chunks=4 length=102400 type="bytes":    1560.2  1612.4  121.2   7.5%
http/simple.js c=500 chunks=4 length=102400 type="bytes":   1454.6  1444.4  102.7   7.1%
http/simple.js c=50 chunks=0 length=4 type="buffer":        5817.7  5801.9  193.1   3.3%
http/simple.js c=500 chunks=0 length=4 type="buffer":       4655.4  4726.7  191.8   4.1%
http/simple.js c=50 chunks=1 length=4 type="buffer":        5736.3  5918.6  298.0   5.0%
http/simple.js c=500 chunks=1 length=4 type="buffer":       4478.3  4400.9  394.2   9.0%
http/simple.js c=50 chunks=4 length=4 type="buffer":        5348.3  5386.7  288.5   5.4%
http/simple.js c=500 chunks=4 length=4 type="buffer":       4215.7  4174.4  254.0   6.1%
http/simple.js c=50 chunks=0 length=1024 type="buffer":     5725.0  5748.3  373.2   6.5%
http/simple.js c=500 chunks=0 length=1024 type="buffer":    4503.7  4430.6  417.6   9.4%
http/simple.js c=50 chunks=1 length=1024 type="buffer":     5584.4  5621.3  189.1   3.4%
http/simple.js c=500 chunks=1 length=1024 type="buffer":    4221.0  4170.0  368.7   8.8%
http/simple.js c=50 chunks=4 length=1024 type="buffer":     4897.6  4860.7  397.4   8.2%
http/simple.js c=500 chunks=4 length=1024 type="buffer":    4015.6  3935.0  343.4   8.7%
http/simple.js c=50 chunks=0 length=102400 type="buffer":   4451.8  4370.2  160.1   3.7%
http/simple.js c=500 chunks=0 length=102400 type="buffer":  3323.9  3356.2  209.9   6.3%
http/simple.js c=50 chunks=1 length=102400 type="buffer":   4168.1  4168.7  256.0   6.1%
http/simple.js c=500 chunks=1 length=102400 type="buffer":  3326.4  3310.8  267.8   8.1%
http/simple.js c=50 chunks=4 length=102400 type="buffer":   3930.1  3945.4  202.3   5.1%
http/simple.js c=500 chunks=4 length=102400 type="buffer":  3128.2  3143.2  139.6   4.4%

Everything with a standard deviation > 1-2% is so unreliable as to be worthless, IMO.

Can someone confirm whether they are seeing similar fluctuations?

cc @nodejs/benchmarking

Metadata

Metadata

Assignees

No one assigned

    Labels

    benchmarkIssues and PRs related to the benchmark subsystem.httpIssues or PRs related to the http subsystem.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions