Closed
Description
I was working on some HTTP parser improvements and I noticed the numbers from the http/simple benchmark fluctuate wildly, even when the system is otherwise quiescent. This is with master, without my changes applied.
Results from 10 runs. Columns are mean(x) median(x) std(x) std(x)/median(x)*100
.
http/simple.js c=50 chunks=0 length=4 type="bytes": 6041.3 6201.2 625.4 10.1%
http/simple.js c=500 chunks=0 length=4 type="bytes": 4706.7 4731.9 264.0 5.6%
http/simple.js c=50 chunks=1 length=4 type="bytes": 5683.0 5600.1 289.0 5.2%
http/simple.js c=500 chunks=1 length=4 type="bytes": 4435.2 4444.8 164.2 3.7%
http/simple.js c=50 chunks=4 length=4 type="bytes": 1157.7 1161.3 13.1 1.1%
http/simple.js c=500 chunks=4 length=4 type="bytes": 2797.2 2835.9 145.1 5.1%
http/simple.js c=50 chunks=0 length=1024 type="bytes": 5950.9 6013.6 318.8 5.3%
http/simple.js c=500 chunks=0 length=1024 type="bytes": 4648.4 4601.7 346.3 7.5%
http/simple.js c=50 chunks=1 length=1024 type="bytes": 5494.6 5365.3 297.9 5.6%
http/simple.js c=500 chunks=1 length=1024 type="bytes": 4367.5 4403.7 375.8 8.5%
http/simple.js c=50 chunks=4 length=1024 type="bytes": 1159.9 1161.8 6.8 0.6%
http/simple.js c=500 chunks=4 length=1024 type="bytes": 2772.3 2744.5 90.9 3.3%
http/simple.js c=50 chunks=0 length=102400 type="bytes": 1587.4 1582.8 28.9 1.8%
http/simple.js c=500 chunks=0 length=102400 type="bytes": 1458.2 1450.0 36.9 2.5%
http/simple.js c=50 chunks=1 length=102400 type="bytes": 952.4 958.8 27.1 2.8%
http/simple.js c=500 chunks=1 length=102400 type="bytes": 864.7 877.1 43.3 4.9%
http/simple.js c=50 chunks=4 length=102400 type="bytes": 1560.2 1612.4 121.2 7.5%
http/simple.js c=500 chunks=4 length=102400 type="bytes": 1454.6 1444.4 102.7 7.1%
http/simple.js c=50 chunks=0 length=4 type="buffer": 5817.7 5801.9 193.1 3.3%
http/simple.js c=500 chunks=0 length=4 type="buffer": 4655.4 4726.7 191.8 4.1%
http/simple.js c=50 chunks=1 length=4 type="buffer": 5736.3 5918.6 298.0 5.0%
http/simple.js c=500 chunks=1 length=4 type="buffer": 4478.3 4400.9 394.2 9.0%
http/simple.js c=50 chunks=4 length=4 type="buffer": 5348.3 5386.7 288.5 5.4%
http/simple.js c=500 chunks=4 length=4 type="buffer": 4215.7 4174.4 254.0 6.1%
http/simple.js c=50 chunks=0 length=1024 type="buffer": 5725.0 5748.3 373.2 6.5%
http/simple.js c=500 chunks=0 length=1024 type="buffer": 4503.7 4430.6 417.6 9.4%
http/simple.js c=50 chunks=1 length=1024 type="buffer": 5584.4 5621.3 189.1 3.4%
http/simple.js c=500 chunks=1 length=1024 type="buffer": 4221.0 4170.0 368.7 8.8%
http/simple.js c=50 chunks=4 length=1024 type="buffer": 4897.6 4860.7 397.4 8.2%
http/simple.js c=500 chunks=4 length=1024 type="buffer": 4015.6 3935.0 343.4 8.7%
http/simple.js c=50 chunks=0 length=102400 type="buffer": 4451.8 4370.2 160.1 3.7%
http/simple.js c=500 chunks=0 length=102400 type="buffer": 3323.9 3356.2 209.9 6.3%
http/simple.js c=50 chunks=1 length=102400 type="buffer": 4168.1 4168.7 256.0 6.1%
http/simple.js c=500 chunks=1 length=102400 type="buffer": 3326.4 3310.8 267.8 8.1%
http/simple.js c=50 chunks=4 length=102400 type="buffer": 3930.1 3945.4 202.3 5.1%
http/simple.js c=500 chunks=4 length=102400 type="buffer": 3128.2 3143.2 139.6 4.4%
Everything with a standard deviation > 1-2% is so unreliable as to be worthless, IMO.
Can someone confirm whether they are seeing similar fluctuations?
cc @nodejs/benchmarking