General speed performance

[piotrmurach]

Hi RSpec team,

Many thanks for making the RSpec 3 to be the best release ever!

Recently I have found that RSpec performance has degraded for me when running test suite for finite_machine. Obviously the times are relative to my machine and the size of test suite but they consistently vary between versions. Version 3.1.0 of rspec runs the whole test suite in under 1.4s (files load in 0.3s), whereas v3.3.0 on average takes 1.6s (files load in 0.32s). Though it may seems like splitting hairs but I try hard to keep my code as well as tests running relatively fast. Thus following questions:

• Did certain parts of the public api got slower during minor releases?
• Does RSpec codebase have any benchmark suite to ensure performance of the api stays at certain level?
• What is RSpec team view on speed performance measuring for the rspec library? (happy to contribute given guidance)

[myronmarston]

Did certain parts of the public api got slower during minor releases?

That's certainly possible, but no users have reported slowness before now. The one thing we added that we're aware made things slightly slower is #1749 -- but note that we did lots of benchmarking/optimization in that PR such that affects of the changes in that PR would be minimal for those who are not using the features that PR enables. Also, that change was first included in 3.2, and the 3.2 release also included a number of performance improvements. By our calculation the perf improvements in 3.2 were far greater than the minor degredation caused by #1749 and in fact my benchmark showed that 3.2 has 30% fewer object allocations than 3.1 and ran noticeably faster than 3.1 (1.33 seconds vs 1.59 seconds in that benchmark). We reported perf improvements as being notable in the 3.2 release and the reports we got from users was that 3.2 was in fact faster for them.

It would be interesting to use git bisect against your suite between 3.1 and 3.3 to see if the slowness was introduced by any particular commit.

Does RSpec codebase have any benchmark suite to ensure performance of the api stays at certain level?

We have lots of benchmarks -- and the other repos have benchmarks, too -- but we just use these as micro-benchmarks to make one-off decisions about what implementations to go with for particular things. We don't run them on an ongoing basis and we don't have anything holistic that benchmarks RSpec from it's API. We've talked about wanting something like that, though (rspec/rspec-dev#45).

What is RSpec team view on speed performance measuring for the rspec library? (happy to contribute given guidance)

We definitely want RSpec to get faster and stay fast. Contributions for the purpose of perf improvements generally need benchmarks included in the PRs. If you have any ideas for improvements or for improving our process/tooling around this, we're all ears!

[myronmarston]

I just remembered -- there was also a change in 3.3 (#1858) that made let and subject slightly slower but we didn't believe it would be noticeable. Still, you can try setting config.threadsafe = false to disable that feature and see if it makes your suite faster.

[piotrmurach]

Thank you for quick reply!

We definitely want RSpec to get faster and stay fast. Contributions for the purpose of perf improvements generally need benchmarks included in the PRs.

I will look through the current benchmarks more thoroughly but having had a quick look it appears that it would be beneficial to have a benchmark that puts majority of the features rspec supports into one benchmark, for example, a simple tiny library of some coupled objects.

The v3.2.0 and v3.3.0 seem to show equal times on average. It is only when I switch to v3.1.0 I get around 0.2s boost for the test suite without any changes. This implies to me that the potentially performance degrading changes have been part of v3.2.0. I'm running tests on ruby 2.0.0. I was wondering if there is certain abundance of certain features that I overuse in my tests that may cause the slowdown. I'm using let and subject sparingly and have been recently replacing them with just repeated code to localise their usage.

[myronmarston]

The slowdown is definitely surprising in your case. Would you be interested in running git bisect from v3.1.0 to v3.2.0 to see if you can pinpoint where the slowness comes from?

[piotrmurach]

Sure thing, I will try to zero in on this, but it will definitely have to wait till tomorrow as I'm currently preparing a library release and this takes precedence.

[fables-tales]

I'm going to try bisecting this now.

[fables-tales]

I cloned the finite machine gem and ran its suite against various RSpec versions. Measuring 5 runs of RSpec v3.1.0 vs v3.4.0 gives:

v3.4.0: [1.615, 1.556, 1.512, 1.544, 1.483], mean 1.542
v3.1.0: [1.765, 1.504, 1.477, 1.480, 1.488], mean 1.543

So... I'm not convinced this affect exists.

@peter-murach if you can provide a specific commit where your suite becomes significantly slower against ruby 2.0.0 at the latest patchlevel? I'm going to close this issue for now until we have a specific commit and a reproducible benchmark.