Speeding up Tests

[pradyunsg]

The test suite is slow. It takes a long time to run them and to get results. Because of this, occasionally the CI builds timeout due to how long it takes for the builds to complete.

There are various issues that affect the speed of the test suite. This issue is an overall tracking issue for them. I'll take a shot at each of those, given that I have the time. The intent is to try to speed up the test suite without compromising the degree of confidence on it.

---

• Investigate CI improvements
  • Containers for Travis (Travis: Improve build speeds and upgrade PyPy version #4590)
  • Breakup tests across 2 workers (CI: Split PyPy tests to speed up end-to-end running time #5436)
• Look into speeding up the virtualenv fixture (Speedup testsuite - venv creation #4706)
• Require all non-pip invoking tests to be unit tests
• Categorize all non-script tests as unit tests
• Increasing unit test coverage
• Remove redundant tests -- Most of the tests in tests/functional/test_install_cleanup.py aren't testing anything #2640
• Simplify or Break up slow tests -- tests take too long to run #1721
• Speed up pip's startup time (Speedup startup time #4768)
• In-Memory tests where appropriate
• Removing exhaustive file checks in scripttest (WIP: Remove exhaustive file checks #7141)

(I'll add more as they come)

[pradyunsg]

Just for self-reference later, here are the integration test run times (as reported by pytest) of the last 5 runs of the tests on Travis CI (on master):

Version	#6914	#6913	#6901	#6899	#6898
2.7	1491.73	1499.83	1461.50	1465.82	1465.74
3.3	1758.89	1662.34	1653.14	1648.96	1588.04
3.4	1589.49	1757.04	1696.77	1687.61	1608.33
3.5	1797.19	1795.63	1645.96	1603.81	1658.20
3.6	1669.28	1759.57	1814.60	1669.06	1695.59
pypy	2566.34	2579.24	2633.35	2575.63	2518.47

[pradyunsg]

#4586 shows a huge speedup in the CI builds. That doesn't mean this isn't useful. :)

[pradyunsg]

It seems that virtualenv fixture takes up a huge part of the test time... A rough test shows that 98 second test run spends 35 seconds in that fixture.

[pfmoore]

Would it be worth using venv when available as a quick fix? I don't know if venv is quicker, but it might be worth a try.

[pradyunsg]

Oh, I missed that last comment. I'll look into it soon. :)

---

FTR - now that YAML tests have been added, a bunch of installation tests can removed and made into YAML fixtures. That would reduce the clutter in data/packages. :)

[pradyunsg]

Update on status quo:

Version	#6914	#7594
2.7	1491.73	645.50
3.6	1669.28	767.25
pypy	2566.34	1500.26

[hugovk]

Here are the start-to-finish "Ran for X min Y sec" waiting times of a full build, for the three master builds before #5436 was merged, and the three after.

Build	Time
8769	34 min 46 sec
8771	35 min 25 sec
8779	34 min 54 sec
Average	35 min 02 sec
8786	20 min 27 sec
8787	19 min 47 sec
8793	19 min 49 sec
Average	20 min 01 sec

[pradyunsg]

@hugovk Not to suggest that the CI improvements aren't awesome; this issue is for tracking improving the test-suite speed -- which affects both local development and CI.

Hence the timings noted above being of "test run times", not CI run times. :)

[pradyunsg]

It might be handy to look at https://github.com/kvas-it/pytest-console-scripts, either as a replacement for our use of scripttest or as a way to extend scripttest.

I prefer the former.