CPU usage of runtests.py

[gvanrossum]

When @gnprice added the -j flag, I tried runtests.py lint with a few different -j settings and found that the more parallellism, the more CPU it used (also less real time, but that was more expected ;-). I wrote down some timings in #1066 (comment), repeated here:

-j4

real  0m13.783s
user  0m38.849s
sys   0m5.867s

-j2 (default)

real  0m16.646s
user  0m28.247s
sys   0m4.880s

-j1

real  0m25.990s
user  0m22.901s
sys   0m4.261s

Is there some busy-waiting going on somewhere?

[JukkaL]

I don't see similar high variability on my Linux desktop when using up to 4 parallel processes (4 cores, 8 threads, Intel Core i7 2600K, stock clock, Ubuntu 14.04):

(mypy)jukka mypy $ time ./runtests.py -j1 lint
PARALLEL 1
SUMMARY  78 tasks selected
SUMMARY  all 78 tasks and 78 tests passed                                       
*** OK ***

real    0m17.610s
user    0m16.160s
sys 0m3.389s
(mypy)jukka mypy $ time ./runtests.py -j2 lint
PARALLEL 2
SUMMARY  78 tasks selected
SUMMARY  all 78 tasks and 78 tests passed                                       
*** OK ***

real    0m8.729s
user    0m15.892s
sys 0m2.926s
(mypy)jukka mypy $ time ./runtests.py -j4 lint
PARALLEL 4
SUMMARY  78 tasks selected
SUMMARY  all 78 tasks and 78 tests passed                                       
*** OK ***

real    0m5.292s
user    0m18.832s
sys 0m3.352s

But see what happens when I try -j8:

(mypy)jukka mypy $ time ./runtests.py -j8 lint
PARALLEL 8
SUMMARY  78 tasks selected
SUMMARY  all 78 tasks and 78 tests passed                                       
*** OK ***

real    0m4.338s
user    0m27.515s
sys 0m4.176s

My guess is that you were running this on a 2-core CPU with 4 threads -- then increasing parallelism from 2 to 4 would only give small benefits, similar to me going from 4 to 8 parallel processes.

I also checked that I got almost linear scaling up to -j2 on my dual-core Retina Macbook Pro but then only minimal improvements as expected.

[gvanrossum]

Yes, I have only two cores (poor me :-). It still surprises me but I guess this is overhead due to excessive switching between cores. Oh well. Never mind me.

[JukkaL]

Core i7 typically has 4/8 logical cores ("threads" in marketing materials) vs 2/4 physical cores. The reason for what you saw seems to be that every core actually slows down when all the logical cores are being utilized (as the extra "logical cores" are sharing physical CPU resources with another logical core using the same physical core). So, when using high levels of parallelism the reported CPU time is higher because all cores run slower. As there are more things running in parallel, the perceived performance is usually still better, but not as much as when having a large number of real CPU cores.