cpu_percent() always returning 100% on osx

[giampaolo]

From jeffery....@gmail.com on October 22, 2010 23:08:04

My machine is essentially idle, but cpu_percent() always returns exactly 100%.

Here is copy/paste from terminal:

Python 2.6.1 ( r261 :67515, Feb 11 2010, 00:51:29) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.cpu_percent()
100.0
>>> What version of psutil are you using? On what operating system? using psutil-0.1.3

Installed from source. Used the command:
python setup.py install 
--prefix=${HOME}/Library/Frameworks/Python.framework/Versions/2.6 

Please provide any additional information below.  
Not sure if it helps, but here is the output from when I installed it:
ph-jeffkline-l01:psutil-0.1.3 kline$ python setup.py install 
--prefix=${HOME}/Library/Frameworks/Python.framework/Versions/2.6
running install
running build
running build_py
creating build
creating build/lib.macosx-10.6-universal-2.6
creating build/lib.macosx-10.6-universal-2.6/psutil
copying psutil/__init__.py -> 
build/lib.macosx-10.6-universal-2.6/psutil
copying psutil/_psbsd.py -> build/lib.macosx-10.6-universal-2.6/psutil
copying psutil/_pslinux.py -> 
build/lib.macosx-10.6-universal-2.6/psutil
copying psutil/_psmswindows.py -> 
build/lib.macosx-10.6-universal-2.6/psutil
copying psutil/_psosx.py -> build/lib.macosx-10.6-universal-2.6/psutil
copying psutil/error.py -> build/lib.macosx-10.6-universal-2.6/psutil
copying psutil/wmi.py -> build/lib.macosx-10.6-universal-2.6/psutil
running build_ext
building '_psutil_osx' extension
creating build/temp.macosx-10.6-universal-2.6
creating build/temp.macosx-10.6-universal-2.6/psutil
creating build/temp.macosx-10.6-universal-2.6/psutil/arch
creating build/temp.macosx-10.6-universal-2.6/psutil/arch/osx
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv 
-Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc 
-arch x86_64 -pipe 
-I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
 -c psutil/_psutil_osx.c -o 
build/temp.macosx-10.6-universal-2.6/psutil/_psutil_osx.o
In file included from psutil/_psutil_osx.c:21:
/usr/include/mach/shared_memory_server.h:48:2: warning: #warning 
"<mach/shared_memory_server.h> is deprecated.  Please use 
<mach/shared_region.h> instead."
psutil/_psutil_osx.c: In function ‘pid_exists’:
psutil/_psutil_osx.c:182: warning: unused variable ‘idx’
psutil/_psutil_osx.c:181: warning: unused variable ‘num_processes’
psutil/_psutil_osx.c:180: warning: unused variable ‘procList’
psutil/_psutil_osx.c: In function ‘get_process_cpu_times’:
psutil/_psutil_osx.c:305: warning: unused variable ‘thread_list’
psutil/_psutil_osx.c:304: warning: unused variable ‘thread_count’
psutil/_psutil_osx.c:303: warning: unused variable ‘t_cpu’
psutil/_psutil_osx.c:302: warning: unused variable ‘t_stime_ms’
psutil/_psutil_osx.c:302: warning: unused variable ‘t_stime’
psutil/_psutil_osx.c:301: warning: unused variable ‘t_utime_ms’
psutil/_psutil_osx.c:301: warning: unused variable ‘t_utime’
psutil/_psutil_osx.c: In function ‘get_memory_info’:
psutil/_psutil_osx.c:451: warning: implicit declaration of function ‘vm_region_64’
psutil/_psutil_osx.c:418: warning: unused variable ‘system_time’
psutil/_psutil_osx.c:418: warning: unused variable ‘user_time’
psutil/_psutil_osx.c: In function ‘get_avail_phymem’:
psutil/_psutil_osx.c:512: warning: implicit declaration of function ‘host_statistics’
psutil/_psutil_osx.c:516: warning: implicit declaration of function ‘mach_error_string’
psutil/_psutil_osx.c:516: warning: format ‘&#37;s’ expects type ‘char 
*’, but argument 3 has type ‘int’
In file included from psutil/_psutil_osx.c:21:
/usr/include/mach/shared_memory_server.h:48:2: warning: #warning 
"<mach/shared_memory_server.h> is deprecated.  Please use 
<mach/shared_region.h> instead."
psutil/_psutil_osx.c: In function ‘pid_exists’:
psutil/_psutil_osx.c:182: warning: unused variable ‘idx’
psutil/_psutil_osx.c:181: warning: unused variable ‘num_processes’
psutil/_psutil_osx.c:180: warning: unused variable ‘procList’
psutil/_psutil_osx.c: In function ‘get_process_cpu_times’:
psutil/_psutil_osx.c:305: warning: unused variable ‘thread_list’
psutil/_psutil_osx.c:304: warning: unused variable ‘thread_count’
psutil/_psutil_osx.c:303: warning: unused variable ‘t_cpu’
psutil/_psutil_osx.c:302: warning: unused variable ‘t_stime_ms’
psutil/_psutil_osx.c:302: warning: unused variable ‘t_stime’
psutil/_psutil_osx.c:301: warning: unused variable ‘t_utime_ms’
psutil/_psutil_osx.c:301: warning: unused variable ‘t_utime’
psutil/_psutil_osx.c: In function ‘get_memory_info’:
psutil/_psutil_osx.c:451: warning: implicit declaration of function ‘vm_region_64’
psutil/_psutil_osx.c:418: warning: unused variable ‘system_time’
psutil/_psutil_osx.c:418: warning: unused variable ‘user_time’
psutil/_psutil_osx.c: In function ‘get_avail_phymem’:
psutil/_psutil_osx.c:512: warning: implicit declaration of function ‘host_statistics’
psutil/_psutil_osx.c:516: warning: implicit declaration of function ‘mach_error_string’
psutil/_psutil_osx.c:516: warning: format ‘&#37;s’ expects type ‘char 
*’, but argument 3 has type ‘int’
In file included from psutil/_psutil_osx.c:21:
/usr/include/mach/shared_memory_server.h:48:2: warning: #warning 
"<mach/shared_memory_server.h> is deprecated.  Please use 
<mach/shared_region.h> instead."
psutil/_psutil_osx.c: In function ‘pid_exists’:
psutil/_psutil_osx.c:182: warning: unused variable ‘idx’
psutil/_psutil_osx.c:181: warning: unused variable ‘num_processes’
psutil/_psutil_osx.c:180: warning: unused variable ‘procList’
psutil/_psutil_osx.c: In function ‘get_process_cpu_times’:
psutil/_psutil_osx.c:307: warning: cast from pointer to integer of different size
psutil/_psutil_osx.c:305: warning: unused variable ‘thread_list’
psutil/_psutil_osx.c:304: warning: unused variable ‘thread_count’
psutil/_psutil_osx.c:303: warning: unused variable ‘t_cpu’
psutil/_psutil_osx.c:302: warning: unused variable ‘t_stime_ms’
psutil/_psutil_osx.c:302: warning: unused variable ‘t_stime’
psutil/_psutil_osx.c:301: warning: unused variable ‘t_utime_ms’
psutil/_psutil_osx.c:301: warning: unused variable ‘t_utime’
psutil/_psutil_osx.c: In function ‘get_memory_info’:
psutil/_psutil_osx.c:417: warning: cast from pointer to integer of different size
psutil/_psutil_osx.c:451: warning: implicit declaration of function ‘vm_region_64’
psutil/_psutil_osx.c:418: warning: unused variable ‘system_time’
psutil/_psutil_osx.c:418: warning: unused variable ‘user_time’
psutil/_psutil_osx.c: In function ‘get_avail_phymem’:
psutil/_psutil_osx.c:512: warning: implicit declaration of function ‘host_statistics’
psutil/_psutil_osx.c:516: warning: implicit declaration of function ‘mach_error_string’
psutil/_psutil_osx.c:516: warning: format ‘&#37;s’ expects type ‘char 
*’, but argument 3 has type ‘int’
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv 
-Os -Wall -Wstrict-prototypes ...

Original issue: http://code.google.com/p/psutil/issues/detail?id=120

[giampaolo]

From g.rodola on October 22, 2010 14:15:12

Owner: jloden
Labels: OpSys-OSX

[giampaolo]

From g.rodola on October 22, 2010 14:15:50

Is that a 64 bit system?

[giampaolo]

From jeffery....@gmail.com on October 22, 2010 14:25:49

Yes, here is output from system profiler:

  Model Name:   MacBook Pro
  Model Identifier: MacBookPro6,2
  Processor Name:   Intel Core i7
  Processor Speed:  2.66 GHz
  Number Of Processors: 1
  Total Number Of Cores:    2

[giampaolo]

From jlo...@gmail.com on October 22, 2010 14:35:58

Can you show the output of the below command as well? 

>>> print psutil.cpu_times(); time.sleep(5); print psutil.cpu_times()
system=3.395108; idle=37.254002; user=4.461708; nice=0.0
system=3.395162; idle=37.254894; user=4.461761; nice=0.0

I'm getting similar results here (not 100% but close, 99.99% which 
probably means the issue is the same). Giampaolo, can you check this 
on any other platforms you have handy and see if this is OS X only? 
I'd like to figure out if this is in the C code or the Python code 
that handles the CPU calculations.

Thanks

[giampaolo]

From jeffery....@gmail.com on October 22, 2010 14:42:15

Here is my output:

>>> import psutil, time
>>> print psutil.cpu_times(); time.sleep(5); print psutil.cpu_times()
system=1e-06; idle=4294.967295; user=2.112688; nice=0.0
system=1e-06; idle=4294.967295; user=2.112688; nice=0.0
>>>

[giampaolo]

From jlo...@gmail.com on October 22, 2010 14:49:39

I'll have to look at this some more later but I think the 
get_system_cpu_times() is not returning valid results for some reason: 

>>> _psutil_osx.get_system_cpu_times(); time.sleep(5);  
_psutil_osx.get_system_cpu_times()
(4.4710289999999997, 0.0, 3.4054090000000001, 37.401888)
(4.4710669999999997, 0.0, 3.4054660000000001, 37.402794999999998)

After 5 seconds it's giving almost identical values, and I suspect 
any differences in the values are probably from floating point math...

[giampaolo]

From g.rodola on October 22, 2010 15:34:13

Linux 64bit, 2 cores:

giampaolo@ubuntu:~/svn/psutil$ python -c "import psutil, time; print 
psutil.cpu_times(); time.sleep(5); print psutil.cpu_times()"
softirq=33.91; iowait=1355.06; system=3944.62; idle=73988.76; 
user=5297.51; irq=0.01; nice=339.7
softirq=33.91; iowait=1355.06; system=3945.14; idle=73997.73; 
user=5298.01; irq=0.01; nice=339.72


FreeBSD 7.0 32bit, 1 core:

[root@freebsd ~/psutil]# python -c "import psutil, time; print 
psutil.cpu_times(); time.sleep(5); print psutil.cpu_times()"
system=696.9296875; irq=7197.453125; idle=32646.4296875; 
user=81.4296875; nice=0.0078125
system=696.984375; irq=7200.296875; idle=32646.4296875; 
user=83.9765625; nice=0.0078125


Windows 7 64 (py 32 bit), 1 core:

C:\Users\user\Desktop\svn\psutil>C:\python27\python -c "import 
psutil, time; print psutil.cpu_times(); time.sleep(5); print psutil.cpu_times()"
idle=16220.328125; system=1174.375; user=81.71875
idle=16224.90625; system=1174.796875; user=81.734375

[giampaolo]

From jlo...@gmail.com on October 22, 2010 18:12:26

Thanks giampaolo, what about cpu_percent, does it look accurate?

[giampaolo]

From jlo...@gmail.com on October 22, 2010 18:12:38

Status: Accepted

[giampaolo]

From g.rodola on October 23, 2010 03:52:59

Aside from an initial "100%" which perhaps we could adjust someway 
results look ok and match top/taskmgr on BSD/Linux/Windows on both 
single and double CPUs.
The script:

import time, psutil, threading

def foo():
    while 1:
        pass
t = threading.Thread(target=foo).start()

while 1:
    print psutil.cpu_percent()
    time.sleep(1)


In r724 I added a test based on the results you pasted in msg #4 
which should fail on OSX so you can at least use that to test results.

[giampaolo]

From jlo...@gmail.com on October 23, 2010 08:50:34

> In r724 I added a test based on the results you pasted in msg #4 which 
> should fail on OSX so you can at least use that to test results.

Thanks... I took a look at this some more last night but I am still working on it. 

I can see the problem (times not updating properly) but I don't know 
if the system call is just returning strange data or if something has 
changed in 10.6 in the way statistics are reported/calculated.

[giampaolo]

From jlo...@gmail.com on October 23, 2010 08:50:43

Status: Started

[giampaolo]

From g.rodola on October 23, 2010 10:02:38

Point is times are updated, but not as much as expected, so I agree 
with you that this is likely to be a math problem. Maybe we shouldn't 
divide per CLOCKS_PER_SEC as we do on BSD:

- (double)r_load.cpu_ticks[CPU_STATE_USER] / CLOCKS_PER_SEC,
+ (double)r_load.cpu_ticks[CPU_STATE_USER],

[giampaolo]

From jlo...@gmail.com on October 23, 2010 10:17:59

I suspect it's a math problem but I don't know why, or what to use 
instead of CLOCKS_PER_SEC... we can't just return it as is because 
it'd be returning clock ticks for the values, and our math in the 
cpu_percent() function depends on seconds. 

In theory, dividing by CLOCKS_PER_SEC is the correct thing to do but 
it's evidently not working correctly, so something is out of whack :-/