Could not import psutil on FreeBSD 9

[giampaolo]

From sunrize...@gmail.com on September 13, 2012 18:34:39

What steps will reproduce the problem?  
1. Install with pip on FreeBSD 9.0-RELEASE-p3, Python 2.7.3, in virtualenv.
2. Open python shell.
3. import psutil 

What is the expected output?  
The module is imported. 

What do you see instead?  
Python 2.7.3 (default, Sep 13 2012, 11:46:06)
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File 
"/usr/home/voodoo_people/python/lib/python2.7/site-packages/psutil/__init__.py",
 line 88, in <module>
    import psutil._psbsd as _psplatform
  File 
"/usr/home/voodoo_people/python/lib/python2.7/site-packages/psutil/_psbsd.py", 
line 28, in <module>
    TOTAL_PHYMEM = _psutil_bsd.get_virtual_mem()[0]
SystemError: error return without exception set 

What version of psutil are you using? What Python version?  
psutil 0.6.1, Python 2.7.3. 

On what operating system? Is it 32bit or 64bit version?  
Please provide any additional information below.

Original issue: http://code.google.com/p/psutil/issues/detail?id=325

[giampaolo]

From g.rodola on September 13, 2012 09:36:14

Have you tried latest SVN revision?

[giampaolo]

From sunrize...@gmail.com on September 13, 2012 10:05:42

The last SVN version:

Python 2.7.3 (default, Sep 13 2012, 11:46:06)
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
/usr/home/voodoo_people/python/lib/python2.7/site-packages/psutil-0.6.2-py2.7-freebsd-9.0-RELEASE-p3-amd64.egg/_psutil_bsd.py:3:
 UserWarning: Module _psutil_bsd was already imported from 
/usr/home/voodoo_people/python/lib/python2.7/site-packages/psutil-0.6.2-py2.7-freebsd-9.0-RELEASE-p3-amd64.egg/_psutil_bsd.pyc,
 but /usr/home/wyse/build/psutil-read-only is being added to sys.path
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "psutil/__init__.py", line 88, in <module>
    import psutil._psbsd as _psplatform
  File "psutil/_psbsd.py", line 28, in <module>
    TOTAL_PHYMEM = _psutil_bsd.get_virtual_mem()[0]
SystemError: error return without exception set
>>>

[giampaolo]

From g.rodola on September 13, 2012 10:07:18

Ok, something is definitively wrong then.
I won't be able to look into this any time soon though.

[giampaolo]

From sunrize...@gmail.com on September 13, 2012 10:19:59

Just got it running with 0.4.1 version.

[giampaolo]

From g.rodola on December 13, 2012 06:31:53

Getting back to this.
Unfortunately I'm not able to reproduce the issue.

Would you be able to try to debug the problem yourself in order to figure out 
what's the function which is causing the error?

You should look here: 
https://code.google.com/p/psutil/source/browse/trunk/psutil/_psutil_bsd.c?spec=svn1550&r=1550#581
 Basically, what's going on in there is that one of those sysctl() or 
sysctlbyname() calls is failing and the first thing to do is figure out which one.
The weird thing is that contrarily to what stated in the man pages errno is not set.

Status: Started
Labels: OpSys-FreeBSD Crash

[giampaolo]

From dstew...@gmail.com on January 28, 2013 05:12:55

Does psutil work fine with GENERIC kernel? This could be 
http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/172803

[giampaolo]

From g.rodola on January 28, 2013 05:29:15

Not sure what you mean by GENERIC kernel but we might try the patch proposed in 
that ticket.
Can someone who's able to reproduce this issue try this patch (save it into a 
file and apply with "patch -p0 < file.patch")?


Index: psutil/_psutil_bsd.c
===================================================================
--- psutil/_psutil_bsd.c    (revisione 1560)
+++ psutil/_psutil_bsd.c    (copia locale)
@@ -584,11 +584,16 @@
 static PyObject*
 get_virtual_mem(PyObject* self, PyObject* args)
 {
-    unsigned int   total, active, inactive, wired, cached, free, buffers;
+    unsigned int   total, active, inactive, wired, cached, free;
     size_t        size = sizeof(total);
    struct vmtotal vm;
    int            mib[] = {CTL_VM, VM_METER};
    long           pagesize = getpagesize();
+#if __FreeBSD_version > 702101
+    long buffers;
+#else
+    int buffers;
+#endif

     if (sysctlbyname("vm.stats.vm.v_page_count", &total, &size, NULL, 0))
         goto error;
@@ -602,7 +607,7 @@
         goto error;
     if (sysctlbyname("vm.stats.vm.v_free_count", &free, &size, NULL, 0))
         goto error;
-    if (sysctlbyname("vfs.bufspace", &buffers, &size, NULL, 0))
+    if (sysctlbyname("vfs.bufspace", &buffers, sizeof(buffers), NULL, 0))
         goto error;

     size = sizeof(vm);

[giampaolo]

From g.rodola on January 28, 2013 05:31:48

Sorry, there's a typo.
Here's the correct patch (not tested):


Index: psutil/_psutil_bsd.c
===================================================================
--- psutil/_psutil_bsd.c    (revisione 1560)
+++ psutil/_psutil_bsd.c    (copia locale)
@@ -584,11 +584,16 @@
 static PyObject*
 get_virtual_mem(PyObject* self, PyObject* args)
 {
-    unsigned int   total, active, inactive, wired, cached, free, buffers;
+    unsigned int   total, active, inactive, wired, cached, free;
     size_t        size = sizeof(total);
    struct vmtotal vm;
    int            mib[] = {CTL_VM, VM_METER};
    long           pagesize = getpagesize();
+#if __FreeBSD_version > 702101
+    long buffers;
+#else
+    int buffers;
+#endif
+size_t buffers_size = sizeof(buffers);

     if (sysctlbyname("vm.stats.vm.v_page_count", &total, &size, NULL, 0))
         goto error;
@@ -602,7 +607,7 @@
         goto error;
     if (sysctlbyname("vm.stats.vm.v_free_count", &free, &size, NULL, 0))
         goto error;
-    if (sysctlbyname("vfs.bufspace", &buffers, &size, NULL, 0))
+    if (sysctlbyname("vfs.bufspace", &buffers, &buffers_size, NULL, 0))
         goto error;

     size = sizeof(vm);

[giampaolo]

From sunrize...@gmail.com on February 28, 2013 01:21:51

Hi again, guys. I was unable to reproduce this issue today.

[sunrize@PRWC013 ~]$ uname -a
FreeBSD PRWC013.local 9.0-RELEASE-p3 FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 
02:52:29 UTC 2012     
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
[sunrize@PRWC013 ~]$ source psutils_test/bin/activate
(psutils_test)[sunrize@PRWC013 ~]$ pip install psutil
Downloading/unpacking psutil
  Downloading psutil-0.6.1.tar.gz (138kB): 138kB downloaded
  Running setup.py egg_info for package psutil

Installing collected packages: psutil
  Running setup.py install for psutil
    building '_psutil_bsd' extension
    cc -fno-strict-aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -O2 -pipe 
-fno-strict-aliasing -fPIC -I/usr/local/include/python2.7 -c 
psutil/_psutil_bsd.c -o 
build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_bsd.o
    cc -fno-strict-aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -O2 -pipe 
-fno-strict-aliasing -fPIC -I/usr/local/include/python2.7 -c 
psutil/_psutil_common.c -o 
build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_common.o
    cc -fno-strict-aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -O2 -pipe 
-fno-strict-aliasing -fPIC -I/usr/local/include/python2.7 -c 
psutil/arch/bsd/process_info.c -o 
build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/arch/bsd/process_info.o
    cc -shared -pthread 
build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_bsd.o 
build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_common.o 
build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/arch/bsd/process_info.o 
-ldevstat -o build/lib.freebsd-9.0-RELEASE-p3-amd64-2.7/_psutil_bsd.so
    building '_psutil_posix' extension
    cc -fno-strict-aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -O2 -pipe 
-fno-strict-aliasing -fPIC -I/usr/local/include/python2.7 -c 
psutil/_psutil_posix.c -o 
build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_posix.o
    cc -shared -pthread 
build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_posix.o -o 
build/lib.freebsd-9.0-RELEASE-p3-amd64-2.7/_psutil_posix.so

Successfully installed psutil
Cleaning up...
(psutils_test)[sunrize@PRWC013 ~]$ python
Python 2.7.3 (default, Oct 24 2012, 15:09:17) 
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.__version__
'0.6.1'
>>>

[giampaolo]

From sunrize...@gmail.com on February 28, 2013 01:26:42

Should i test this patch anyway?

[giampaolo]

From g.rodola on February 28, 2013 01:44:59

> I was unable to reproduce this issue today.

Damn. Is it the exact same system you were using in the first place? Is 
something changed? Point is if we're not able to reproduce the original issue 
there's no point in committing the patch.

[giampaolo]

From sunrize...@gmail.com on February 28, 2013 01:54:52

Yes, everything was the same: same os, same psutil and python version. But it 
was not the same machine. Ok, i'll try to figure out, whether that server is 
available now.

[giampaolo]

From g.rodola on March 05, 2013 13:58:41

This is now mitigated by revision d6cf2f8ba8c0 which makes psutil raise an 
exception later than sooner, so that at least we don't crash at import time.
Instead we'll get a TypeError on get_memory_percent().
Did you get the chance to put hands on that machine?

[giampaolo]

From utis...@gmail.com on April 13, 2013 04:55:50

I just experienced this, and so did another user.

I've committed the patch to the FreeBSD ports tree, because it works well. 
http://www.freebsd.org/cgi/query-pr.cgi?pr=172803 Chris

[giampaolo]

From g.rodola on April 13, 2013 05:29:04

Oh nice! Can you confirm the attached patch fixes the issue?

Attachment: 325.patch