inspect.getmembers passes exceptions from object's properties through

[rominf]

BPO	35108
Nosy	@1st1, @tirkarthi, @RichardBruskiewich
PRs	bpo-35108: handle exceptions in inspect.getmembers #18330

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-10-30.08:20:35.443>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = "inspect.getmembers passes exceptions from object's properties through"
updated_at = <Date 2020-02-03.15:21:31.209>
user = 'https://bugs.python.org/rominf'

bugs.python.org fields:

activity = <Date 2020-02-03.15:21:31.209>
actor = 'tomaugspurger'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-10-30.08:20:35.443>
creator = 'rominf'
dependencies = []
files = []
hgrepos = []
issue_num = 35108
keywords = ['patch']
message_count = 5.0
messages = ['328885', '328890', '328891', '329254', '361208']
nosy_count = 4.0
nosy_names = ['yselivanov', 'rominf', 'xtreak', 'richardbruskiewich']
pr_nums = ['18330']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35108'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[rominf]

I use inspect.getmembers for getting members of splinter.webdriver.BaseWebDriver. The problem is that it has a property status_code raises NotImplementedError: https://github.com/cobrateam/splinter/blob/master/splinter/driver/webdriver/__init__.py#L191. This exception passes through try block in https://github.com/python/cpython/blob/master/Lib/inspect.py#L343 because it doesn't contain:
except Exception:
pass
section.
In the result, instead of members of the object, I get an exception.

I think there are two possible expected behaviors:

1. Just ignore the exceptions and return members without the members, that raise exceptions.
2. Instead of members, return tuple like:
   ...
   try:
   ...
   except Exception as e:
   RAISES_EXCEPTION = object()
   value = (RAISES_EXCEPTION, e)
   ...

[tirkarthi]

Thanks for the report. A simple reproducible in Python without splinter code. I think it happens for instances of the class where the attribute is a property raising exception and executing inspect.getmembers(Foo) works fine.

# bpo-35108.py

import inspect

class Foo():

    @property
    def bar(self):
        raise NotImplementedError

print(inspect.getmembers(Foo()))

$. /python.exe ../backups/bpo35108.py
Traceback (most recent call last):
  File "../backups/bpo35108.py", line 9, in <module>
    print(inspect.getmembers(Foo()))
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/inspect.py", line 344, in getmembers
    value = getattr(object, key)
  File "../backups/bpo35108.py", line 7, in bar
    raise NotImplementedError
NotImplementedError

[tirkarthi]

Sorry I misread your title stating it's about object getattr for a property has to execute the underlying code it seems as explained in https://stackoverflow.com/a/30143990/2610955 . I don't know if there is a way through which an attribute access in case of property can be verified without executing it. In case of catching the exception and returning the attributes that don't raise exception since it depends on the runtime execution of the property where the exception might occur at one point in time and doesn't occur in another point of time causing confusion during debugging.

[tirkarthi]

There are some relevant docs in the inspect module about properties triggering code execution and thus using getattr_static to fetch attributes. Ref : https://docs.python.org/3/library/inspect.html#fetching-attributes-statically . I tried the below and it passes for your code but there might be still cases where getattr_static triggers an exception too where we need to decide whether to skip the attribute from being listed and if so with a test for the scenario.

Using getattr_static first causes test_inspect.TestPredicates.test_get_slot_members to fail by including slot_member. Thus we try for getattr and if it raises an exception that is not AttributeError (@Property might also raise AttributeError consciously at runtime) and then use getattr_static in case of exception other than AttributeError. This helps in passing the test suite and also the example attached listing bar which raises NotImplementedError or other Exceptions.

diff --git a/Lib/inspect.py b/Lib/inspect.py
index b8a142232b..9df2173e0c 100644
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -341,7 +341,12 @@ def getmembers(object, predicate=None):
         # like calling their __get__ (see bug python/cpython#46118), so fall back to
         # looking in the __dict__.
         try:
-            value = getattr(object, key)
+            try:
+                value = getattr(object, key)
+            except Exception as e:
+                if isinstance(e, AttributeError):
+                    raise e
+                value = getattr_static(object, key)
             # handle the duplicate key
             if key in processed:
                 raise AttributeError

I am adding Yury. Removing 3.5 and 3.4 since they are in security fixes only mode.

Also see bpo-30533 a proposal for implementation of getmembers that uses getattr_static instead of getattr.

Thanks

[RichardBruskiewich]

This "bug" is buzzing around my project head right now, interfering with the operation of the Python Fire CLI library when it attempts to interrogate the Python Pandas DataFrame using the inspect.getmembers() call. See pandas-dev/pandas#31474 and pandas-dev/pandas#31549.

I have code that uses Fire and Pandas, but have to "dumb it down" to use Pandas 0.24.3 rather than the latest 0.25.1 which raises a "NotImplementedError" which leaks out of the getmembers() call. The Pandas people are passing the buck to you folks in the Python community.

This is terribly frustrating for we minions in the real work trying to implementing real working software systems leveraging all these wonderful libraries (and the Python language).

When is this "bug" going to be fixed? Help!

[iritkatriel]

@tirkarthi Perhaps NotImplementedError can be handled as a special case? It indicates that the property is not really there, so it should be ok to omit it from the output.