weird introspection question, specifically with certain objects in cython files

[craigcitro]

There are two weird introspection issues that Jon Hanke pointed out, and I took the time to track down. In particular, there are issues with introspection on top-level def and cpdef methods in any .pyx file.

Here are two examples of what can go on:

sage: sage.rings.polynomial.polynomial_element.make_generic_polynomial??
Type:             builtin_function_or_method
Base Class:       <type 'builtin_function_or_method'>
String Form:   <built-in function make_generic_polynomial>
Namespace:        Interactive
Source:
def make_generic_polynomial(parent, coeffs):
    return parent(coeffs)

The problem here is that the source file that this function comes from does not appear anywhere, even though it should.

However, things can get worse:

Type:           builtin_function_or_method
Base Class:     <type 'builtin_function_or_method'>
String Form:    <built-in function is_Polynomial>
Namespace:      Interactive
Source:
def is_Polynomial(f):
    """
    Return True if f is of type univariate polynomial.

    INPUT:
        f -- an object

    EXAMPLES:
        sage: from sage.rings.polynomial.polynomial_element import is_Polynomial
        sage: R.<x> = ZZ[]
        sage: is_Polynomial(x^3 + x + 1)
        True
        sage: S.<y> = R[]
        sage: f = y^3 + x*y -3*x; f
        y^3 + x*y - 3*x
        sage: is_Polynomial(f)
        True

    However this function does not return True for genuine multivariate
    polynomial type objects or symbolic polynomials, since those are not
    of the same data type as univariate polynomials:
        sage: R.<x,y> = QQ[]
        sage: f = y^3 + x*y -3*x; f
        y^3 + x*y - 3*x
        sage: is_Polynomial(f)
        False
        sage: var('x,y')
        (x, y)
        sage: f = y^3 + x*y -3*x; f
        y^3 + x*y - 3*x
        sage: is_Polynomial(f)
        False
    """
    return PY_TYPE_CHECK(f, Polynomial)Class Docstring:
    <attribute '__doc__' of 'builtin_function_or_method' objects>

I have several issues with this:

• the info in the top block is wrong
• no source file appears
• both the formatting and placement of the Class Docstring bit at the end are weird.

Now, it's actually much worse than just that -- is_Polynomial is (currently) imported into the global namespace, and the entry in the global namespace is pretty messed up:

sage: is_Polynomial
<functools.partial object at 0x7ca3b40>

sage: is_Polynomial??
Error getting source: could not find class definition
Type:             partial
Base Class:       <type 'functools.partial'>
String Form:   <functools.partial object at 0x7ca3b40>
Namespace:        Interactive
File:             /Users/craigcitro/three-two/local/lib/python/functools.py
Docstring [source file open failed]:
    partial(func, *args, **keywords) - new function with partial application
        of the given arguments and keywords.

It seems to think that is_Polynomial is a completely different type, in fact a Python builtin!!

So I've only dug around a bit, but here's what I've found. First, this looks bad:

sage: sage.rings.polynomial.polynomial_element.is_Polynomial
<built-in function is_Polynomial>

sage: sage.rings.polynomial.all.is_Polynomial
<built-in function is_Polynomial>

sage: sage.rings.all.is_Polynomial
<built-in function is_Polynomial>

sage: sage.all.is_Polynomial
<functools.partial object at 0x7ca3b40>

sage: sage.all.is_Polynomial == sage.rings.all.is_Polynomial
False

This could mean that in the process of doing imports after sage.rings.all, some module imports is_Polynomial and breaks something.

Interestingly, the docstring printed above is not the docstring that the module has for is_Polynomial:

sage: sage.rings.all.__dict__['is_Polynomial'].__doc__
"File: sage/rings/polynomial/polynomial_element.pyx (starting at line 72)\n\n    Return True if f is of type univariate polynomial.\n\n    INPUT:\n        f -- an object\n\n    EXAMPLES:\n        sage: from sage.rings.polynomial.polynomial_element import is_Polynomial\n        sage: R.<x> = ZZ[]\n        sage: is_Polynomial(x^3 + x + 1)\n        True\n        sage: S.<y> = R[]\n        sage: f = y^3 + x*y -3*x; f\n        y^3 + x*y - 3*x\n        sage: is_Polynomial(f)\n        True\n\n    However this function does not return True for genuine multivariate\n    polynomial type objects or symbolic polynomials, since those are not\n    of the same data type as univariate polynomials:\n        sage: R.<x,y> = QQ[]\n        sage: f = y^3 + x*y -3*x; f\n        y^3 + x*y - 3*x\n        sage: is_Polynomial(f)\n        False\n        sage: var('x,y')\n        (x, y)\n        sage: f = y^3 + x*y -3*x; f\n        y^3 + x*y - 3*x\n        sage: is_Polynomial(f)\n        False\n    "

I'm starting to believe that this is two separate isssues: one for imports, and one for the docstring getting mis-parsed during introspection. I'm adding Rob Bradshaw to the first, because I think he'll know right offhand if anything funny is going on with the way the polynomial_element module gets initialized in the Cython code, and Nick Alexander, because I'm sure he'll be able to quickly point me in the right direction on the introspection stuff.

Please comment if you find anything -- I'll happily try and fix if someone points me in the right direction.

CC: @ncalexan @robertwb @wjp

Component: user interface

Author: Willem Jan Palenstijn

Issue created by migration from https://trac.sagemath.org/ticket/4489

[williamstein]

comment:2

REMARK: This is an issue only in Ipython. In the notebook the output looks, e.g., like this for prod??:

File: /Users/was/s/devel/sage/sage/misc/misc_c.pyx
Source Code (starting at line 54):
def prod(x, z=None, Py_ssize_t recursion_cutoff = 5):

[wjp]

comment:4

Part of the mentioned issues have been fixed as part of #3342, I think.

The remaining problem, that the sage command line doesn't show cython source files is what the patch I'm attaching should take care of.

It patches sageinspect.py, so I'm attaching patches for both sage and sagenb, to keep the files in sync as much as possible.

The approach I'm taking is monkey-patching inspect.getabsfile, replacing it by sage_getfile. The IPython code getting the file is deeply embedded in a long function, and calls inspect.getabsfile directly, so there seems to be no more elegant way of overriding the behaviour.

[wjp]

Attachment: 4489_ipython_getabsfile.patch.gz

[wjp]

comment:5

Attachment: sagenb_4489_ipython_getabsfile.patch.gz

[wjp]

Author: Willem Jan Palenstijn

[jhpalmieri]

comment:7

Are the issues here solved by the patches at #11298?

[wjp]

comment:8

No, this patch solves a separate issue.

[tkluck]

comment:9

As far as I can tell, this bug has been resolved in the mean time. In my limited testing, the source code for cython functions are correctly loaded. Also, this patch doesn't apply anymore.

I would close this ticket, but I haven't got the right permissions. I'll set it to needs_info instead, so that it is clear that it currently does not need review.