Cython support

[darthlukan]

Feature Request:

I'm working on a Python3.5 project which uses Cython v0.23.4 for several modules via the pyximport method. We're using pytest 3.1.3 and pytest-cov 2.5.1. Currently pytest-cov only reports on .py files. Our tests are written with the convention of test_.py and within those, calls to "def" functions and methods within *.pyx files are made in order to test the output (inside of our *.pyx modules we have a "def" signature for every "cdef" implementation so that non-cython modules in our application can easily call cythonized code in pure python).

Example:

cdef int _return_pos_int():
    return 1

def return_pos_int():
    return _return_pos_int()

If we were testing the example above, our tests would call the function with the "def return_pos_int()" signature.

The current pytest-cov output ignores covered .pyx files, even if there are .py tests accessing those modules. Ideally, pytest-cov would be able to determine the coverage for cython .pyx modules as it does with normal .py modules.

[RonnyPfannschmidt]

i believe this is sorted out via http://cython.readthedocs.io/en/latest/src/tutorial/profiling_tutorial.html

[darthlukan]

Hi @RonnyPfannschmidt - I checked that doc and tried adding the suggested plugins entry in the run section of .coveragerc, it had no effect on the reported output.

For reference, here is my .coveragerc file content:

[report]
show_missing = true
omit = *test_*

[run]
plugins = Cython.Coverage

The content of my run_tests.sh:

#!/bin/sh

PYTHONPATH=:$(pwd) pytest \
    --cov=/home/darthlukan/projects/lms_api \
    --cov-report=term \
    --cov-config=$(pwd)/.coveragerc

And here's the resulting report output:

============================================================================================================= test session starts =============================================================================================================
platform linux -- Python 3.5.2, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/darthlukan/projects/lms_api, inifile:
plugins: cov-2.5.1
collected 8 items 

route_handlers/test_users.py ...
tools/test_flsutils.py ....
tools/test_lciutils.py .

----------- coverage: platform linux, python 3.5.2-final-0 -----------
Name                          Stmts   Miss  Cover   Missing
-----------------------------------------------------------
__init__.py                       0      0   100%
api_server.py                   190    190     0%   1-384
constants.py                     13      0   100%
route_handlers/__init__.py        0      0   100%
route_handlers/companies.py      85     85     0%   1-143
route_handlers/results.py       168    168     0%   1-321
tools/__init__.py                 0      0   100%
tools/initdb.py                 121    121     0%   1-275
tools/lciutils.py                57     11    81%   20, 47-49, 108-110, 121, 134-136
tools/logger.py                  46     26    43%   28-29, 33-36, 40-43, 47-50, 54-57, 61-64, 68, 74-77
-----------------------------------------------------------
TOTAL                           680    601    12%


========================================================================================================== 8 passed in 0.32 seconds ===========================================================================================================

Within the route_handlers directory we have users.pyx, uids.pyx, and content.pyx, and in tools we have flsutils.pyx. Those files still do not appear in the output as you can see.

Don't mind the horrible coverage rating on the files that are picked up, we're working on it >.>

[RonnyPfannschmidt]

i see, thanks for the feedback - personally i cant do much more on this - @nedbat and @ionelmc are more familiar with the details

[ionelmc]

I'm trying this now, these changes seem to make it work.

Two things to consider:

• setting the CYTHON_TRACE=1 macro ain't enough, you also need to litter all your pyx files with # cython: linetrace=True pragmas.
• you need to compile inplace (eg: python setup.py clean --all build_ext --force --inplace if your using the cython support in setuptools)

@darthlukan Your coverage report looks weird. Why do you have a bare __init__.py there? Where are the .so files being built/installed?

Also, try removing source filters (just use a bare --cov till you see hits on pyx files).

[darthlukan]

@ionelmc - I'll take a look at your changes. Thanks!

I'm not using setuptools, instead I'm using the pyximport method of compilation on init, so there are no .so, .pdi, or .c files in my project/application. Just .py and .pyx.

We don't typically add code to __init__.py in our projects, it's there only for imports as sub directories are considered "regular packages" under the parent, not namespace packages, for our purposes. Why __init__.py files appear is probably because I didn't filter them out via settings (I was more concerned with the .pyx recognition issue).

I'll try using the more traditional approach of compiling prior to test running in our test execution script (disabling pyximport), coupled with your pragma suggestions and see if that does it. Thanks for all of the feedback!

[ionelmc]

@darthlukan where are the .so files located then? if in the same place as the .pyx files then probably you're only missing the macro definitions, eg:

from os import environ
environ['CFLAGS'] = '-DCYTHON_TRACE=1'
import pyximport
pyximport.install()

[webknjaz]

Does this still need to remain open? I successfully implemented reporting coverage for Cython PYX files in at least 2 projects: https://github.com/ansible/pylibssh and https://github.com/aio-libs/yarl.

My gotchas are:

• plugins = Cython.Coverage;
• linetrace=true should be set (it implies profile=true) when converting from PYX to C — this will generate corresponding ifdef-guarded blocks of tracing code; when using alternative (non-setuptools based) conversion methods, it's possible to call the cython module on CLI and pass those directives via -X; otherwise, yes — add code comments to each file. This is also used by the Cython.Coverage plugin which uses the C files with tracing markers to map the coverage back to the PYX files for the report;
• -DCYTHON_TRACE=1 in CFLAGS so that GCC compiling from C to a .so — this will only have effect if the C files contain the tracing code from the previous step.

In one project, I do this fancy CI thing where I compile a matrix of wheels in separate jobs, based on the contents of an sdist built earlier (instead of a Git checkout). Then, I test those exact wheels. But the sdist doesn't contain the C-files with tracing, so when I use it as a base to run tests against those wheels, I need to have the C-files with tracing, which is why I currently generate just them in the source dir (without compiling to .so) and pip install *.whl (without -e). Like this:

$ python -Im cython -X linetrace=true -X embedsignature=true -X emit_code_comments=true yarl/*.pyx

The gotcha is to remember using the same directives as for the original builds (especially the linetrace one).

Works like a charm!