pytest 3.0 breaks Jenkins' "Publish JUnit test result report" for doctests

[icemac]

Upgrading from pytest 2.9.2 to 3.0.x breaks the build step "Publish JUnit test result report" in Jenkins if a doctest is involved in the tests.

Jenins Error message:

ERROR: Build step failed with exception
.../PYTHON/Python2.7/README.txt is not a directory.
    at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:488)
    at hudson.util.DirScanner$Glob.scan(DirScanner.java:128)
    at hudson.FilePath$42.invoke(FilePath.java:2162)
    at hudson.FilePath$42.invoke(FilePath.java:2155)
    at hudson.FilePath.act(FilePath.java:1018)
    at hudson.FilePath.act(FilePath.java:996)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2155)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2141)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2124)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2109)
    at hudson.plugins.junitattachments.GetTestDataMethodObject.attachFilesForReport(GetTestDataMethodObject.java:121)
    at hudson.plugins.junitattachments.GetTestDataMethodObject.getAttachments(GetTestDataMethodObject.java:110)
    at hudson.plugins.junitattachments.AttachmentPublisher.contributeTestData(AttachmentPublisher.java:54)
    at hudson.plugins.junitattachments.AttachmentPublisher.contributeTestData(AttachmentPublisher.java:30)
    at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:183)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720)
    at hudson.model.Build$BuildExecution.post2(Build.java:185)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:665)
    at hudson.model.Run.execute(Run.java:1745)
    at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:410)
Build step 'Veröffentliche JUnit-Testergebnisse.' marked build as failure

Jenkins version: 2.11
JUnit Plugin version: 1.18
py: 1.4.31
pluggy: 0.3.1
plugins: flake8-0.6, cov-2.3.1, remove-stale-bytecode-2.1
Python versions: 2.7 up to 3.5 and PyPy

Contents of pytest.ini:

[pytest]
addopts = src README.txt  --flake8

The generated junit.xml file differs in the following way:

pytest 2.9.2:

<testcase classname="" file="README.txt" name="README.txt" time="0.027095079422"></testcase>

pytest 3.0.x:

<testcase classname="README.txt" file="README.txt" name="README.txt" time="0.00580596923828"></testcase>

Aka the classname is no longer empty.

[nicoddemus]

Thanks for the detailed report!

Can you confirm that this is only a problem for doctests and report for normal Python tests work as expected?

[icemac]

@nicoddemus This seems to be only a problem for doctests. Actually only doctest files seem to have this problem. In an other project using --doctest-modules still works fine.
Normal Python unitests work as expected.

[icemac]

@nicoddemus Looking at other of our projects I found out that the problem seems only to occur if the doctest file is outside the package source directory or at the package root.

Something like this is okay for Jenkins:

<testcase classname="src.gocept.collmex.doctest.txt" file="src/gocept/collmex/doctest.txt" name="doctest.txt" time="11.561240196228027">

[nicoddemus]

Bisected to 74862b8.

[nicoddemus]

I have investigated this further and this is what I found:

Files

For sake of discussion, consider these files in the current directory:

README.txt:

Hello!

>>> 1 + 1
2

test_module.py:

def test_1(): pass

2.9.2

Text files are collected as a weird "module/item" hybrid (can be seen by in 74862b8) that acts both as a collector and an item:

> README.txt  (module and item)

Which explains the output:

$ py.test -v
...
collected 2 items

README.txt PASSED
test_module.py::test_1 PASSED

Note that README.txt is both the test container (collector) and test item itself, while for normal python tests we have the container (test_module.py) and the test item (test_1).

Here's the XML file produced:

<testcase classname="" file="README.txt" name="tmp.README.txt" time="0.02702927589416504"></testcase>
<testcase classname="tmp.test_module" file="test_module.py" line="0" name="test_1" time="0.0"></testcase>

classname here is empty because it doesn't have a parent, but that's an accident of the doctest implementation and has nothing to do with pytest's junitxml plugin.

3.0

We have changed this so now the text file is properly collected as a module, which can have items or not depending on its contents:

> README.txt (module)
  > README.txt (item) (always 1 item for text files with doctests, no item at all if there is no doctests in the text file)

Here's the output from pytest to illustrate this:

$ py.test -v
...
collected 2 items

README.txt::README.txt PASSED
test_module.py::test_1 PASSED

(Ignore the fact that maybe README.txt::README.txt should be README.txt::DocTest).

Here's the XML file:

<testcase classname="tmp.README.txt" file="README.txt" name="README.txt" time="0.0"></testcase>
<testcase classname="tmp.test_module" file="test_module.py" line="0" name="test_1" time="0.0"></testcase>

When you look at classname="tmp.test_module", it makes you wonder why that doesn't generate any kind of error in Jenkins because tmp.test_module does not exist, only tmp.test_module.py and I doubt Jenkins knows about or treats Python specially. However it breaks because tmp.README.txt is an existing file but not a directory.

Also the fact that this works without a problem on Jenkins:

<testcase classname="src.gocept.collmex.doctest.txt" file="src/gocept/collmex/doctest.txt" name="doctest.txt" time="0.0">

Seems very strange to me and I'm inclined to think this is actually a bug in the JUnitXML Jenkins plugin. Perhaps this should be posted on Jenkin's JUnit bug tracker and see what they have to say, specially why this breaks:

<testcase classname="tmp.README.txt" file="README.txt" name="README.txt" time="0.0"></testcase>

while this is considered valid:

<testcase classname="src.gocept.collmex.doctest.txt" file="src/gocept/collmex/doctest.txt" name="doctest.txt" time="0.0">

I'm afraid that just hacking pytest at this point without understanding why Jenkins is barfing at what's being produced might be problematic in the future.

@icemac, could you please post this issue on the Jenkins tracker and see what they have to say?

[icemac]

@nicoddemus I added JENKINS-37764 to the Jenkins issue tracker.

[nicoddemus]

Thanks! I'm watching the issue myself.

[icemac]

@nicoddemus There is silence in the Jenkins issue tracker. :( But the issue still persists.

[nicoddemus]

Thanks @icemac, I noticed your comment on their issue tracker.

I'm not against modifying the output for doctests if reasonable.

[sallner]

This problem still persists. A possible workaround is the manual conversion of the doctests to unittests with the help of DocFileSuite.

import doctest

def test_suite():
    """Test doctest as unittest."""
    # This should help with https://github.com/pytest-dev/pytest/issues/1862
    return doctest.DocFileSuite('README.rst')

This results in an entry with classname being a longer dotted string.