info / fixes about xUnit plugin

- see #209; add a utility function for XSLT transformation. - minor doc fixes. - fix incorrect version, I called out xunit plugin 2.2.4 earlier and I must not have looked at the version properly.
xmlrunner · Feb 17, 2020 · 3377a08 · 3377a08
1 parent f58d7b7
commit 3377a08
Show file tree

Hide file tree

Showing 4 changed files with 245 additions and 29 deletions.
diff --git a/README.md b/README.md
@@ -14,35 +14,82 @@ A unittest test runner that can save test results to XML files in xUnit format.
 The files can be consumed by a wide range of tools, such as build systems, IDEs
 and continuous integration servers.
 
-## Schema
 
-There are many schemas with minor differences.
-We use one that is compatible with Jenkins xUnit plugin, a copy is
-available under `tests/vendor/jenkins/xunit-plugin/junit-10.xsd` (see attached license).
+## Requirements
+
+* Python 3.5+
+* Please note Python 2.7 end-of-life was in Jan 2020, last version supporting 2.7 was 2.5.2
+* Please note Python 3.4 end-of-life was in Mar 2019, last version supporting 3.4 was 2.5.2
+* Please note Python 2.6 end-of-life was in Oct 2013, last version supporting 2.6 was 1.14.0
+
+
+## Limited support for `unittest.TestCase.subTest`
+
+https://docs.python.org/3/library/unittest.html#unittest.TestCase.subTest
+
+`unittest` has the concept of sub-tests for a `unittest.TestCase`; this doesn't map well to an existing xUnit concept, so you won't find it in the schema. What that means, is that you lose some granularity
+in the reports for sub-tests.
+
+`unittest` also does not report successful sub-tests, so the accounting won't be exact.
 
-- [Jenkins (junit-10.xsd), xunit plugin (2014-2018)](https://github.com/jenkinsci/xunit-plugin/blob/14c6e39c38408b9ed6280361484a13c6f5becca7/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd), please note the latest versions (2.2.4 and above are not backwards compatible)
+## Jenkins plugins
+
+- Jenkins JUnit plugin : https://plugins.jenkins.io/junit/
+- Jenkins xUnit plugin : https://plugins.jenkins.io/xunit/
+
+### Jenkins JUnit plugin
+
+This plugin does not perform XSD validation (at time of writing) and should parse the XML file without issues.
+
+### Jenkins xUnit plugin version 1.100
+
+- [Jenkins (junit-10.xsd), xunit plugin (2014-2018)](https://github.com/jenkinsci/xunit-plugin/blob/14c6e39c38408b9ed6280361484a13c6f5becca7/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd), version `1.100`.
+
+This plugin does perfom XSD validation and uses the more lax XSD. This should parse the XML file without issues.
+
+### Jenkins xUnit plugin version 1.104+
+
+- [Jenkins (junit-10.xsd), xunit plugin (2018-current)](https://github.com/jenkinsci/xunit-plugin/blob/ae25da5089d4f94ac6c4669bf736e4d416cc4665/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd), version `1.104`+.
+
+This plugin does perfom XSD validation and uses the more strict XSD.
+
+See https://github.com/xmlrunner/unittest-xml-reporting/issues/209
+
+```
+import io
+import unittest
+import xmlrunner
+
+# run the tests storing results in memory
+out = io.BytesIO()
+unittest.main(
+    testRunner=xmlrunner.XMLTestRunner(output=out),
+    failfast=False, buffer=False, catchbreak=False, exit=False)
+```
+
+Transform the results removing extra attributes.
+```
+from xmlrunner.extra.xunit_plugin import transform
+
+with open('TEST-report.xml', 'wb') as report:
+    report.write(transform(out.getvalue()))
+
+```
+
+## JUnit Schema ?
+
+There are many tools claiming to write JUnit reports, so you will find many schemas with minor differences.
+
+We used the XSD that was available in the Jenkins xUnit plugin version `1.100`; a copy is available under `tests/vendor/jenkins/xunit-plugin/.../junit-10.xsd` (see attached license).
 
 You may also find these resources useful:
 
 - https://stackoverflow.com/questions/4922867/what-is-the-junit-xml-format-specification-that-hudson-supports
 - https://stackoverflow.com/questions/11241781/python-unittests-in-jenkins
-- [Jenkins (junit-10.xsd), xunit plugin 2.2.4+](https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd)
 - [JUnit-Schema (JUnit.xsd)](https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd)
 - [Windyroad (JUnit.xsd)](http://windyroad.com.au/dl/Open%20Source/JUnit.xsd)
 - [a gist (Jenkins xUnit test result schema)](https://gist.github.com/erikd/4192748)
 
-## Things that are somewhat broken
-
-Python 3 has the concept of sub-tests for a `unittest.TestCase`; this doesn't map well to an existing
-xUnit concept, so you won't find it in the schema. What that means, is that you lose some granularity
-in the reports for sub-tests.
-
-## Requirements
-
-* Python 3.5+
-* Please note Python 2.7 end-of-life was in Jan 2020, last version supporting 2.7 was 2.5.2
-* Please note Python 3.4 end-of-life was in Mar 2019, last version supporting 3.4 was 2.5.2
-* Please note Python 2.6 end-of-life was in Oct 2013, last version supporting 2.6 was 1.14.0
 
 ## Installation
 

diff --git a/tests/testsuite.py b/tests/testsuite.py
@@ -27,23 +27,21 @@
 from unittest import mock
 
 
-def _load_schema():
-    path = os.path.join(os.path.dirname(__file__),
-                        'vendor/jenkins/xunit-plugin',
-                        'junit-10.xsd')
+def _load_schema(version):
+    path = os.path.join(
+        os.path.dirname(__file__),
+        'vendor/jenkins/xunit-plugin', version, 'junit-10.xsd')
     with open(path, 'r') as schema_file:
         schema_doc = etree.parse(schema_file)
         schema = etree.XMLSchema(schema_doc)
         return schema
     raise RuntimeError('Could not load JUnit schema')  # pragma: no cover
 
 
-JUnitSchema = _load_schema()
-
-
-def validate_junit_report(text):
+def validate_junit_report(version, text):
     document = etree.parse(BytesIO(text))
-    JUnitSchema.assertValid(document)
+    schema = _load_schema(version)
+    schema.assertValid(document)
 
 
 class TestCaseSubclassWithNoSuper(unittest.TestCase):
@@ -650,7 +648,7 @@ def test_junitxml_xsd_validation_order(self):
         self.assertTrue(i_properties < i_testcase <
                         i_system_out < i_system_err)
         # XSD validation - for good measure.
-        validate_junit_report(output)
+        validate_junit_report('14c6e39c38408b9ed6280361484a13c6f5becca7', output)
 
     def test_junitxml_xsd_validation_empty_properties(self):
         suite = unittest.TestSuite()
@@ -665,7 +663,31 @@ def test_junitxml_xsd_validation_empty_properties(self):
         outdir.seek(0)
         output = outdir.read()
         self.assertNotIn('<properties>'.encode('utf8'), output)
-        validate_junit_report(output)
+        validate_junit_report('14c6e39c38408b9ed6280361484a13c6f5becca7', output)
+
+    def test_xunit_plugin_transform(self):
+        suite = unittest.TestSuite()
+        suite.addTest(self.DummyTest('test_fail'))
+        suite.addTest(self.DummyTest('test_pass'))
+        suite.properties = None
+        outdir = BytesIO()
+        runner = xmlrunner.XMLTestRunner(
+            stream=self.stream, output=outdir, verbosity=self.verbosity,
+            **self.runner_kwargs)
+        runner.run(suite)
+        outdir.seek(0)
+        output = outdir.read()
+
+        validate_junit_report('14c6e39c38408b9ed6280361484a13c6f5becca7', output)
+        with self.assertRaises(etree.DocumentInvalid):
+            validate_junit_report('ae25da5089d4f94ac6c4669bf736e4d416cc4665', output)
+
+        from xmlrunner.extra.xunit_plugin import transform
+        transformed = transform(output)
+        validate_junit_report('14c6e39c38408b9ed6280361484a13c6f5becca7', transformed)
+        validate_junit_report('ae25da5089d4f94ac6c4669bf736e4d416cc4665', transformed)
+        self.assertIn('test_pass'.encode('utf8'), transformed)
+        self.assertIn('test_fail'.encode('utf8'), transformed)
 
     def test_xmlrunner_elapsed_times(self):
         self.runner_kwargs['elapsed_times'] = False

diff --git a/.../vendor/jenkins/xunit-plugin/junit-10.xsd → ...8b9ed6280361484a13c6f5becca7/junit-10.xsd b/.../vendor/jenkins/xunit-plugin/junit-10.xsd → ...8b9ed6280361484a13c6f5becca7/junit-10.xsd
diff --git a/tests/vendor/jenkins/xunit-plugin/ae25da5089d4f94ac6c4669bf736e4d416cc4665/junit-10.xsd b/tests/vendor/jenkins/xunit-plugin/ae25da5089d4f94ac6c4669bf736e4d416cc4665/junit-10.xsd
@@ -0,0 +1,147 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+The MIT License (MIT)
+
+Copyright (c) 2014, Gregory Boissinot
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
+    <xs:simpleType name="SUREFIRE_TIME">
+        <xs:restriction base="xs:string">
+            <xs:pattern value="(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?"/>
+        </xs:restriction>
+    </xs:simpleType>
+
+    <xs:complexType name="rerunType" mixed="true"> <!-- mixed (XML contains text) to be compatible with version previous than 2.22.1 -->
+        <xs:sequence>
+            <xs:element name="stackTrace" type="xs:string" minOccurs="0" /> <!-- optional to be compatible with version previous than 2.22.1 -->
+            <xs:element name="system-out" type="xs:string" minOccurs="0" />
+            <xs:element name="system-err" type="xs:string" minOccurs="0" />
+        </xs:sequence>
+        <xs:attribute name="message" type="xs:string" />
+        <xs:attribute name="type" type="xs:string" use="required" />
+    </xs:complexType>
+
+    <xs:element name="failure">
+        <xs:complexType mixed="true">
+            <xs:attribute name="type" type="xs:string"/>
+            <xs:attribute name="message" type="xs:string"/>
+        </xs:complexType>
+    </xs:element>
+
+    <xs:element name="error">
+        <xs:complexType mixed="true">
+            <xs:attribute name="type" type="xs:string"/>
+            <xs:attribute name="message" type="xs:string"/>
+        </xs:complexType>
+    </xs:element>
+
+    <xs:element name="skipped">
+        <xs:complexType mixed="true">
+            <xs:attribute name="type" type="xs:string"/>
+            <xs:attribute name="message" type="xs:string"/>
+        </xs:complexType>
+    </xs:element>
+
+    <xs:element name="properties">
+        <xs:complexType>
+            <xs:sequence>
+                <xs:element ref="property" minOccurs="0" maxOccurs="unbounded"/>
+            </xs:sequence>
+        </xs:complexType>
+    </xs:element>
+
+    <xs:element name="property">
+        <xs:complexType>
+            <xs:attribute name="name" type="xs:string" use="required"/>
+            <xs:attribute name="value" type="xs:string" use="required"/>
+        </xs:complexType>
+    </xs:element>
+
+    <xs:element name="system-err" type="xs:string"/>
+    <xs:element name="system-out" type="xs:string"/>
+    <xs:element name="rerunFailure" type="rerunType"/>
+    <xs:element name="rerunError" type="rerunType"/>
+    <xs:element name="flakyFailure" type="rerunType"/>
+    <xs:element name="flakyError" type="rerunType"/>
+
+    <xs:element name="testcase">
+        <xs:complexType>
+            <xs:sequence>
+                <xs:choice minOccurs="0" maxOccurs="unbounded">
+                    <xs:element ref="skipped"/>
+                    <xs:element ref="error"/>
+                    <xs:element ref="failure"/>
+                    <xs:element ref="rerunFailure" minOccurs="0" maxOccurs="unbounded"/>
+                    <xs:element ref="rerunError" minOccurs="0" maxOccurs="unbounded"/>
+                    <xs:element ref="flakyFailure" minOccurs="0" maxOccurs="unbounded"/>
+                    <xs:element ref="flakyError" minOccurs="0" maxOccurs="unbounded"/>
+                    <xs:element ref="system-out"/>
+                    <xs:element ref="system-err"/>
+                </xs:choice>
+            </xs:sequence>
+            <xs:attribute name="name" type="xs:string" use="required"/>
+            <xs:attribute name="time" type="xs:string"/>
+            <xs:attribute name="classname" type="xs:string"/>
+            <xs:attribute name="group" type="xs:string"/>
+        </xs:complexType>
+    </xs:element>
+
+    <xs:element name="testsuite">
+        <xs:complexType>
+            <xs:choice minOccurs="0" maxOccurs="unbounded">
+                <xs:element ref="testsuite"/>
+                <xs:element ref="properties"/>
+                <xs:element ref="testcase"/>
+                <xs:element ref="system-out"/>
+                <xs:element ref="system-err"/>
+            </xs:choice>
+            <xs:attribute name="name" type="xs:string" use="required"/>
+            <xs:attribute name="tests" type="xs:string" use="required"/>
+            <xs:attribute name="failures" type="xs:string" use="required"/>
+            <xs:attribute name="errors" type="xs:string" use="required"/>
+            <xs:attribute name="group" type="xs:string" />
+            <xs:attribute name="time" type="SUREFIRE_TIME"/>
+            <xs:attribute name="skipped" type="xs:string" />
+            <xs:attribute name="timestamp" type="xs:string" />
+            <xs:attribute name="hostname" type="xs:string" />
+            <xs:attribute name="id" type="xs:string" />
+            <xs:attribute name="package" type="xs:string" />
+            <xs:attribute name="file" type="xs:string"/>
+            <xs:attribute name="log" type="xs:string"/>
+            <xs:attribute name="url" type="xs:string"/>
+            <xs:attribute name="version" type="xs:string"/>
+        </xs:complexType>
+    </xs:element>
+
+    <xs:element name="testsuites">
+        <xs:complexType>
+            <xs:sequence>
+                <xs:element ref="testsuite" minOccurs="0" maxOccurs="unbounded" />
+            </xs:sequence>
+            <xs:attribute name="name" type="xs:string" />
+            <xs:attribute name="time" type="SUREFIRE_TIME"/>
+            <xs:attribute name="tests" type="xs:string" />
+            <xs:attribute name="failures" type="xs:string" />
+            <xs:attribute name="errors" type="xs:string" />
+        </xs:complexType>
+    </xs:element>
+
+</xs:schema>