Stylesheet testing: structural/content tests

[tomschr]

Problem

Our stylesheets has a long history. However it is not always sure if they do what they are expected to do: new upstream DocBook stylesheet changes, or an incompatible fix can destroy the result. Debugging is hard.

Suggested Solution

To fix this unfortunate situation, I would suggest an approach which have an input XML and some test file (basically containing XPath expressions and their expected results).

I got this idea is after I've implemented some test scenarios for openSUSE/rstxml2docbook. I think, we could (in theory) apply the same approach too when testing our stylesheets.

JSON Approch

Here it is my suggestion:

1. Create a input DocBook XML file (for example, admon-001.xml). This file should contain a minimalistic example (as always when creating tests).
2. Create a JSON file with the same base but with extension .params.json. This file contains a list of XPath expressions what to test.
3. Run the test suite.

The following JSON file comes from the openSUSE/rstxml2docbook project. It has tests which check if some admonition elements are there:

[
    ["namespace-uri(/*)", "http://docbook.org/ns/docbook"],
    ["boolean(//d:note)", true],
    ["local-name(/*)", "book"],
    ["/*/@xml:lang", ["en"]],
    ["/d:book/d:chapter/d:title/text()", ["Nothing"]],
    ["/d:book/d:chapter/d:note/d:para/text()", ["A note."]],
    ["/d:book/d:chapter/d:important/d:para/text()", ["Important."]],
    ["/d:book/d:chapter/d:tip/d:para/text()", ["A tip."]]
]

The JSON file:

• contains a list which consist of entries.
• Each entry have two items.
• The first is an XPath expression which is tested against the result XML.
• The second entry is the result of this test (what you expect).

In the above example, the d prefix of the DocBook namespace is predefined. Same for other namespaces.

Pros

I see the following benefits:

• Easy to add a test case: add a XML file and its corresponding test file and you're done!
• Tests can be written by a trainee or an "untrained" person which doesn't need to know the inner details about XSLT and the stylesheets.
• JSON is for this purpose easy enough. Basically, the approach can be extended to use other formats like YAML or TOML, if needed.
• Uses Python ;-)
• JSON can be easily imported and used in Python

Cons

• Maybe the JSON structure isn't enough. What about some unusual namespaces, configuration options etc. should they be integrated as well? The current structure doesn't allow that.
• XPath expressions has to be written in such a case that it not only returns an element ("/d:book") which is hard to test, but it tests a real condition ("boolean(/d:book)")
• Needs Python/Pytest

@sknorr What do you think?