Tests failed on Arch Linux

[felixonmars]

I am trying to package python-xmlsec for Arch, but unfortunately most tests are failing here. Versions:

Python 3.6.6/3.7.0
libxml2 2.9.8
xmlsec 1.2.26

Cflags: -DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_DL_LIBLTDL=1 -I/usr/include/xmlsec1 -I/usr/include/libxml2

Example test failure (others are very similar):

============================= test session starts ==============================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /build/python-xmlsec/src/python-xmlsec-1.3.3, inifile:
collected 64 items

tests/test_ds.py ..FE.FEFEFEFEFE..FEFEFEFEFE                             [ 25%]
tests/test_enc.py FEFEFEFEFE..                                           [ 35%]
tests/test_keys.py .........................                             [ 75%]
tests/test_templates.py FEFEFEFEFEFEFEFEFEFEFEFE                         [ 93%]
tests/test_tree.py FEFEFEFE                                              [100%]

==================================== ERRORS ====================================
____________ ERROR at teardown of TestSignContext.test_register_id _____________

self = <tests.test_ds.TestSignContext testMethod=test_register_id>

    def test_register_id(self):
        ctx = xmlsec.SignatureContext()
>       root = self.load_xml("sign_template.xml")

tests/test_ds.py:22:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/base.py:97: in load_xml
    root = etree.parse(self.path(name)).getroot()
src/lxml/etree.pyx:3426: in lxml.etree.parse
    ???
src/lxml/parser.pxi:1840: in lxml.etree._parseDocument
    ???
src/lxml/parser.pxi:1866: in lxml.etree._parseDocumentFromURL
    ???
src/lxml/parser.pxi:1770: in lxml.etree._parseDocFromFile
    ???
src/lxml/parser.pxi:1163: in lxml.etree._BaseParser._parseDocFromFile
    ???
src/lxml/parser.pxi:601: in lxml.etree._ParserContext._handleParseResultDoc
    ???
src/lxml/parser.pxi:711: in lxml.etree._handleParseResult
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E     File "b'/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'", line 0
E   lxml.etree.XMLSyntaxError: <no detail available>

src/lxml/parser.pxi:651: XMLSyntaxError
----------------------------- Captured stderr call -----------------------------
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=59:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'; xml error: 0: NULL
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=59:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'; xml error: 0: NULL
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=59:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'; xml error: 0: NULL
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=59:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'; xml error: 0: NULL
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=59:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'; xml error: 0: NULL
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=59:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'; xml error: 0: NULL
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=59:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'; xml error: 0: NULL
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=59:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'; xml error: 0: NULL
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=59:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'; xml error: 0: NULL
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=59:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='/build/python-xmlsec/src/python-xmlsec-1.3.3/tests/data/sign_template.xml'; xml error: 0: NULL
...
=============== 32 failed, 32 passed, 32 error in 1.73 seconds ================

[hoefling]

@felixonmars if this is still an issue, can you please check against the latest 1.3.8 release?

[felixonmars]

Yes, it's still present in 1.3.7/1.3.8 releases.

Full logs w/versions: https://paste.xinu.at/HRD/

[hoefling]

@felixonmars my first guess would be something is borked with the python-lxml dependency. If I resort to the manylinux wheel, e.g.

check() {
  cd python-xmlsec-$pkgver
  pip install --force-reinstall lxml
  python setup.py pytest
}

in the PKGBUILD, the tests pass. I will try looking further into it.

[felixonmars]

It seems the lxml here has some issues with runpy, that it fails the second time running the same example:

doc/source/examples {0}$ python
Python 3.8.3 (default, May 17 2020, 18:15:42)
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import runpy
>>> runpy.run_path('encrypt.py')
b'<xenc:CipherValue xmlns:xenc="http://www.w3.org/2001/04/xmlenc#" xmlns:dsig="http://www.w3.org/2000/09/xmldsig#">kMAH68us7GEVk42MEU8DZ/FunGVNfZD0wj+ObupHtw2oIhiGbooNnjmDORGlhRtl\nGAj/DzVP3UP7INPgP9STkyqPRazgI3Q5T8DlLQwZhXqQpBG0xQ/sezBhXykCBa6j\np97qXLKT/xn5XslxH3iEN74WwIjmUGkDS3e3O1rM3Ab3bWQKPgho1Hwxaeq+8UJK\nIVsEq4gIHsxn70o64MWE6s5X3NT+zPx1BoVT2MujRwzpywLjX6K9muNn5IDpETwt\nZ9AqnN9babR9FU1jWNBOPr/7auOQ1gvDaqVGsKcnqUINr3W9RKTPNtTsle6tBovg\nET7FbEBj19aXeueDWFNOxQ==</xenc:CipherValue>\n'
{'__name__': '<run_path>', '__doc__': None, '__package__': '', '__loader__': None, '__spec__': None, '__file__': 'encrypt.py', '__cached__': None, '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '__package__': '', '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>), '__build_class__': <built-in function __build_class__>, '__import__': <built-in function __import__>, 'abs': <built-in function abs>, 'all': <built-in function all>, 'any': <built-in function any>, 'ascii': <built-in function ascii>, 'bin': <built-in function bin>, 'breakpoint': <built-in function breakpoint>, 'callable': <built-in function callable>, 'chr': <built-in function chr>, 'compile': <built-in function compile>, 'delattr': <built-in function delattr>, 'dir': <built-in function dir>, 'divmod': <built-in function divmod>, 'eval': <built-in function eval>, 'exec': <built-in function exec>, 'format': <built-in function format>, 'getattr': <built-in function getattr>, 'globals': <built-in function globals>, 'hasattr': <built-in function hasattr>, 'hash': <built-in function hash>, 'hex': <built-in function hex>, 'id': <built-in function id>, 'input': <built-in function input>, 'isinstance': <built-in function isinstance>, 'issubclass': <built-in function issubclass>, 'iter': <built-in function iter>, 'len': <built-in function len>, 'locals': <built-in function locals>, 'max': <built-in function max>, 'min': <built-in function min>, 'next': <built-in function next>, 'oct': <built-in function oct>, 'ord': <built-in function ord>, 'pow': <built-in function pow>, 'print': <built-in function print>, 'repr': <built-in function repr>, 'round': <built-in function round>, 'setattr': <built-in function setattr>, 'sorted': <built-in function sorted>, 'sum': <built-in function sum>, 'vars': <built-in function vars>, 'None': None, 'Ellipsis': Ellipsis, 'NotImplemented': NotImplemented, 'False': False, 'True': True, 'bool': <class 'bool'>, 'memoryview': <class 'memoryview'>, 'bytearray': <class 'bytearray'>, 'bytes': <class 'bytes'>, 'classmethod': <class 'classmethod'>, 'complex': <class 'complex'>, 'dict': <class 'dict'>, 'enumerate': <class 'enumerate'>, 'filter': <class 'filter'>, 'float': <class 'float'>, 'frozenset': <class 'frozenset'>, 'property': <class 'property'>, 'int': <class 'int'>, 'list': <class 'list'>, 'map': <class 'map'>, 'object': <class 'object'>, 'range': <class 'range'>, 'reversed': <class 'reversed'>, 'set': <class 'set'>, 'slice': <class 'slice'>, 'staticmethod': <class 'staticmethod'>, 'str': <class 'str'>, 'super': <class 'super'>, 'tuple': <class 'tuple'>, 'type': <class 'type'>, 'zip': <class 'zip'>, '__debug__': True, 'BaseException': <class 'BaseException'>, 'Exception': <class 'Exception'>, 'TypeError': <class 'TypeError'>, 'StopAsyncIteration': <class 'StopAsyncIteration'>, 'StopIteration': <class 'StopIteration'>, 'GeneratorExit': <class 'GeneratorExit'>, 'SystemExit': <class 'SystemExit'>, 'KeyboardInterrupt': <class 'KeyboardInterrupt'>, 'ImportError': <class 'ImportError'>, 'ModuleNotFoundError': <class 'ModuleNotFoundError'>, 'OSError': <class 'OSError'>, 'EnvironmentError': <class 'OSError'>, 'IOError': <class 'OSError'>, 'EOFError': <class 'EOFError'>, 'RuntimeError': <class 'RuntimeError'>, 'RecursionError': <class 'RecursionError'>, 'NotImplementedError': <class 'NotImplementedError'>, 'NameError': <class 'NameError'>, 'UnboundLocalError': <class 'UnboundLocalError'>, 'AttributeError': <class 'AttributeError'>, 'SyntaxError': <class 'SyntaxError'>, 'IndentationError': <class 'IndentationError'>, 'TabError': <class 'TabError'>, 'LookupError': <class 'LookupError'>, 'IndexError': <class 'IndexError'>, 'KeyError': <class 'KeyError'>, 'ValueError': <class 'ValueError'>, 'UnicodeError': <class 'UnicodeError'>, 'UnicodeEncodeError': <class 'UnicodeEncodeError'>, 'UnicodeDecodeError': <class 'UnicodeDecodeError'>, 'UnicodeTranslateError': <class 'UnicodeTranslateError'>, 'AssertionError': <class 'AssertionError'>, 'ArithmeticError': <class 'ArithmeticError'>, 'FloatingPointError': <class 'FloatingPointError'>, 'OverflowError': <class 'OverflowError'>, 'ZeroDivisionError': <class 'ZeroDivisionError'>, 'SystemError': <class 'SystemError'>, 'ReferenceError': <class 'ReferenceError'>, 'MemoryError': <class 'MemoryError'>, 'BufferError': <class 'BufferError'>, 'Warning': <class 'Warning'>, 'UserWarning': <class 'UserWarning'>, 'DeprecationWarning': <class 'DeprecationWarning'>, 'PendingDeprecationWarning': <class 'PendingDeprecationWarning'>, 'SyntaxWarning': <class 'SyntaxWarning'>, 'RuntimeWarning': <class 'RuntimeWarning'>, 'FutureWarning': <class 'FutureWarning'>, 'ImportWarning': <class 'ImportWarning'>, 'UnicodeWarning': <class 'UnicodeWarning'>, 'BytesWarning': <class 'BytesWarning'>, 'ResourceWarning': <class 'ResourceWarning'>, 'ConnectionError': <class 'ConnectionError'>, 'BlockingIOError': <class 'BlockingIOError'>, 'BrokenPipeError': <class 'BrokenPipeError'>, 'ChildProcessError': <class 'ChildProcessError'>, 'ConnectionAbortedError': <class 'ConnectionAbortedError'>, 'ConnectionRefusedError': <class 'ConnectionRefusedError'>, 'ConnectionResetError': <class 'ConnectionResetError'>, 'FileExistsError': <class 'FileExistsError'>, 'FileNotFoundError': <class 'FileNotFoundError'>, 'IsADirectoryError': <class 'IsADirectoryError'>, 'NotADirectoryError': <class 'NotADirectoryError'>, 'InterruptedError': <class 'InterruptedError'>, 'PermissionError': <class 'PermissionError'>, 'ProcessLookupError': <class 'ProcessLookupError'>, 'TimeoutError': <class 'TimeoutError'>, 'open': <built-in function open>, 'quit': Use quit() or Ctrl-D (i.e. EOF) to exit, 'exit': Use exit() or Ctrl-D (i.e. EOF) to exit, 'copyright': Copyright (c) 2001-2020 Python Software Foundation.
All Rights Reserved.

Copyright (c) 2000 BeOpen.com.
All Rights Reserved.

Copyright (c) 1995-2001 Corporation for National Research Initiatives.
All Rights Reserved.

Copyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam.
All Rights Reserved., 'credits':     Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands
    for supporting Python development.  See www.python.org for more information., 'license': Type license() to see the full license text, 'help': Type help() for interactive help, or help(object) for help about object., '_': None}, 'etree': <module 'lxml.etree' from '/usr/lib/python3.8/site-packages/lxml/etree.cpython-38-x86_64-linux-gnu.so'>, 'xmlsec': <module 'xmlsec' from '/usr/lib/python3.8/site-packages/xmlsec.cpython-38-x86_64-linux-gnu.so'>, 'manager': <xmlsec.KeysManager object at 0x7ff0fbae5b30>, 'key': <xmlsec.Key object at 0x7ff0f94395b0>, 'template': <Element Envelope at 0x7ff0f943ae40>, 'enc_data': <Element {http://www.w3.org/2001/04/xmlenc#}EncryptedData at 0x7ff0f9433440>, 'key_info': <Element {http://www.w3.org/2000/09/xmldsig#}KeyInfo at 0x7ff0f943ae80>, 'enc_key': <Element {http://www.w3.org/2001/04/xmlenc#}EncryptedKey at 0x7ff0f943aec0>, 'data': <Element Data at 0x7ff0f943af80>, 'enc_ctx': <xmlsec.EncryptionContext object at 0x7ff0f94395f0>, 'enc_method': <Element {http://www.w3.org/2001/04/xmlenc#}EncryptionMethod at 0x7ff0f943e040>, 'cipher_value': <Element {http://www.w3.org/2001/04/xmlenc#}CipherValue at 0x7ff0f943afc0>}
>>> runpy.run_path('encrypt.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "encrypt.py", line 8, in <module>
    template = etree.parse('enc1-doc.xml').getroot()
  File "src/lxml/etree.pyx", line 3521, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1839, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1865, in lxml.etree._parseDocumentFromURL
  File "src/lxml/parser.pxi", line 1769, in lxml.etree._parseDocFromFile
  File "src/lxml/parser.pxi", line 1163, in lxml.etree._BaseParser._parseDocFromFile
  File "src/lxml/parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 711, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 651, in lxml.etree._raiseParseError
  File "b'enc1-doc.xml'", line 0
lxml.etree.XMLSyntaxError

Using subprocess.Popen here instead of runpy makes the tests pass.

[lilydjwg]

Here is my findings. Put together a xml file and run this:

from lxml import etree
import ctypes

xmlsec = ctypes.CDLL('/usr/lib/libxmlsec1.so')
xmlsec.xmlSecInit()

etree.parse('enc1-doc.xml')
print('ok')
etree.parse('enc1-doc.xml')

It reproduces. The second parse call only triggers a stat call but no file open and read calls.

lxml, xmlsec and libxml2 are all coming from my Arch Linux.

It looks like the lxml wheel from PyPI (installed by pip) has its own libxml2, but xmlsec dynamically links to the system one, so xmlsec changes one libxml2 while lxml uses another, avoiding this issue. (But I don't know if it's OK to have two libxml2 libraries like this.)

I don't know how to use the C libraries so no reproduction tries without Python.

[hoefling]

@lilydjwg your comment was of an immense help and it brought me on the right track, thank you!

I found out that this issue is caused by lxml reusing the XML parser object: e.g.

from lxml import etree
import xmlsec

etree.parse('doc.xml', parser=etree.XMLParser())
etree.parse('doc.xml', parser=etree.XMLParser())

works, but

from lxml import etree
import xmlsec

p = etree.XMLParser()
etree.parse('doc.xml', parser=p)
etree.parse('doc.xml', parser=p)

raises. The same issue arises when calling etree.parse('doc.xml') - the default parser stored globally is reused.

The workaround to this issue is thus simple - resetting the default parser between the load_xml invocations should be enough to fix the failing tests. However, this doesn't fix the underlying issue #96 yet. I'll poke around some more and open an issue against lxml launchpad if won't be able to resolve it myself.

[hoefling]

In the meantime, a simple patch like e.g.

diff --git a/tests/base.py b/tests/base.py
index b05de1d..5ec356f 100644
--- a/tests/base.py
+++ b/tests/base.py
@@ -94,6 +94,7 @@ class TestMemoryLeaks(unittest.TestCase):
 
     def load_xml(self, name, xpath=None):
         """returns xml.etree"""
+        etree.set_default_parser(parser=etree.XMLParser())
         root = etree.parse(self.path(name)).getroot()
         if xpath is None:
             return root
diff --git a/tests/test_doc_examples.py b/tests/test_doc_examples.py
index 2fc490f..53d2377 100644
--- a/tests/test_doc_examples.py
+++ b/tests/test_doc_examples.py
@@ -42,3 +42,5 @@ def test_doc_example(example):
     """
     with cd(example.parent):
         runpy.run_path(str(example))
+    from lxml import etree
+    etree.set_default_parser(parser=etree.XMLParser())

will fix the test run, but I'm sure this is not what you're after in the end.

[hoefling]

For reference: https://bugs.launchpad.net/lxml/+bug/1880251

[odidev]

@hoefling, Thanks for the above solution.
All tests are passing after upgrading lxml to 4.5.2 in amd64 platform, but in ARM64/AArch64 platform even after upgrading lxml, I am getting same lxml syntax error and all tests are passing after incorporating above mentioned changes.

It will be helpful if the above mentioned changes are added or suggest if I am missing any installation of any other dependencies.

[hoefling]

@odidev if you're getting the same error with lxml==4.5.2, then the issue is not completely resolved on lxml side (I also think there's more to do as the example I posted in this comment fails with 4.5.2). I'd open a new issue with a reproduction example at lxml's launchpad.

[hoefling]

Follow-up issue: https://bugs.launchpad.net/lxml/+bug/1887848