Problems running tests

[FabioLolix]

Hello, I'm the maintainer of the AUR pkgbuild https://aur.archlinux.org/packages/python-pygccxml, performing tests is recomended by Arch packaging guidelines, I'm testing several things, reports below are made building from the last commit of develop branch

Castxml v0.6.2 build againt calng/llvm v16.0.6

Using

 python -m unittests.test_all
 python -m coverage report -m

The first issues is that it want pygccxml already installed which isn't possible in this case

Successfully built pygccxml-2.4.0-py3-none-any.whl
==> Starting check()...
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml/unittests/test_all.py", line 11, in <module>
    from . import decl_string_tester
  File "/home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml/unittests/decl_string_tester.py", line 9, in <module>
    from . import autoconfig
  File "/home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml/unittests/autoconfig.py", line 26, in <module>
    from pygccxml import parser  # nopep8
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: cannot import name 'parser' from 'pygccxml' (unknown location)
==> ERROR: A failure occurred in check().

Building the package and running again bring

pygccxml/unittests/data/core_types.hpp:57:38: error: ISO C++17 does not allow dynamic exception specifications [-Wdynamic-exception-spec]

Successfully built pygccxml-2.4.0-py3-none-any.whl
==> Starting check()...
/home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml/unittests/data/core_types.hpp:57:38: error: ISO C++17 does not allow dynamic exception specifications [-Wdynamic-exception-spec]
        int some_function( double hi) const throw( exception ){
                                            ^~~~~~~~~~~~~~~~~~
/home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml/unittests/data/core_types.hpp:57:38: note: use 'noexcept(false)' instead
        int some_function( double hi) const throw( exception ){
                                            ^~~~~~~~~~~~~~~~~~
                                            noexcept(false)
1 error generated.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml/unittests/test_all.py", line 203, in <module>
    sys.exit(run_suite())
             ^^^^^^^^^^^
  File "/home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml/unittests/test_all.py", line 193, in run_suite
    result = unittest.TextTestRunner(verbosity=2).run(create_suite())
                                                      ^^^^^^^^^^^^^^
  File "/home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml/unittests/test_all.py", line 188, in create_suite
    main_suite.addTest(tester.create_suite())
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml/unittests/decl_printer_tester.py", line 70, in create_suite
    unittest.TestLoader().loadTestsFromTestCase(testCaseClass=Test))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/loader.py", line 93, in loadTestsFromTestCase
    loaded_suite = self.suiteClass(map(testCaseClass, testCaseNames))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/suite.py", line 24, in __init__
    self.addTests(tests)
  File "/usr/lib/python3.11/unittest/suite.py", line 57, in addTests
    for test in tests:
  File "/home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml/unittests/decl_printer_tester.py", line 40, in __init__
    self.decls = prj_reader.read_files(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pygccxml/parser/project_reader.py", line 264, in read_files
    return self.__parse_file_by_file(files)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pygccxml/parser/project_reader.py", line 292, in __parse_file_by_file
    decls = reader.read_file(header)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pygccxml/parser/source_reader.py", line 303, in read_file
    return self.read_cpp_source_file(source_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pygccxml/parser/source_reader.py", line 322, in read_cpp_source_file
    xml_file = self.create_xml_file(ffname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pygccxml/parser/source_reader.py", line 263, in create_xml_file
    raise RuntimeError(
RuntimeError: Error occurred while running CASTXML xml file does not exist
==> ERROR: A failure occurred in check().

Instead using pytest -v I get

unittests/test_overrides.py::Test::test FAILED [ 64%]

Successfully built pygccxml-2.4.0-py3-none-any.whl
==> Starting check()...
========================================================================================= test session starts =========================================================================================
platform linux -- Python 3.11.5, pytest-7.4.2, pluggy-1.3.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/fabio/Dev/Github/PKGBUILD-AUR_fix/p/python-pygccxml-dev-git/src/pygccxml
collected 39 items                                                                                                                                                                                    

unittests/test_argument_without_name.py::Test::test_argument_without_name PASSED                                                                                                                [  2%]
unittests/test_castxml_wrong_epic.py::Test::test_castxml_epic_version_check PASSED                                                                                                              [  5%]
unittests/test_ccflags.py::Test::test PASSED                                                                                                                                                    [  7%]
unittests/test_comments.py::Test::test PASSED                                                                                                                                                   [ 10%]
unittests/test_config.py::Test::test_config PASSED                                                                                                                                              [ 12%]
unittests/test_copy_constructor.py::Test::test PASSED                                                                                                                                           [ 15%]
unittests/test_cpp_standards.py::Test::test PASSED                                                                                                                                              [ 17%]
unittests/test_create_decl_string.py::Test::test PASSED                                                                                                                                         [ 20%]
unittests/test_deprecation.py::Test::test PASSED                                                                                                                                                [ 23%]
unittests/test_directory_cache.py::Test::test_directory_cache_twice PASSED                                                                                                                      [ 25%]
unittests/test_directory_cache.py::Test::test_directory_cache_with_compression PASSED                                                                                                           [ 28%]
unittests/test_directory_cache.py::Test::test_directory_cache_without_compression PASSED                                                                                                        [ 30%]
unittests/test_directory_cache.py::Test::test_directory_existing_dir PASSED                                                                                                                     [ 33%]
unittests/test_elaborated_types.py::Test::test_is_elaborated_type PASSED                                                                                                                        [ 35%]
unittests/test_find_noncopyable_vars.py::Test::test PASSED                                                                                                                                      [ 38%]
unittests/test_function_pointer.py::Test::test_function_pointer PASSED                                                                                                                          [ 41%]
unittests/test_hash.py::Test::test_declarations_hashes PASSED                                                                                                                                   [ 43%]
unittests/test_hash.py::Test::test_type_qualifiers_t_hash PASSED                                                                                                                                [ 46%]
unittests/test_hash.py::Test::test_types_hashes PASSED                                                                                                                                          [ 48%]
unittests/test_map_gcc5.py::Test::test_map_gcc5 PASSED                                                                                                                                          [ 51%]
unittests/test_non_copyable_recursive.py::Test::test_infinite_recursion_base_classes PASSED                                                                                                     [ 53%]
unittests/test_non_copyable_recursive.py::Test::test_infinite_recursion_sstream PASSED                                                                                                          [ 56%]
unittests/test_null_comparison.py::Test::test_argument_null_comparison PASSED                                                                                                                   [ 58%]
unittests/test_order.py::Test::test_order PASSED                                                                                                                                                [ 61%]
unittests/test_overrides.py::Test::test FAILED                                                                                                                                                  [ 64%]
unittests/test_pattern_parser.py::Test::test_matcher PASSED                                                                                                                                     [ 66%]
unittests/test_pattern_parser.py::Test::test_split PASSED                                                                                                                                       [ 69%]
unittests/test_pattern_parser.py::Test::test_template_split_std_vector PASSED                                                                                                                   [ 71%]
unittests/test_smart_pointer.py::Test::test_auto_pointer_value_type PASSED                                                                                                                      [ 74%]
unittests/test_smart_pointer.py::Test::test_is_auto_pointer PASSED                                                                                                                              [ 76%]
unittests/test_smart_pointer.py::Test::test_is_smart_pointer PASSED                                                                                                                             [ 79%]
unittests/test_smart_pointer.py::Test::test_smart_pointer_value_type PASSED                                                                                                                     [ 82%]
unittests/test_utils.py::Test::test PASSED                                                                                                                                                      [ 84%]
unittests/test_utils.py::Test::test_deprecation_wrapper PASSED                                                                                                                                  [ 87%]
unittests/test_va_list_tag_removal.py::Test::test_keep_va_list_tag PASSED                                                                                                                       [ 89%]
unittests/test_va_list_tag_removal.py::Test::test_remove_va_list_tag PASSED                                                                                                                     [ 92%]
unittests/test_warn_missing_include_dirs.py::Test::test_config PASSED                                                                                                                           [ 94%]
unittests/test_xml_generators.py::Test::test_casxtml_epic_version_1 PASSED                                                                                                                      [ 97%]
unittests/test_xml_generators.py::Test::test_old_xml_generators PASSED                                                                                                                          [100%]

============================================================================================== FAILURES ===============================================================================================
______________________________________________________________________________________________ Test.test ______________________________________________________________________________________________

self = <unittests.test_overrides.Test testMethod=test>

    def setUp(self):
    
        if not self.global_ns:
>           decls = parser.parse([self.header], self.config)

unittests/test_overrides.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.11/site-packages/pygccxml/parser/__init__.py:51: in parse
    declarations = parser.read_files(files, compilation_mode)
/usr/lib/python3.11/site-packages/pygccxml/parser/project_reader.py:264: in read_files
    return self.__parse_file_by_file(files)
/usr/lib/python3.11/site-packages/pygccxml/parser/project_reader.py:292: in __parse_file_by_file
    decls = reader.read_file(header)
/usr/lib/python3.11/site-packages/pygccxml/parser/source_reader.py:303: in read_file
    return self.read_cpp_source_file(source_file)
/usr/lib/python3.11/site-packages/pygccxml/parser/source_reader.py:323: in read_cpp_source_file
    decls, files = self.__parse_xml_file(xml_file)
/usr/lib/python3.11/site-packages/pygccxml/parser/source_reader.py:421: in __parse_xml_file
    patcher.fix_calldef_decls(
/usr/lib/python3.11/site-packages/pygccxml/parser/patcher.py:261: in fix_calldef_decls
    default_arg_patcher(decl)
/usr/lib/python3.11/site-packages/pygccxml/parser/patcher.py:23: in __call__
    fixer = self.__find_fixer(decl, arg)
/usr/lib/python3.11/site-packages/pygccxml/parser/patcher.py:32: in __find_fixer
    elif self.__is_invalid_integral(func, arg):
/usr/lib/python3.11/site-packages/pygccxml/parser/patcher.py:75: in __is_invalid_integral
    if not declarations.is_integral(type_):
/usr/lib/python3.11/site-packages/pygccxml/declarations/type_traits.py:213: in is_integral
    return remove_alias(type_) in _integral_def
/usr/lib/python3.11/site-packages/pygccxml/declarations/cpptypes.py:31: in __eq__
    return self.decl_string == other.decl_string
/usr/lib/python3.11/site-packages/pygccxml/declarations/cpptypes.py:50: in decl_string
    self.cache.decl_string = self.build_decl_string()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pygccxml.declarations.cpptypes.elaborated_t object at 0x7fa96a7843d0>, with_defaults = True

    def build_decl_string(self, with_defaults=True):
>       if hasattr(self.base.declaration, "elaborated_type_specifier"):
E       AttributeError: 'int_t' object has no attribute 'declaration'

/usr/lib/python3.11/site-packages/pygccxml/declarations/cpptypes.py:596: AttributeError
======================================================================================= short test summary info =======================================================================================
FAILED unittests/test_overrides.py::Test::test - AttributeError: 'int_t' object has no attribute 'declaration'
==================================================================================== 1 failed, 38 passed in 9.98s =====================================================================================
==> ERROR: A failure occurred in check().

[FabioLolix]

I'm the maintainer of the AUR pkgbuild https://aur.archlinux.org/packages/python-pygccxml

The package have been moved to Arch official repository and is now maintained by @carlsmedstad

[iMichka]

Hey. I was looking into this. I'm currently cleaning up tests (moving to pytest for a better readability and ease of use) and will also add some more CI combinations later on.

I also fixed a few bugs already while cleaning up the tests.

I hope to be able to ship a new release soon, and we can then check this from there.

[maitbot]

Hey. I was looking into this. I'm currently cleaning up tests (moving to pytest
I hope to be able to ship a new release soon, and we can then check this from there.

The move to use pytest looks like a good idea.

FYI, I've enabled testing when building Debian packages for 2.6.0, and get failures.
Build logs showing test results are now available at
https://buildd.debian.org/status/fetch.php?pkg=pygccxml&arch=all&ver=2.6.0-1&stamp=1733103542&raw=0

(using castxml 0.6.10 and likely newer compilers than the github ci.)
Hope you think fixing this is easier than I think it is :)

[iMichka]

Ok that's a lot of errors.

An easy one which should give us some visibility will be fixed by this: #216

I have another one in the pipe #163, this is an issue with spaces that I have been aware of for some time now: I have almost a fix for it but it's not an easy one.

[FabioLolix]

Hi, with v3.0.0 I was able to pass all tests

tests/test_xmlfile_reader.py::test_read_xml_file PASSED                  [100%]

======================= 284 passed in 212.43s (0:03:32) ========================

The only thing left is it want pygccxml already installed before making tests

[iMichka]

Cool.

The only thing left is it want pygccxml already installed before making tests

I can fix this by adding

[tool.pytest.ini_options]
pythonpath = [
  "src"
]

in the pyproject.toml file. https://stackoverflow.com/questions/50155464/using-pytest-with-a-src-layer/50156706#50156706
Running python3 -m pytest tests should then work, without needing to install the package before running the tests.

Though initially I always wanted to make sure we test against the installed version, just in case we miss some files when shipping a release. I had the impression that was the best way to do testing ...
But having to install pygccxml each time has also proven to make local debugging more tedious

[FabioLolix]

I can fix this by adding

[tool.pytest.ini_options]
pythonpath = [
"src"
]

in the pyproject.toml file.

Thanks! I tested it FabioLolix/PKGBUILD-WIP@3cdc412 and only one test is failing for ModuleNotFoundError: No module named 'pygccxml'

tests/test_xmlfile_reader.py::test_read_xml_file PASSED                  [100%]

=================================== FAILURES ===================================
_________________________________ test_example _________________________________

    def test_example():
        """Runs the example in the docs directory"""
    
        env = os.environ.copy()
    
        # Get the path to current directory
        path = os.path.dirname(os.path.realpath(__file__))
        # Set the COVERAGE_PROCESS_START env. variable.
        # Allows to cover files run in a subprocess
        # http://nedbatchelder.com/code/coverage/subprocess.html
        env["COVERAGE_PROCESS_START"] = path + "/../.coveragerc"
    
        # Find all the examples files
        file_paths = []
        for root, _, filenames in os.walk(path + "/../docs/examples"):
            for file_path in fnmatch.filter(filenames, '*.py'):
                file_paths.append(os.path.join(root, file_path))
    
        for file_path in file_paths:
            return_code = subprocess.call(
                ["python", path + "/example_tester_wrap.py", file_path],
                env=env)
>           assert return_code == 0
E           assert 1 == 0

tests/test_example.py:33: AssertionError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/build/python-pygccxml-git/src/pygccxml/tests/example_tester_wrap.py", line 33, in <module>
    exec(code, None, None)
    ~~~~^^^^^^^^^^^^^^^^^^
  File "/build/python-pygccxml-git/src/pygccxml/tests/../docs/examples/caching/example.py", line 6, in <module>
    from pygccxml import utils
ModuleNotFoundError: No module named 'pygccxml'
=========================== short test summary info ============================
FAILED tests/test_example.py::test_example - assert 1 == 0
================== 1 failed, 283 passed in 214.78s (0:03:34) ===================
==> ERROR: A failure occurred in check().
```

[iMichka]

I'm a still wondering if this is the best solution.
I do not know how AUR pkgbuild works.

How are tests run on other python packages? In general tests are run against the installed version, not the source code.
Do you have examples of other packages which we could look at (ideally a package with a pyproject.toml file, and that uses pytest).

In our CI what we do is running these two commands:

pip install '.[test]'
pytest tests

The first one will install test dependencies, and pygccxml. If you work in a virtualenv this should be straightforward.

[carlsmedstad]

@FabioLolix, @iMichka Thanks for working on this. Latest Arch Linux package builds with all the tests passing:
https://gitlab.archlinux.org/archlinux/packaging/packages/python-pygccxml/-/commit/b889aec7f2ebddba006b7a84951cac44a12ac65c

[iMichka]

Cool thanks @carlsmedstad

@FabioLolix is there still something to do for AUR? Or can we close this issue?

[FabioLolix]

hey, Carl did a workaround so we can close

Do you have examples of other packages which we could look at (ideally a package with a pyproject.toml file, and that uses pytest).

Mutagen do https://github.com/quodlibet/mutagen/blob/main/pyproject.toml

https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-mutagen-git

---

1k+ Arch packages run pytest on the locally build package https://archlinux.org/packages/extra/any/python-pytest/, maybe I'll return on this but not soon I guess

[FabioLolix]

Thanks for your work