[clang][python][test] Move python binding tests to lit framework #146486
+87
−69
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As discussed in PR llvm#142353, the current testsuite of the `clang` Python bindings has several issues: - It `libclang.so` cannot be loaded into `python` to run the testsuite, the whole `ninja check-all` aborts. - The result of running the testsuite isn't report like the `lit`-based tests, rendering them almost invisible. - The testsuite is disabled in a non-obvious way (`RUN_PYTHON_TESTS`) in `tests/CMakeLists.txt`, which again doesn't show up in the test results. All these issues can be avoided by integrating the Python bindings tests with `lit`, which is what this patch does: - The actual test lives in `clang/test/bindings/python/bindings.sh` and is run by `lit`. - The current `clang/bindings/python/tests` directory (minus the now-subperfluous `CMakeLists.txt`) is moved into the same directory. - The check if `libclang` is loadable (originally from PR llvm#142353) is now handled via a new `lit` feature, `libclang-loadable`. - The various ways to disable the tests have been turned into `XFAIL`s as appropriate. This isn't complete and not completely tested yet. Tested on `sparc-sun-solaris2.11`, `sparcv9-sun-solaris2.11`, `i386-pc-solaris2.11`, `amd64-pc-solaris2.11`, `i686-pc-linux-gnu`, and `x86_64-pc-linux-gnu`.
llvmbot
added
clang
|
@llvm/pr-subscribers-clang Author: Jannick Kremer (DeinAlptraum) ChangesAs discussed in PR #142353, the current testsuite of the
All these issues can be avoided by integrating the Python bindings tests with
Tested on Full diff: https://github.com/llvm/llvm-project/pull/146486.diff 40 Files Affected:
diff --git a/.github/workflows/libclang-python-tests.yml b/.github/workflows/libclang-python-tests.yml
index 50ef4acf2feb1..43b50cec61716 100644
--- a/.github/workflows/libclang-python-tests.yml
+++ b/.github/workflows/libclang-python-tests.yml
@@ -10,15 +10,15 @@ on:
- 'main'
paths:
- 'clang/bindings/python/**'
+ - 'clang/test/bindings/python/**'
- 'clang/tools/libclang/**'
- - 'clang/CMakeList.txt'
- '.github/workflows/libclang-python-tests.yml'
- '.github/workflows/llvm-project-tests.yml'
pull_request:
paths:
- 'clang/bindings/python/**'
+ - 'clang/test/bindings/python/**'
- 'clang/tools/libclang/**'
- - 'clang/CMakeList.txt'
- '.github/workflows/libclang-python-tests.yml'
- '.github/workflows/llvm-project-tests.yml'
diff --git a/clang/CMakeLists.txt b/clang/CMakeLists.txt
index 94607a8e8473c..72386132bce9f 100644
--- a/clang/CMakeLists.txt
+++ b/clang/CMakeLists.txt
@@ -530,7 +530,6 @@ if( CLANG_INCLUDE_TESTS )
clang_unit_site_config=${CMAKE_CURRENT_BINARY_DIR}/test/Unit/lit.site.cfg
)
add_subdirectory(test)
- add_subdirectory(bindings/python/tests)
if(CLANG_BUILT_STANDALONE)
umbrella_lit_testsuite_end(check-all)
diff --git a/clang/bindings/python/tests/CMakeLists.txt b/clang/bindings/python/tests/CMakeLists.txt
deleted file mode 100644
index a0ddabc21bb41..0000000000000
--- a/clang/bindings/python/tests/CMakeLists.txt
+++ /dev/null
@@ -1,66 +0,0 @@
-# Test target to run Python test suite from main build.
-
-# Avoid configurations including '-include' from interfering with
-# our tests by setting CLANG_NO_DEFAULT_CONFIG.
-add_custom_target(check-clang-python
- COMMAND ${CMAKE_COMMAND} -E env
- CLANG_NO_DEFAULT_CONFIG=1
- CLANG_LIBRARY_PATH=$<TARGET_FILE_DIR:libclang>
- "${Python3_EXECUTABLE}" -m unittest discover
- DEPENDS libclang
- WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/..)
-
-set(RUN_PYTHON_TESTS TRUE)
-set_target_properties(check-clang-python PROPERTIES FOLDER "Clang/Tests")
-
-# Tests require libclang.so which is only built with LLVM_ENABLE_PIC=ON
-if(NOT LLVM_ENABLE_PIC)
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-# Do not try to run if libclang was built with sanitizers because
-# the sanitizer library will likely be loaded too late to perform
-# interception and will then fail.
-# We could use LD_PRELOAD/DYLD_INSERT_LIBRARIES but this isn't
-# portable so its easier just to not run the tests when building
-# with ASan.
-if(NOT LLVM_USE_SANITIZER STREQUAL "")
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-# Tests fail on Windows, and need someone knowledgeable to fix.
-# It's not clear whether it's a test or a valid binding problem.
-if(WIN32)
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-# The Python FFI interface is broken on AIX: https://bugs.python.org/issue38628.
-if(${CMAKE_SYSTEM_NAME} MATCHES "AIX")
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-# AArch64, Hexagon, and Sparc have known test failures that need to be
-# addressed.
-# SystemZ has broken Python/FFI interface:
-# https://reviews.llvm.org/D52840#1265716
-if(${LLVM_NATIVE_ARCH} MATCHES "^(AArch64|Hexagon|Sparc|SystemZ)$")
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-# Tests will fail if cross-compiling for a different target, as tests will try
-# to use the host Python3_EXECUTABLE and make FFI calls to functions in target
-# libraries.
-if(CMAKE_CROSSCOMPILING)
- # FIXME: Consider a solution that allows better control over these tests in
- # a crosscompiling scenario. e.g. registering them with lit to allow them to
- # be explicitly skipped via appropriate LIT_ARGS, or adding a mechanism to
- # allow specifying a python interpreter compiled for the target that could
- # be executed using qemu-user.
- message(WARNING "check-clang-python not added to check-all as these tests fail in a cross-build setup")
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-if(RUN_PYTHON_TESTS)
- set_property(GLOBAL APPEND PROPERTY
- LLVM_ALL_ADDITIONAL_TEST_TARGETS check-clang-python)
-endif()
diff --git a/clang/test/CMakeLists.txt b/clang/test/CMakeLists.txt
index e5b4a3bb84645..a9b88d4f4d9ec 100644
--- a/clang/test/CMakeLists.txt
+++ b/clang/test/CMakeLists.txt
@@ -205,6 +205,17 @@ add_custom_target(clang-test)
add_dependencies(clang-test check-clang)
set_target_properties(clang-test PROPERTIES FOLDER "Clang/Tests")
+# Allow running Clang Python binding tests separately from CI.
+add_lit_testsuite(check-clang-python "Running the Clang Python tests"
+ ${CMAKE_CURRENT_BINARY_DIR}
+ #LIT ${LLVM_LIT}
+ PARAMS ${CLANG_TEST_PARAMS}
+ DEPENDS ${CLANG_TEST_DEPS}
+ ARGS ${CLANG_TEST_EXTRA_ARGS} --filter=bindings.sh
+ # Avoid running tests twice.
+ EXCLUDE_FROM_CHECK_ALL
+ )
+
# FIXME: This logic can be removed once all buildbots have moved
# debuginfo-test from clang/test to llvm/projects or monorepo.
if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/debuginfo-tests)
diff --git a/clang/test/bindings/python/bindings.sh b/clang/test/bindings/python/bindings.sh
new file mode 100755
index 0000000000000..c5cc6e4e25ffe
--- /dev/null
+++ b/clang/test/bindings/python/bindings.sh
@@ -0,0 +1,33 @@
+#!/bin/sh
+
+# UNSUPPORTED: !libclang-loadable
+
+# Tests fail on Windows, and need someone knowledgeable to fix.
+# It's not clear whether it's a test or a valid binding problem.
+# XFAIL: target={{.*windows.*}}
+
+# The Python FFI interface is broken on AIX: https://bugs.python.org/issue38628.
+# XFAIL: target={{.*-aix.*}}
+
+# Hexagon has known test failures that need to be addressed.
+# SystemZ has broken Python/FFI interface:
+# https://reviews.llvm.org/D52840#1265716
+# XFAIL: target={{(hexagon|s390x)-.*}}
+# python SEGVs on Linux/sparc64 when loading libclang.so. Seems to be an FFI
+# issue, too.
+# XFAIL: target={{sparc.*-.*-linux.*}}
+
+# Tests will fail if cross-compiling for a different target, as tests will try
+# to use the host Python3_EXECUTABLE and make FFI calls to functions in target
+# libraries.
+#
+# FIXME: Consider a solution that allows better control over these tests in
+# a crosscompiling scenario. e.g. registering them with lit to allow them to
+# be explicitly skipped via appropriate LIT_ARGS, or adding a mechanism to
+# allow specifying a python interpreter compiled for the target that could
+# be executed using qemu-user.
+# REQUIRES: native
+
+# RUN: env PYTHONPATH=%S/../../../bindings/python \
+# RUN: CLANG_LIBRARY_PATH=%libdir \
+# RUN: %python -m unittest discover -s %S/tests
diff --git a/clang/test/bindings/python/lit.local.cfg b/clang/test/bindings/python/lit.local.cfg
new file mode 100644
index 0000000000000..cc3bdf8ba97d4
--- /dev/null
+++ b/clang/test/bindings/python/lit.local.cfg
@@ -0,0 +1,41 @@
+def is_libclang_loadable():
+ # Do not try to run if libclang was built with sanitizers because
+ # the sanitizer library will likely be loaded too late to perform
+ # interception and will then fail.
+ # We could use LD_PRELOAD/DYLD_INSERT_LIBRARIES but this isn't
+ # portable so its easier just to not run the tests when building
+ # with ASan.
+ if config.llvm_use_sanitizer != "":
+ return False
+ try:
+ sys.path.append(os.path.join(config.clang_src_dir, "bindings/python"))
+ from clang.cindex import Config
+ conf = Config()
+ Config.set_library_path(config.clang_lib_dir)
+ conf.lib
+ return True
+ except Exception as e:
+ # Expected failure modes are considered benign when nothing can be
+ # done about them.
+ #
+ # Cannot load a 32-bit libclang.so into a 64-bit python.
+ if "wrong ELF class: ELFCLASS32" in str(e):
+ return False
+ # If libclang.so is missing, it must have been disabled intentionally,
+ # e.g. by building with LLVM_ENABLE_PIC=OFF.
+ elif "No such file or directory" in str(e):
+ return False
+ # Unexpected failure modes need to be investigated to either fix an
+ # underlying bug or accept the failure, so return True. This causes
+ # tests to run and FAIL, drawing developer attention.
+ else:
+ print("warning: unhandled failure in is_libclang_loadable: "
+ + str(e), file=sys.stderr)
+ return True
+
+if is_libclang_loadable():
+ config.available_features.add("libclang-loadable")
+
+config.substitutions.append(('%libdir', config.clang_lib_dir))
+
+config.suffixes = ['.sh']
diff --git a/clang/bindings/python/tests/__init__.py b/clang/test/bindings/python/tests/__init__.py
similarity index 100%
rename from clang/bindings/python/tests/__init__.py
rename to clang/test/bindings/python/tests/__init__.py
diff --git a/clang/bindings/python/tests/cindex/INPUTS/a.inc b/clang/test/bindings/python/tests/cindex/INPUTS/a.inc
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/a.inc
rename to clang/test/bindings/python/tests/cindex/INPUTS/a.inc
diff --git a/clang/bindings/python/tests/cindex/INPUTS/b.inc b/clang/test/bindings/python/tests/cindex/INPUTS/b.inc
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/b.inc
rename to clang/test/bindings/python/tests/cindex/INPUTS/b.inc
diff --git a/clang/bindings/python/tests/cindex/INPUTS/compile_commands.json b/clang/test/bindings/python/tests/cindex/INPUTS/compile_commands.json
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/compile_commands.json
rename to clang/test/bindings/python/tests/cindex/INPUTS/compile_commands.json
diff --git a/clang/bindings/python/tests/cindex/INPUTS/header1.h b/clang/test/bindings/python/tests/cindex/INPUTS/header1.h
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/header1.h
rename to clang/test/bindings/python/tests/cindex/INPUTS/header1.h
diff --git a/clang/bindings/python/tests/cindex/INPUTS/header2.h b/clang/test/bindings/python/tests/cindex/INPUTS/header2.h
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/header2.h
rename to clang/test/bindings/python/tests/cindex/INPUTS/header2.h
diff --git a/clang/bindings/python/tests/cindex/INPUTS/header3.h b/clang/test/bindings/python/tests/cindex/INPUTS/header3.h
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/header3.h
rename to clang/test/bindings/python/tests/cindex/INPUTS/header3.h
diff --git a/clang/bindings/python/tests/cindex/INPUTS/hello.cpp b/clang/test/bindings/python/tests/cindex/INPUTS/hello.cpp
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/hello.cpp
rename to clang/test/bindings/python/tests/cindex/INPUTS/hello.cpp
diff --git a/clang/bindings/python/tests/cindex/INPUTS/include.cpp b/clang/test/bindings/python/tests/cindex/INPUTS/include.cpp
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/include.cpp
rename to clang/test/bindings/python/tests/cindex/INPUTS/include.cpp
diff --git a/clang/bindings/python/tests/cindex/INPUTS/parse_arguments.c b/clang/test/bindings/python/tests/cindex/INPUTS/parse_arguments.c
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/parse_arguments.c
rename to clang/test/bindings/python/tests/cindex/INPUTS/parse_arguments.c
diff --git a/clang/bindings/python/tests/cindex/INPUTS/testfile.c b/clang/test/bindings/python/tests/cindex/INPUTS/testfile.c
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/testfile.c
rename to clang/test/bindings/python/tests/cindex/INPUTS/testfile.c
diff --git a/clang/bindings/python/tests/cindex/__init__.py b/clang/test/bindings/python/tests/cindex/__init__.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/__init__.py
rename to clang/test/bindings/python/tests/cindex/__init__.py
diff --git a/clang/bindings/python/tests/cindex/test_access_specifiers.py b/clang/test/bindings/python/tests/cindex/test_access_specifiers.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_access_specifiers.py
rename to clang/test/bindings/python/tests/cindex/test_access_specifiers.py
diff --git a/clang/bindings/python/tests/cindex/test_cdb.py b/clang/test/bindings/python/tests/cindex/test_cdb.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_cdb.py
rename to clang/test/bindings/python/tests/cindex/test_cdb.py
diff --git a/clang/bindings/python/tests/cindex/test_code_completion.py b/clang/test/bindings/python/tests/cindex/test_code_completion.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_code_completion.py
rename to clang/test/bindings/python/tests/cindex/test_code_completion.py
diff --git a/clang/bindings/python/tests/cindex/test_comment.py b/clang/test/bindings/python/tests/cindex/test_comment.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_comment.py
rename to clang/test/bindings/python/tests/cindex/test_comment.py
diff --git a/clang/bindings/python/tests/cindex/test_cursor.py b/clang/test/bindings/python/tests/cindex/test_cursor.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_cursor.py
rename to clang/test/bindings/python/tests/cindex/test_cursor.py
diff --git a/clang/bindings/python/tests/cindex/test_cursor_kind.py b/clang/test/bindings/python/tests/cindex/test_cursor_kind.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_cursor_kind.py
rename to clang/test/bindings/python/tests/cindex/test_cursor_kind.py
diff --git a/clang/bindings/python/tests/cindex/test_diagnostics.py b/clang/test/bindings/python/tests/cindex/test_diagnostics.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_diagnostics.py
rename to clang/test/bindings/python/tests/cindex/test_diagnostics.py
diff --git a/clang/bindings/python/tests/cindex/test_enums.py b/clang/test/bindings/python/tests/cindex/test_enums.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_enums.py
rename to clang/test/bindings/python/tests/cindex/test_enums.py
diff --git a/clang/bindings/python/tests/cindex/test_exception_specification_kind.py b/clang/test/bindings/python/tests/cindex/test_exception_specification_kind.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_exception_specification_kind.py
rename to clang/test/bindings/python/tests/cindex/test_exception_specification_kind.py
diff --git a/clang/bindings/python/tests/cindex/test_file.py b/clang/test/bindings/python/tests/cindex/test_file.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_file.py
rename to clang/test/bindings/python/tests/cindex/test_file.py
diff --git a/clang/bindings/python/tests/cindex/test_index.py b/clang/test/bindings/python/tests/cindex/test_index.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_index.py
rename to clang/test/bindings/python/tests/cindex/test_index.py
diff --git a/clang/bindings/python/tests/cindex/test_lib.py b/clang/test/bindings/python/tests/cindex/test_lib.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_lib.py
rename to clang/test/bindings/python/tests/cindex/test_lib.py
diff --git a/clang/bindings/python/tests/cindex/test_linkage.py b/clang/test/bindings/python/tests/cindex/test_linkage.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_linkage.py
rename to clang/test/bindings/python/tests/cindex/test_linkage.py
diff --git a/clang/bindings/python/tests/cindex/test_location.py b/clang/test/bindings/python/tests/cindex/test_location.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_location.py
rename to clang/test/bindings/python/tests/cindex/test_location.py
diff --git a/clang/bindings/python/tests/cindex/test_rewrite.py b/clang/test/bindings/python/tests/cindex/test_rewrite.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_rewrite.py
rename to clang/test/bindings/python/tests/cindex/test_rewrite.py
diff --git a/clang/bindings/python/tests/cindex/test_source_range.py b/clang/test/bindings/python/tests/cindex/test_source_range.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_source_range.py
rename to clang/test/bindings/python/tests/cindex/test_source_range.py
diff --git a/clang/bindings/python/tests/cindex/test_tls_kind.py b/clang/test/bindings/python/tests/cindex/test_tls_kind.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_tls_kind.py
rename to clang/test/bindings/python/tests/cindex/test_tls_kind.py
diff --git a/clang/bindings/python/tests/cindex/test_token_kind.py b/clang/test/bindings/python/tests/cindex/test_token_kind.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_token_kind.py
rename to clang/test/bindings/python/tests/cindex/test_token_kind.py
diff --git a/clang/bindings/python/tests/cindex/test_tokens.py b/clang/test/bindings/python/tests/cindex/test_tokens.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_tokens.py
rename to clang/test/bindings/python/tests/cindex/test_tokens.py
diff --git a/clang/bindings/python/tests/cindex/test_translation_unit.py b/clang/test/bindings/python/tests/cindex/test_translation_unit.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_translation_unit.py
rename to clang/test/bindings/python/tests/cindex/test_translation_unit.py
diff --git a/clang/bindings/python/tests/cindex/test_type.py b/clang/test/bindings/python/tests/cindex/test_type.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_type.py
rename to clang/test/bindings/python/tests/cindex/test_type.py
diff --git a/clang/bindings/python/tests/cindex/util.py b/clang/test/bindings/python/tests/cindex/util.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/util.py
rename to clang/test/bindings/python/tests/cindex/util.py
|
|
@llvm/pr-subscribers-github-workflow Author: Jannick Kremer (DeinAlptraum) ChangesAs discussed in PR #142353, the current testsuite of the
All these issues can be avoided by integrating the Python bindings tests with
Tested on Full diff: https://github.com/llvm/llvm-project/pull/146486.diff 40 Files Affected:
diff --git a/.github/workflows/libclang-python-tests.yml b/.github/workflows/libclang-python-tests.yml
index 50ef4acf2feb1..43b50cec61716 100644
--- a/.github/workflows/libclang-python-tests.yml
+++ b/.github/workflows/libclang-python-tests.yml
@@ -10,15 +10,15 @@ on:
- 'main'
paths:
- 'clang/bindings/python/**'
+ - 'clang/test/bindings/python/**'
- 'clang/tools/libclang/**'
- - 'clang/CMakeList.txt'
- '.github/workflows/libclang-python-tests.yml'
- '.github/workflows/llvm-project-tests.yml'
pull_request:
paths:
- 'clang/bindings/python/**'
+ - 'clang/test/bindings/python/**'
- 'clang/tools/libclang/**'
- - 'clang/CMakeList.txt'
- '.github/workflows/libclang-python-tests.yml'
- '.github/workflows/llvm-project-tests.yml'
diff --git a/clang/CMakeLists.txt b/clang/CMakeLists.txt
index 94607a8e8473c..72386132bce9f 100644
--- a/clang/CMakeLists.txt
+++ b/clang/CMakeLists.txt
@@ -530,7 +530,6 @@ if( CLANG_INCLUDE_TESTS )
clang_unit_site_config=${CMAKE_CURRENT_BINARY_DIR}/test/Unit/lit.site.cfg
)
add_subdirectory(test)
- add_subdirectory(bindings/python/tests)
if(CLANG_BUILT_STANDALONE)
umbrella_lit_testsuite_end(check-all)
diff --git a/clang/bindings/python/tests/CMakeLists.txt b/clang/bindings/python/tests/CMakeLists.txt
deleted file mode 100644
index a0ddabc21bb41..0000000000000
--- a/clang/bindings/python/tests/CMakeLists.txt
+++ /dev/null
@@ -1,66 +0,0 @@
-# Test target to run Python test suite from main build.
-
-# Avoid configurations including '-include' from interfering with
-# our tests by setting CLANG_NO_DEFAULT_CONFIG.
-add_custom_target(check-clang-python
- COMMAND ${CMAKE_COMMAND} -E env
- CLANG_NO_DEFAULT_CONFIG=1
- CLANG_LIBRARY_PATH=$<TARGET_FILE_DIR:libclang>
- "${Python3_EXECUTABLE}" -m unittest discover
- DEPENDS libclang
- WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/..)
-
-set(RUN_PYTHON_TESTS TRUE)
-set_target_properties(check-clang-python PROPERTIES FOLDER "Clang/Tests")
-
-# Tests require libclang.so which is only built with LLVM_ENABLE_PIC=ON
-if(NOT LLVM_ENABLE_PIC)
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-# Do not try to run if libclang was built with sanitizers because
-# the sanitizer library will likely be loaded too late to perform
-# interception and will then fail.
-# We could use LD_PRELOAD/DYLD_INSERT_LIBRARIES but this isn't
-# portable so its easier just to not run the tests when building
-# with ASan.
-if(NOT LLVM_USE_SANITIZER STREQUAL "")
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-# Tests fail on Windows, and need someone knowledgeable to fix.
-# It's not clear whether it's a test or a valid binding problem.
-if(WIN32)
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-# The Python FFI interface is broken on AIX: https://bugs.python.org/issue38628.
-if(${CMAKE_SYSTEM_NAME} MATCHES "AIX")
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-# AArch64, Hexagon, and Sparc have known test failures that need to be
-# addressed.
-# SystemZ has broken Python/FFI interface:
-# https://reviews.llvm.org/D52840#1265716
-if(${LLVM_NATIVE_ARCH} MATCHES "^(AArch64|Hexagon|Sparc|SystemZ)$")
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-# Tests will fail if cross-compiling for a different target, as tests will try
-# to use the host Python3_EXECUTABLE and make FFI calls to functions in target
-# libraries.
-if(CMAKE_CROSSCOMPILING)
- # FIXME: Consider a solution that allows better control over these tests in
- # a crosscompiling scenario. e.g. registering them with lit to allow them to
- # be explicitly skipped via appropriate LIT_ARGS, or adding a mechanism to
- # allow specifying a python interpreter compiled for the target that could
- # be executed using qemu-user.
- message(WARNING "check-clang-python not added to check-all as these tests fail in a cross-build setup")
- set(RUN_PYTHON_TESTS FALSE)
-endif()
-
-if(RUN_PYTHON_TESTS)
- set_property(GLOBAL APPEND PROPERTY
- LLVM_ALL_ADDITIONAL_TEST_TARGETS check-clang-python)
-endif()
diff --git a/clang/test/CMakeLists.txt b/clang/test/CMakeLists.txt
index e5b4a3bb84645..a9b88d4f4d9ec 100644
--- a/clang/test/CMakeLists.txt
+++ b/clang/test/CMakeLists.txt
@@ -205,6 +205,17 @@ add_custom_target(clang-test)
add_dependencies(clang-test check-clang)
set_target_properties(clang-test PROPERTIES FOLDER "Clang/Tests")
+# Allow running Clang Python binding tests separately from CI.
+add_lit_testsuite(check-clang-python "Running the Clang Python tests"
+ ${CMAKE_CURRENT_BINARY_DIR}
+ #LIT ${LLVM_LIT}
+ PARAMS ${CLANG_TEST_PARAMS}
+ DEPENDS ${CLANG_TEST_DEPS}
+ ARGS ${CLANG_TEST_EXTRA_ARGS} --filter=bindings.sh
+ # Avoid running tests twice.
+ EXCLUDE_FROM_CHECK_ALL
+ )
+
# FIXME: This logic can be removed once all buildbots have moved
# debuginfo-test from clang/test to llvm/projects or monorepo.
if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/debuginfo-tests)
diff --git a/clang/test/bindings/python/bindings.sh b/clang/test/bindings/python/bindings.sh
new file mode 100755
index 0000000000000..c5cc6e4e25ffe
--- /dev/null
+++ b/clang/test/bindings/python/bindings.sh
@@ -0,0 +1,33 @@
+#!/bin/sh
+
+# UNSUPPORTED: !libclang-loadable
+
+# Tests fail on Windows, and need someone knowledgeable to fix.
+# It's not clear whether it's a test or a valid binding problem.
+# XFAIL: target={{.*windows.*}}
+
+# The Python FFI interface is broken on AIX: https://bugs.python.org/issue38628.
+# XFAIL: target={{.*-aix.*}}
+
+# Hexagon has known test failures that need to be addressed.
+# SystemZ has broken Python/FFI interface:
+# https://reviews.llvm.org/D52840#1265716
+# XFAIL: target={{(hexagon|s390x)-.*}}
+# python SEGVs on Linux/sparc64 when loading libclang.so. Seems to be an FFI
+# issue, too.
+# XFAIL: target={{sparc.*-.*-linux.*}}
+
+# Tests will fail if cross-compiling for a different target, as tests will try
+# to use the host Python3_EXECUTABLE and make FFI calls to functions in target
+# libraries.
+#
+# FIXME: Consider a solution that allows better control over these tests in
+# a crosscompiling scenario. e.g. registering them with lit to allow them to
+# be explicitly skipped via appropriate LIT_ARGS, or adding a mechanism to
+# allow specifying a python interpreter compiled for the target that could
+# be executed using qemu-user.
+# REQUIRES: native
+
+# RUN: env PYTHONPATH=%S/../../../bindings/python \
+# RUN: CLANG_LIBRARY_PATH=%libdir \
+# RUN: %python -m unittest discover -s %S/tests
diff --git a/clang/test/bindings/python/lit.local.cfg b/clang/test/bindings/python/lit.local.cfg
new file mode 100644
index 0000000000000..cc3bdf8ba97d4
--- /dev/null
+++ b/clang/test/bindings/python/lit.local.cfg
@@ -0,0 +1,41 @@
+def is_libclang_loadable():
+ # Do not try to run if libclang was built with sanitizers because
+ # the sanitizer library will likely be loaded too late to perform
+ # interception and will then fail.
+ # We could use LD_PRELOAD/DYLD_INSERT_LIBRARIES but this isn't
+ # portable so its easier just to not run the tests when building
+ # with ASan.
+ if config.llvm_use_sanitizer != "":
+ return False
+ try:
+ sys.path.append(os.path.join(config.clang_src_dir, "bindings/python"))
+ from clang.cindex import Config
+ conf = Config()
+ Config.set_library_path(config.clang_lib_dir)
+ conf.lib
+ return True
+ except Exception as e:
+ # Expected failure modes are considered benign when nothing can be
+ # done about them.
+ #
+ # Cannot load a 32-bit libclang.so into a 64-bit python.
+ if "wrong ELF class: ELFCLASS32" in str(e):
+ return False
+ # If libclang.so is missing, it must have been disabled intentionally,
+ # e.g. by building with LLVM_ENABLE_PIC=OFF.
+ elif "No such file or directory" in str(e):
+ return False
+ # Unexpected failure modes need to be investigated to either fix an
+ # underlying bug or accept the failure, so return True. This causes
+ # tests to run and FAIL, drawing developer attention.
+ else:
+ print("warning: unhandled failure in is_libclang_loadable: "
+ + str(e), file=sys.stderr)
+ return True
+
+if is_libclang_loadable():
+ config.available_features.add("libclang-loadable")
+
+config.substitutions.append(('%libdir', config.clang_lib_dir))
+
+config.suffixes = ['.sh']
diff --git a/clang/bindings/python/tests/__init__.py b/clang/test/bindings/python/tests/__init__.py
similarity index 100%
rename from clang/bindings/python/tests/__init__.py
rename to clang/test/bindings/python/tests/__init__.py
diff --git a/clang/bindings/python/tests/cindex/INPUTS/a.inc b/clang/test/bindings/python/tests/cindex/INPUTS/a.inc
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/a.inc
rename to clang/test/bindings/python/tests/cindex/INPUTS/a.inc
diff --git a/clang/bindings/python/tests/cindex/INPUTS/b.inc b/clang/test/bindings/python/tests/cindex/INPUTS/b.inc
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/b.inc
rename to clang/test/bindings/python/tests/cindex/INPUTS/b.inc
diff --git a/clang/bindings/python/tests/cindex/INPUTS/compile_commands.json b/clang/test/bindings/python/tests/cindex/INPUTS/compile_commands.json
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/compile_commands.json
rename to clang/test/bindings/python/tests/cindex/INPUTS/compile_commands.json
diff --git a/clang/bindings/python/tests/cindex/INPUTS/header1.h b/clang/test/bindings/python/tests/cindex/INPUTS/header1.h
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/header1.h
rename to clang/test/bindings/python/tests/cindex/INPUTS/header1.h
diff --git a/clang/bindings/python/tests/cindex/INPUTS/header2.h b/clang/test/bindings/python/tests/cindex/INPUTS/header2.h
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/header2.h
rename to clang/test/bindings/python/tests/cindex/INPUTS/header2.h
diff --git a/clang/bindings/python/tests/cindex/INPUTS/header3.h b/clang/test/bindings/python/tests/cindex/INPUTS/header3.h
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/header3.h
rename to clang/test/bindings/python/tests/cindex/INPUTS/header3.h
diff --git a/clang/bindings/python/tests/cindex/INPUTS/hello.cpp b/clang/test/bindings/python/tests/cindex/INPUTS/hello.cpp
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/hello.cpp
rename to clang/test/bindings/python/tests/cindex/INPUTS/hello.cpp
diff --git a/clang/bindings/python/tests/cindex/INPUTS/include.cpp b/clang/test/bindings/python/tests/cindex/INPUTS/include.cpp
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/include.cpp
rename to clang/test/bindings/python/tests/cindex/INPUTS/include.cpp
diff --git a/clang/bindings/python/tests/cindex/INPUTS/parse_arguments.c b/clang/test/bindings/python/tests/cindex/INPUTS/parse_arguments.c
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/parse_arguments.c
rename to clang/test/bindings/python/tests/cindex/INPUTS/parse_arguments.c
diff --git a/clang/bindings/python/tests/cindex/INPUTS/testfile.c b/clang/test/bindings/python/tests/cindex/INPUTS/testfile.c
similarity index 100%
rename from clang/bindings/python/tests/cindex/INPUTS/testfile.c
rename to clang/test/bindings/python/tests/cindex/INPUTS/testfile.c
diff --git a/clang/bindings/python/tests/cindex/__init__.py b/clang/test/bindings/python/tests/cindex/__init__.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/__init__.py
rename to clang/test/bindings/python/tests/cindex/__init__.py
diff --git a/clang/bindings/python/tests/cindex/test_access_specifiers.py b/clang/test/bindings/python/tests/cindex/test_access_specifiers.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_access_specifiers.py
rename to clang/test/bindings/python/tests/cindex/test_access_specifiers.py
diff --git a/clang/bindings/python/tests/cindex/test_cdb.py b/clang/test/bindings/python/tests/cindex/test_cdb.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_cdb.py
rename to clang/test/bindings/python/tests/cindex/test_cdb.py
diff --git a/clang/bindings/python/tests/cindex/test_code_completion.py b/clang/test/bindings/python/tests/cindex/test_code_completion.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_code_completion.py
rename to clang/test/bindings/python/tests/cindex/test_code_completion.py
diff --git a/clang/bindings/python/tests/cindex/test_comment.py b/clang/test/bindings/python/tests/cindex/test_comment.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_comment.py
rename to clang/test/bindings/python/tests/cindex/test_comment.py
diff --git a/clang/bindings/python/tests/cindex/test_cursor.py b/clang/test/bindings/python/tests/cindex/test_cursor.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_cursor.py
rename to clang/test/bindings/python/tests/cindex/test_cursor.py
diff --git a/clang/bindings/python/tests/cindex/test_cursor_kind.py b/clang/test/bindings/python/tests/cindex/test_cursor_kind.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_cursor_kind.py
rename to clang/test/bindings/python/tests/cindex/test_cursor_kind.py
diff --git a/clang/bindings/python/tests/cindex/test_diagnostics.py b/clang/test/bindings/python/tests/cindex/test_diagnostics.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_diagnostics.py
rename to clang/test/bindings/python/tests/cindex/test_diagnostics.py
diff --git a/clang/bindings/python/tests/cindex/test_enums.py b/clang/test/bindings/python/tests/cindex/test_enums.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_enums.py
rename to clang/test/bindings/python/tests/cindex/test_enums.py
diff --git a/clang/bindings/python/tests/cindex/test_exception_specification_kind.py b/clang/test/bindings/python/tests/cindex/test_exception_specification_kind.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_exception_specification_kind.py
rename to clang/test/bindings/python/tests/cindex/test_exception_specification_kind.py
diff --git a/clang/bindings/python/tests/cindex/test_file.py b/clang/test/bindings/python/tests/cindex/test_file.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_file.py
rename to clang/test/bindings/python/tests/cindex/test_file.py
diff --git a/clang/bindings/python/tests/cindex/test_index.py b/clang/test/bindings/python/tests/cindex/test_index.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_index.py
rename to clang/test/bindings/python/tests/cindex/test_index.py
diff --git a/clang/bindings/python/tests/cindex/test_lib.py b/clang/test/bindings/python/tests/cindex/test_lib.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_lib.py
rename to clang/test/bindings/python/tests/cindex/test_lib.py
diff --git a/clang/bindings/python/tests/cindex/test_linkage.py b/clang/test/bindings/python/tests/cindex/test_linkage.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_linkage.py
rename to clang/test/bindings/python/tests/cindex/test_linkage.py
diff --git a/clang/bindings/python/tests/cindex/test_location.py b/clang/test/bindings/python/tests/cindex/test_location.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_location.py
rename to clang/test/bindings/python/tests/cindex/test_location.py
diff --git a/clang/bindings/python/tests/cindex/test_rewrite.py b/clang/test/bindings/python/tests/cindex/test_rewrite.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_rewrite.py
rename to clang/test/bindings/python/tests/cindex/test_rewrite.py
diff --git a/clang/bindings/python/tests/cindex/test_source_range.py b/clang/test/bindings/python/tests/cindex/test_source_range.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_source_range.py
rename to clang/test/bindings/python/tests/cindex/test_source_range.py
diff --git a/clang/bindings/python/tests/cindex/test_tls_kind.py b/clang/test/bindings/python/tests/cindex/test_tls_kind.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_tls_kind.py
rename to clang/test/bindings/python/tests/cindex/test_tls_kind.py
diff --git a/clang/bindings/python/tests/cindex/test_token_kind.py b/clang/test/bindings/python/tests/cindex/test_token_kind.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_token_kind.py
rename to clang/test/bindings/python/tests/cindex/test_token_kind.py
diff --git a/clang/bindings/python/tests/cindex/test_tokens.py b/clang/test/bindings/python/tests/cindex/test_tokens.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_tokens.py
rename to clang/test/bindings/python/tests/cindex/test_tokens.py
diff --git a/clang/bindings/python/tests/cindex/test_translation_unit.py b/clang/test/bindings/python/tests/cindex/test_translation_unit.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_translation_unit.py
rename to clang/test/bindings/python/tests/cindex/test_translation_unit.py
diff --git a/clang/bindings/python/tests/cindex/test_type.py b/clang/test/bindings/python/tests/cindex/test_type.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/test_type.py
rename to clang/test/bindings/python/tests/cindex/test_type.py
diff --git a/clang/bindings/python/tests/cindex/util.py b/clang/test/bindings/python/tests/cindex/util.py
similarity index 100%
rename from clang/bindings/python/tests/cindex/util.py
rename to clang/test/bindings/python/tests/cindex/util.py
|
|
This is the next attempt at #145855. @dyung if you have any other feedback or suggestions, I'd love to hear it! (I am not at all proficient with lit unfortunately, but would hate to let rorth's work go to waste here) |
|
My only suggestion would be to try testing the cross build configurations. At least the ones on the bots I manage don't require anything special and can use a general ubuntu linux and Windows machines to reproduce the build/test that the bot performs. That way you could verify the fix beforehand. |
|
Thanks - just did that locally and this indeed passes with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussed in PR #142353, the current testsuite of the
clangPython bindings has several issues:libclang.socannot be loaded intopythonto run the testsuite, the wholeninja check-allaborts.lit-based tests, rendering them almost invisible.RUN_PYTHON_TESTS) intests/CMakeLists.txt, which again doesn't show up in the test results.All these issues can be avoided by integrating the Python bindings tests with
lit, which is what this patch does:clang/test/bindings/python/bindings.shand is run bylit.clang/bindings/python/testsdirectory (minus the now-subperfluousCMakeLists.txt) is moved into the same directory.libclangis loadable (originally from PR [clang][python][test] Check if libclang.so is loadable #142353) is now handled via a newlitfeature,libclang-loadable.XFAILs as appropriate. This isn't complete and not completely tested yet.Tested on
sparc-sun-solaris2.11,sparcv9-sun-solaris2.11,i386-pc-solaris2.11,amd64-pc-solaris2.11,i686-pc-linux-gnu, andx86_64-pc-linux-gnu.