Test discovery should be fault tolerant and display all successfully imported files

[Conchylicultor]

Currently, pytest discovery fail for the entire project if a single import is missing on a single file:

E   ModuleNotFoundError: No module named 'xyz'
ERROR my_test.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
Exception: pytest discovery failed (exit code 2)

Only 1/600 file failed, but VS Code does not show the 599 tests which were correctly discovered.

Many other project have "optional" deps which add extra features. Those extra deps shouldn't be required when the full project do not need to be tested.
My project has many independent modules, each requiring a specific set of dependencies (pip install .[module0]). Contributors should not be forced to install the full dependencies of all modules / features when they only want to test a subpart.

Related issues:

Those issues are not duplicate but seems related:

• Perform test discovery per-file when a file is updated/added. #4586: Perform test discovery per-file when a file is updated/added: This issue focus about performances, as re-discovering the full set of files each time is slow.
• Test discovery fails if package is not installed #10469: Test discovery fails if package is not installed: This is about a specifc case where a specific import was not correctly detected.

[karthiknadig]

Were you able to get pytest or unittest to show tests via command line, even with import errors?

[Conchylicultor]

Yes, pytest --collect-only correctly detect the valid tests:

============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/epot/__A__/Code/datasets, configfile: pytest.ini
plugins: xdist-1.34.0, forked-1.3.0
collected 1685 items / 1 error / 1684 selected

<Package tensorflow_datasets>
  <Module import_test.py>
    <UnitTestCase ImportTest>
      <TestCaseFunction test_import>
      <TestCaseFunction test_session>
  <Module import_testing_test.py>
    <UnitTestCase ImportTestingTest>

...  # The 1684 detected tests are correctly displayed

=========================== short test summary info ============================
ERROR tensorflow_datasets/scripts/documentation/build_api_docs_test.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
======================== 5 warnings, 1 error in 35.20s =========================

Additionally, pytest --continue-on-collection-errors correctly execute the valid tests.

[Conchylicultor]

Shouldn't the info needed label be removed ?

[karthiknadig]

--continue-on-collection-errors makes test discovery bit better when there are issues with loading test files. However, adding support that will require a bit of work in the current test adapter. We have plans to update the adapter to be more robust. We also have to see if similar support can be provided on other supported test frameworks.