Meta issue: improving test runner

[sobolevn]

Right now there are several things I would love to see improved in our unit tests:

• Add ability to run a single .test file. For example, when I need to run check-namedtuple.test, I need to go to testcheck.py, comment all other tests cases and then run it. It is very annoying. Maybe there's a hidden way? If this is the case, then we need to update the docs. Now you can run separate mypy test modules 🎉 #11668
• Add duplicate case name detection. Right now we can have tests with non-unique names. For example:

  mypy/test-data/unit/check-newsemanal.test

  Line 2308 in cee5d3e

  [case testNewAnalyzerCastForward2]

  and

  mypy/test-data/unit/check-newsemanal.test

  Line 2317 in cee5d3e

  [case testNewAnalyzerCastForward2]

• Add 3.11-dev to test suite and allow it to fail, later we can disable failures, but for now it is too unstable: Add 3.11-dev as TODO item says #11677

I will send a couple of PRs with these features if no one has any rejections 🙂

[sobolevn]

My progress:

(.venv) ~/Desktop/mypy  master ✗                                                  5 ⚠️
» pytest mypy/test/testcheck.py --collect-only================================ test session starts =================================
platform darwin -- Python 3.10.0, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/sobolev/Desktop/mypy, configfile: pytest.ini
plugins: cov-2.12.1, xdist-1.34.0, forked-1.3.0
collected 68 items                                                                   

<Package test>
  <Module testcheck.py>
    <DataSuiteCollector TypeCheckSuite>
      <DataFileCollector check-basic.test>
        <DataDrivenTestCase testEmptyFile>
        <DataDrivenTestCase testAssignmentAndVarDef>
        <DataDrivenTestCase testConstructionAndAssignment>
        <DataDrivenTestCase testInheritInitFromObject>
        <DataDrivenTestCase testImplicitInheritInitFromObject>
        <DataDrivenTestCase testTooManyConstructorArgs>
        <DataDrivenTestCase testVarDefWithInit>
        <DataDrivenTestCase testInheritanceBasedSubtyping>
        <DataDrivenTestCase testDeclaredVariableInParentheses>
        <DataDrivenTestCase testIncompatibleAssignmentAmbiguousShortnames>
        <DataDrivenTestCase testFunction>
        <DataDrivenTestCase testNotCallable>
        <DataDrivenTestCase testSubtypeArgument>
        <DataDrivenTestCase testInvalidArgumentCount>
        <DataDrivenTestCase testMissingPositionalArguments>
        <DataDrivenTestCase testLocalVariables>
        <DataDrivenTestCase testLocalVariableScope>
        <DataDrivenTestCase testFunctionArguments>
        <DataDrivenTestCase testLocalVariableInitialization>
        <DataDrivenTestCase testVariableInitializationWithSubtype>
        <DataDrivenTestCase testInvalidReturn>
        <DataDrivenTestCase testTopLevelContextAndInvalidReturn>
        <DataDrivenTestCase testEmptyReturnInAnyTypedFunction>
        <DataDrivenTestCase testEmptyYieldInAnyTypedFunction>
        <DataDrivenTestCase testModule__name__>
        <DataDrivenTestCase testModule__doc__>
        <DataDrivenTestCase testModule__file__>
        <DataDrivenTestCase test__package__>
        <DataDrivenTestCase testLocalVariableShadowing>
        <DataDrivenTestCase testGlobalDefinedInBlockWithType>
        <DataDrivenTestCase testFunctionSignatureAsComment>
        <DataDrivenTestCase testMethodSignatureAsComment>
        <DataDrivenTestCase testTrailingCommaParsing>
        <DataDrivenTestCase testInitReturnTypeError>
        <DataDrivenTestCase testWritesCache>
        <DataDrivenTestCase testWritesCacheErrors>
        <DataDrivenTestCase testModuleAsTypeNoCrash>
        <DataDrivenTestCase testModuleAsTypeNoCrash2>
        <DataDrivenTestCase testPartialTypeComments>
        <DataDrivenTestCase testNoneHasBool>
        <DataDrivenTestCase testNoneHasBoolShowNoneErrorsFalse>
        <DataDrivenTestCase testAssignmentInvariantNoteForList>
        <DataDrivenTestCase testAssignmentInvariantNoteForDict>
        <DataDrivenTestCase testDistinctTypes>
        <DataDrivenTestCase testReturnAnyFromFunctionDeclaredToReturnObject>
        <DataDrivenTestCase testImportModuleAsClassMember>
        <DataDrivenTestCase testLocalImportModuleAsClassMember>
      <DataFileCollector check-union-or-syntax.test>
        <DataDrivenTestCase testUnionOrSyntaxWithTwoBuiltinsTypes>
        <DataDrivenTestCase testUnionOrSyntaxWithThreeBuiltinsTypes>
        <DataDrivenTestCase testUnionOrSyntaxWithTwoTypes>
        <DataDrivenTestCase testUnionOrSyntaxWithThreeTypes>
        <DataDrivenTestCase testUnionOrSyntaxWithLiteral>
        <DataDrivenTestCase testUnionOrSyntaxWithBadOperator>
        <DataDrivenTestCase testUnionOrSyntaxWithBadOperands>
        <DataDrivenTestCase testUnionOrSyntaxWithGenerics>
        <DataDrivenTestCase testUnionOrSyntaxWithQuotedFunctionTypes>
        <DataDrivenTestCase testUnionOrSyntaxWithQuotedVariableTypes>
        <DataDrivenTestCase testUnionOrSyntaxWithTypeAliasWorking>
        <DataDrivenTestCase testUnionOrSyntaxWithinRuntimeContextNotAllowed>
        <DataDrivenTestCase testUnionOrSyntaxWithinRuntimeContextNotAllowed2>
        <DataDrivenTestCase testUnionOrSyntaxInComment>
        <DataDrivenTestCase testUnionOrSyntaxFutureImport>
        <DataDrivenTestCase testUnionOrSyntaxMissingFutureImport>
        <DataDrivenTestCase testUnionOrSyntaxInStubFile>
        <DataDrivenTestCase testUnionOrSyntaxInMiscRuntimeContexts>
        <DataDrivenTestCase testUnionOrSyntaxRuntimeContextInStubFile>
        <DataDrivenTestCase testUnionOrSyntaxInIsinstance>
        <DataDrivenTestCase testUnionOrSyntaxInIsinstanceNotSupported>

============================ 68 tests collected in 0.24s =============================

Now works.

But, for some reason - this does not:

(.venv) ~/Desktop/mypy  master ✗                                                     
» pytest mypy/test/testcheck.py::check-basic.test --collect-only
================================ test session starts =================================
platform darwin -- Python 3.10.0, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/sobolev/Desktop/mypy, configfile: pytest.ini
plugins: cov-2.12.1, xdist-1.34.0, forked-1.3.0
collected 0 items                                                                    

============================ no tests collected in 0.14s =============================
ERROR: not found: /Users/sobolev/Desktop/mypy/mypy/test/testcheck.py::check-basic.test
(no name '/Users/sobolev/Desktop/mypy/mypy/test/testcheck.py::check-basic.test' in any of [<Module testcheck.py>])

🤔

[bluetech]

But, for some reason - this does not:

It needs to be

pytest mypy/test/testcheck.py::TypeCheckSuite::check-basic.test --collect-only

If you add -q to the --collect-only it gives a flat format which makes the nodeids more clear. I think it makes sense that the class is in the nodeid, but it should be possible to get rid of it if you want.

[hauntsaninja]

One other wish list item for me is to make it easy to exclude tests that depend on lxml. This would be useful for 3.11-dev / for future prerelease Pythons. It would also be useful for wheel builds, we currently do pytest -k 'not (reports.test or testreports)' when testing wheels to work around issues with lxml installation.

[ethanhs]

I believe all of the original work here that @sobolevn wanted to accomplish is done. For the lxml tests I opened #12813

So once that is merged I think we can close this.