Skip to content

No test left behind: stop skipping tests #26088

Open

Description

Ideas for better catching behavior changes in the product code. Specifically:

  • Detect when a negative case stops failing
  • Detect when a negative case starts failing in a different way

Fundamental approach: don't skip tests.

In NorthwindAggregateOperatorsQueryTests, we had:

  • Negative cases that were no longer failing
  • Negative cases that were skipped for all providers, but worked on some. For example:
    • Failed on relational, but passed on in-memory
    • Failed on relational, but passed on Cosmos
    • Failed on SQL Server, but passed on SQLite
  • Negative cases that failed in different ways on different providers

Specifics:

  • If a test throws, catch the exception
    • Where feasible, also validate the exception message or error number
  • Always call base where possible, rather than repeating the query in an overridden test
  • Add a standard comment where we have a bug or enhancement tracking the issue. For example:
    • // Contains over subquery. Issue #17246.
  • Always have an AssertSql call in Cosmos and SQL Server tests
    • Where we expect a provider-specific class to verify SQL, then add a test that checks all test methods are overridden.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions