Ideas for better catching behavior changes in the product code. Specifically:

• Detect when a negative case stops failing
• Detect when a negative case starts failing in a different way

Fundamental approach: don't skip tests.

In NorthwindAggregateOperatorsQueryTests, we had:

• Negative cases that were no longer failing
• Negative cases that were skipped for all providers, but worked on some. For example:
  • Failed on relational, but passed on in-memory
  • Failed on relational, but passed on Cosmos
  • Failed on SQL Server, but passed on SQLite
• Negative cases that failed in different ways on different providers

Specifics:

• If a test throws, catch the exception
  • Where feasible, also validate the exception message or error number
• Always call base where possible, rather than repeating the query in an overridden test
• Add a standard comment where we have a bug or enhancement tracking the issue. For example:
  • // Contains over subquery. Issue #17246.
• Always have an AssertSql call in Cosmos and SQL Server tests
  • Where we expect a provider-specific class to verify SQL, then add a test that checks all test methods are overridden.