Open
Description
Ideas for better catching behavior changes in the product code. Specifically:
- Detect when a negative case stops failing
- Detect when a negative case starts failing in a different way
Fundamental approach: don't skip tests.
In NorthwindAggregateOperatorsQueryTests, we had:
- Negative cases that were no longer failing
- Negative cases that were skipped for all providers, but worked on some. For example:
- Failed on relational, but passed on in-memory
- Failed on relational, but passed on Cosmos
- Failed on SQL Server, but passed on SQLite
- Negative cases that failed in different ways on different providers
Specifics:
- If a test throws, catch the exception
- Where feasible, also validate the exception message or error number
- Always call base where possible, rather than repeating the query in an overridden test
- Add a standard comment where we have a bug or enhancement tracking the issue. For example:
// Contains over subquery. Issue #17246.
- Always have an
AssertSql
call in Cosmos and SQL Server tests- Where we expect a provider-specific class to verify SQL, then add a test that checks all test methods are overridden.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment