Test passes but log shows a failed assert #9155
Comments
|
Thanks for the simple example. I can see that the async nature of the app code will cause this situation where Cypress passes the test, and then updates the last assertion to fail. Reproducible example
<body>
<ul>
<li>foo</li>
<li>foo</li>
<li>bar</li>
</ul>
<script>
setTimeout(() => {
document
.querySelector('li:last-child')
.remove()
}, 1000)
</script>
</body>`spec.js it('all li elements have text "foo"', () => {
cy.visit('index.html')
cy.get('li').should(($lis) => {
$lis.each((_, element) => {
expect(element.textContent).to.equal('foo')
})
})
})WorkaroundThe ideal way to write this test would be to assert first that the removed element doesn't exist first: it('all li elements have text "foo"', () => {
cy.visit('index.html')
cy.get('li').should('have.length', 2).and(($lis) => {
$lis.each((_, element) => {
expect(element.textContent).to.equal('foo')
})
})
}) |
cypress-bot
bot
added
the
stage: needs investigating
jennifer-shehane
added
type: unexpected behavior
|
Yeah, this is unexpected. Once the test completes, the assertions should not change their state. In this particular case I would make sure the expected number of assertions has passed as described in https://www.cypress.io/blog/2020/01/16/when-can-the-test-stop/ |
|
Note that the real-world testcase that lead to this report types text into a filter field and checks if the filtered list only has elements containing the filter text. The number of results (and hence the number of assertions) is not known beforehand, so the proposed workarounds won't work |
|
Sure they will. It is just that the test has no idea when the application updates itself and when the test should finish. What if the app updates itself after 10 seconds? Or 15 seconds? So you have to observe something for the test to "know" when it can check the filtered results. Worst comes to worst, you can always add |
|
I settled for this workaround: it('all li elements have text "foo"', () => {
cy.visit('index.html')
cy.get('li').should(($lis) => {
let nonFoo = []
$lis.each((_, element) => {
if (element.textContent !== 'foo') nonFoo.push(element)
})
expect(nonFoo, 'non-foo lis').to.be.empty
})
})Though I agree with @bahmutov that in general, it would be better to wait until the DOM is stable. My current project is effectively black box testing, however, so I prefer a solution that only looks at the html |
|
This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided. |
|
Verified the behavior is still occuring and updated the test code |
|
@JessefSpecialisterren thank you for confirming this issue is still happening. I will update the ticket. |
|
I replicated this issue in the latest version of Cypress and will forward this ticket to the appropriate team. They will evaluate the priority of this ticket and consider their capacity to pick it up. Please note that this does not guarantee that this issue will be resolved. |
chrisbreiding
added
E2E
Current behavior
Under certain circumstances, Cypress shows a test as passed even though the log contains a failed assert:
Desired behavior
In this particular case, the failed assert should not be shown. It was not executed in the invocation of the
shouldcallback that made the test succeed, because the third li element had been removed by that time.Test code to reproduce
index.html
spec.cy.js
Versions
Cypress: 12.12.0
Node: v16.16.0
Browser: Chrome 113
OS: Windows 10.0.19044.2965
The text was updated successfully, but these errors were encountered: