Error Improvements: Part 3

[chrisbreiding]

Error Improvements (Part 2) is soon to be released and provides a foundation for many more improvements that can made on top of it. The following is a list of further work that can be done. It is in no particular order and any given item's inclusion is not a guarantee it will be done. The items need some discussion and prioritization. Once that occurs, items can be broken out into their own issues, so smaller PRs can be made addressing them.

• BUG: Errors that are currently not covered by tests and may not extra work to support source map/code frames:

  • Errors in hooks
  • Returning a promise from a test and rejecting it (cy.js fail isn't called)
  • jQuery sizzle errors (invalid selector)

• FEATURE: Add the code frame to the error message/stack, so it appears in stdout and other reporters Add code frame for error messages in stdout #7819

• FEATURE: Show diffs in reporter (Show diffs for assertions logged in browser console #3129 & .should('deep.equal') does not show a full diff comparison on failures #4084)

• FEATURE: Code frames and stack translation relies on inline source maps. If we detect there is no source map, show a message about their benefits and points to directions on how to enable them.

• FEATURE: Utilize the stack capturing and source mapping for purposes other than errors

  • Each test in the reporter can have a link that can open the user's IDE to that test's definition

• FEATURE: Should we show the stack trace by default in reporter?

  • It's actually useful and not overly verbose with internal lines
  • Maybe show 3/4 lines, with a 'Show more' button

• FEATURE: Some error messages have multiple error messages. Currently they just display:

  • 'Learn more' links next to each other in the reporter
  • The URLs one after another in stdout
  • We should differentiate them somehow
    • Give them different labels
    • Each may be associated with different parts of the message, so they should be integrated into the message better
    • Maybe the message should have placeholders for them that get replaced differently depending on interactive/non-interactive modes

• PERF: Profile how much time Bluebird spends when we enable longStackTraces (cypress.js) and maybe turn it always off or always on

• FEATURE: Splice out cypress internals when there's a user error (particularly Scenario 2, when error is in a command callback)

  • Alternatively, display them different in the reporter (e.g. less opacity, collapsed with option to open)

• BUG: Chai assertion error messages use quotes instead of backticks

  • "expected 'actual' to equal 'expected'" should be "expected actual to equal expected"

• FEATURE: Send new error properties to the dashboard

  • Currently we only send the message and stack
  • Send the code frame, parsed stack, and any other useful properties
  • Include information about the command, type of assertion error, etc
  • Enable opening in IDE from the dashboard
  • Jeff is working on a proposal for what the dashboard would find useful

• FEATURE: Capture and display info like current URL with an error (See this comment)

• FEATURE: Explore opening stack lines in DevTools

  • Either do it instead of opening in IDE (maybe only in certain circumstances) or as an extra option

• FEATURE: Improve how errors are logged (use console.group)

• FEATURE: For File Opener Preference modal:

  • Add a learn more link (same as settings for it has in Desktop GUI)
  • 'Settings tab' should be clickable link to go to ‘Settings’ in the Desktop GUI

• FEATURE: Utilize source maps in user's app code

  • This is a technical challenge because we can't access the user's code as easily as the spec code
  • Would need to pull it out of the user's code or have the user opt in and provide a way for us to access it
  • Once we can, it enables:
    • Showing code frames for errors in app code
    • If visit synchronously fails in app code:
      • Code frame that points to cy.visit in spec code, as it is now
      • A 2nd code frame that points to the app code
      • This might apply to other commands as well, where the error ocurred in the user's app, but we can still associate it with a command (click, etc)

• BUG: When printing error to console, the stack lines show fully qualified URLs instead of source mapped URLs

• BUG: When importing/requiring a file outside the project root, the path is not correct (../foo flattens into webpack:///foo)

• Make individual lines in the code frame clickable (then open in IDE)

In progress/done

• Improve Mocha done errors (called twice, timed out, etc) - Issue
• Improve errors thrown in cy.on('fail') handlers - Issue | PR
• cy.task errors need the original error appended like other Node.js errors - Issue | PR

Related Issues

• Module API doesn’t fail the promise with Cypress error message Module API doesn't fail the promise with Cypress failure messages #3216
• Show warning message in command log Show warning messages in command log #4917
• Add warning to command log for 'uncaught exception' Issue Command Log 'warning' event during uncaught:exception #4933
• Add codeframe for error messages in headless mode Add code frame for error messages in stdout #7819

[Saibamen]

Code in headless moved to separate ticket: #7819