diff --git a/docs/cloud/faq.mdx b/docs/cloud/faq.mdx index 459e0b7788..959f24d3e8 100644 --- a/docs/cloud/faq.mdx +++ b/docs/cloud/faq.mdx @@ -320,10 +320,6 @@ Wind back the clock to any point in an application's test execution and directly width="80%" /> -### What types of testing can I debug with Test Replay? - -Anything you can do with Cypress - E2E, Component, API, etc. - ### How is Cypress Test Replay different from other "replay" services? There is an important distinction between _session_ replay services (LogRocket, FullStory, DataDog, etc) and Cypress Test Replay. In session replay, user actions are captured as the application is used and delivered back in a replay-able format, usually video or stitched DOM snapshots. These are valuable tools for gathering product insight such as user behavior, per session. @@ -336,39 +332,6 @@ Nothing. It's free and included in all Cypress Cloud plans. -### Does the replay reflect the real time it took for the test to run? - -Yes! The test is captured in Cypress Cloud as it executed in your CI run. That means the replay will show things like [slow typing](/api/cypress-api/keyboard-api#Slow-down-typing-by-increasing-the-keystroke-delay) or any [`cy.wait()`](/api/commands/wait) included in your test. - -### Are there performance implications when using Test Replay? - -Expect an even performance exchange with Test Replay vs. video. There is more activity to capture and replay tests but without the overhead of recording, compressing, and storing video assets. `video` capture is set to `false` by default starting in Cypress `v13`. See the full [v13 changelog](/app/references/changelog#13-0-0). - -### Is everything captured and replayed in Test Replay? - -Our aim is to create an impactful debugging experience that covers the most ground for users. We will continue to expand support. Currently Test Replay does not support the following: - -#### DOM elements - -- Video ([see issue](https://github.com/cypress-io/cypress/issues/29903)) -- Audio ([see issue](https://github.com/cypress-io/cypress/issues/29905)) -- Objects that do not have `type="image/svg+xml"` ([see issue](https://github.com/cypress-io/cypress/issues/29904)) -- Shadow DOM with `slotAssignment="manual"` ([see issue](https://github.com/cypress-io/cypress/issues/29107)) -- Shadow DOM canvas ([see issue](https://github.com/cypress-io/cypress/issues/30438)) - -#### Browsers - -- WebKit browser (Safari) ([see issue](https://github.com/cypress-io/cypress/issues/28895)) -- Firefox browser ([see issue](https://github.com/cypress-io/cypress/issues/28894)) - -#### Other - -- Server sent events ([see issue](https://github.com/cypress-io/cypress/issues/29908)) -- Web sockets ([see issue](https://github.com/cypress-io/cypress/issues/29907)) -- localStorage & sessionStorage ([see issue](https://github.com/cypress-io/cypress/issues/29909)) -- Cookies ([see issue](https://github.com/cypress-io/cypress/issues/29910)) -- Console props of Cypress commands outside of `cy.request()` ([see issue](https://github.com/cypress-io/cypress/issues/29911)) - ### Can I use Test Replay for tests recorded in different browsers? Test Replay leverages [Chrome DevTools Protocol(CDP)](https://chromedevtools.github.io/devtools-protocol/), so currently supports Chromium-based browsers (Chrome, Edge, and Electron) only. diff --git a/docs/cloud/features/test-replay.mdx b/docs/cloud/features/test-replay.mdx index 852ea8176f..4f9a7b333c 100644 --- a/docs/cloud/features/test-replay.mdx +++ b/docs/cloud/features/test-replay.mdx @@ -12,16 +12,6 @@ sidebar_position: 40 ::: -## What Is Test Replay? - -:::caution - - Test Replay is available when recording tests -to Cypress Cloud using Cypress `v13` and up. See the [Cypress v13 changelog](/app/references/changelog#13-0-0). - -Test Replay currently only supports Chromium-based browsers (Chrome, Edge, Electron). - -::: Test Replay in Cypress Cloud allows developers to accurately and efficiently debug failed and flaky [continuous integration](/app/continuous-integration/overview) test runs. It captures every test run detail, enabling developers to replay it and inspect the DOM, network requests, console logs, JavaScript errors, and element rendering as they happened during `cypress run`. @@ -30,34 +20,54 @@ Test Replay in Cypress Cloud allows developers to accurately and efficiently deb title="Test Replay Product Demo" /> -### Test Replay vs Screenshots and Videos +## Key Benefits -Before Cypress `v13`, test failures in CI have historically been captured through screenshots, videos, and stack trace outputs, but these artifacts provide limited information. Developers often need more context to understand the root cause of failures, requiring them to reproduce the issue locally, which may not always be possible due to environmental or network conditions. This knowledge gap between test failures and debugging capabilities leads to increased debugging time and potentially deploying unresolved bugs to production. +Test failures in CI have historically been captured through screenshots, videos, and stack trace outputs, +but these artifacts provide limited information. Developers often need more context to understand the +root cause of failures, requiring them to reproduce the issue locally, which may not always +be possible due to environmental or network conditions. +This knowledge gap between test failures and debugging capabilities leads to increased debugging +time and potentially deploying unresolved bugs to production. -### How Does Test Replay Advance Your Workflow? + Time travel to the exact moment of test failure during runs that happened during Continuous Integration -Test Replay complements the existing visual artifacts provided by Cypress Cloud. It eliminates the need for local test replication by enabling developers to time-travel to the exact moment of test failure. Developers gain access to previously inaccessible details and feedback loops, enabling faster and more accurate debugging, and instilling confidence in handling test failures in CI. Replacing video recordings with Test Replay provides a superior debugging experience that is sharable and supports team collaboration with efficient workflows. + Inspect the DOM, network requests, console logs, JavaScript errors, and element rendering -## Getting Started + Debug failed and flaky test runs with more context and accuracy -:::info + Eliminate the need to reproduce test failures locally - Test Replay is available to **all** Cypress Cloud plans + Shareable and collaborative debugging experience + +Test Replay complements the existing visual artifacts provided by Cypress Cloud. It eliminates the need for +local test replication by enabling developers to time-travel to the exact moment of test failure. +Developers gain access to previously inaccessible details and feedback loops, enabling +faster and more accurate debugging, and instilling confidence in handling test failures in CI. +Replacing video recordings with Test Replay provides a superior debugging experience that is +sharable and supports team collaboration with efficient workflows. + +## Get Started + +:::success + +Test Replay is available to **all** Cypress Cloud plans at no additional cost, subject to usage limits. ::: -Test Replay is automatically enabled when updating to `v13` of Cypress. If desired, an opt-out is provided through project-level settings. Learn about what this means for capturing video of test failure in Cypress Cloud under ["See also"](#See-also) below. -All you need to do after [recording test runs](/cloud/get-started/setup) is log in to your Cypress Cloud account and leverage these new debugging capabilities. No additional software, configuration or changes to the existing test recording process is required. For example, you do not need to edit your [Cypress configuration](/app/references/configuration) file or pass any flags or new commands when recording via [the command line](/app/references/command-line#Commands) in order to enable or use Test Replay. +All you need to do after [recording test runs](/cloud/get-started/setup) is log in to your Cypress Cloud +account and leverage Test Replay's new debugging capabilities. No additional configuration +or changes to the existing tests are required. + -## Accessing Test Replay +### Access Test Replay Accessing the replay of your test is as simple as reviewing any other test artifact in Cypress Cloud. -1. Hover on a [test for review](/cloud/features/recorded-runs#Tests-for-Review) under the Latest Runs Overview. +1. Hover on a [test for review](/cloud/features/recorded-runs#Tests-for-Review) under a run's **Overview**. -## Debugging With Test Replay +### Debug With Test Replay Both options above display the Test Replay interface, which we'll unpack from top to bottom in more detail below. -### Run header +#### Run header The left side of the header displays the Cypress Cloud test run ID number (#), with the Git branch name along with the spec, platform and browser information. Hovering on this exposes additional test meta data. @@ -85,7 +95,7 @@ The left side of the header displays the Cypress Cloud test run ID number (#), w alt="Test Replay header" /> -### Command Log +#### Command Log The left side of the screen displays the [command log](/app/core-concepts/open-mode#Command-Log) just like in the Cypress app, where you can step through, inspect and [time travel](/app/core-concepts/open-mode#Time-traveling) debug your test. @@ -95,7 +105,7 @@ The left side of the screen displays the [command log](/app/core-concepts/open-m width="60%" /> -### Developer Tools +#### Developer Tools Dive into network-level traffic, console events and inspect the application under test just as you can in the browser. @@ -105,7 +115,7 @@ Dive into network-level traffic, console events and inspect the application unde width="60%" /> -### Replay Controls +#### Replay Controls Toggle each test attempt, play back errors at a variety of speeds or scrub through the captured error by dragging the playhead along the timeline. @@ -114,6 +124,104 @@ Toggle each test attempt, play back errors at a variety of speeds or scrub throu alt="Test Replay scrubber controls" /> +## How Test Replay Works + +Our aim is to create an impactful debugging experience that covers the most ground for users. +Cypress captures a wide range of data during end-to-end and component test runs. +This data is then stored in Cypress Cloud and replayed for debugging through Test Replay. + +### What is captured + +- Test commands +- Network requests +- Console logs +- JavaScript errors +- Element rendering +- DOM mutations +- CSS styles +- SVG elements +- Iframes +- Shadow DOM +- Canvas elements (v13.5.0+, please contact us at support@cypress.io to enable capturing canvas elements) + +### What is not captured + +Currently Test Replay does not support the following: + +- Tests recorded in Cypress App below v13. +- WebKit browser (Safari) ([see issue](https://github.com/cypress-io/cypress/issues/28895)) +- Firefox browser ([see issue](https://github.com/cypress-io/cypress/issues/28894)) +- Video ([see issue](https://github.com/cypress-io/cypress/issues/29903)) +- Audio ([see issue](https://github.com/cypress-io/cypress/issues/29905)) +- Objects that do not have `type="image/svg+xml"` ([see issue](https://github.com/cypress-io/cypress/issues/29904)) +- Shadow DOM with `slotAssignment="manual"` ([see issue](https://github.com/cypress-io/cypress/issues/29107)) +- Shadow DOM canvas ([see issue](https://github.com/cypress-io/cypress/issues/30438)) +- Server sent events ([see issue](https://github.com/cypress-io/cypress/issues/29908)) +- Web sockets ([see issue](https://github.com/cypress-io/cypress/issues/29907)) +- localStorage & sessionStorage ([see issue](https://github.com/cypress-io/cypress/issues/29909)) +- Cookies ([see issue](https://github.com/cypress-io/cypress/issues/29910)) +- Console props of Cypress commands outside of `cy.request()` ([see issue](https://github.com/cypress-io/cypress/issues/29911)) + +### Performance + +#### Recording + +Capturing this additional data can use additional CPU while recording, but you can +expect an even performance exchange with Test Replay versus video recording. +There is more activity to capture and replay tests but without the overhead of recording, +compressing, and storing video assets. We recommend disabling video recording when using +Test Replay to optimize performance even further. + +You can inspect the size of the Test Replay data that is uploaded to the Cloud +and the time spent uploading the Test Replay data by inspecting the standard output of the test run. + +```shell + (Uploading Cloud Artifacts) + + - Video - Nothing to upload + - Screenshot - Nothing to upload + - Test Replay - 298 kB + + Uploading Cloud Artifacts: . + + (Uploaded Cloud Artifacts) + + - Test Replay - Done Uploading 298 kB in 633.40ms 1/1 +``` + +#### Canvas Elements + +Capturing canvas elements can be resource-intensive. If you have a large number of canvas elements or large canvas elements +in your application, you may experience performance issues when capturing Test Replay, so we recommend +monitoring the performance of your tests after enabling capturing of canvas elements and raising any issues with us. + +Capturing canvas is disabled by default. To enable capturing canvas elements, please contact us at support@cypress.io. + +### Runner UI + +The Cypress App []"Runner UI"](/app/core-concepts/open-mode#Test-Runner) is made up of a +[command log](/app/core-concepts/open-mode#Command-Log) which displays every command +executed in your spec. The UI also includes the URL bar, browser selector, and the viewport size +control above the application under test. + +The Runner UI is visible during `cypress open` mode, but is also rendered +([headless](/app/references/command-line#Options)-ly) during `cypress run` so that it displays +within captured screenshots or videos, to assist in debugging CI failures. + +But rendering the Runner UI while running tests via `cypress run` can have performance implications, +resulting in a slower test runtime, especially on lower resourced machines. + +With Test Replay, Cypress Cloud regenerates the entire UI for viewing the run and debugging CI failures. +Additionally, displaying the Cypress command log in videos and screenshots is less valuable for users +using Test Replay. + +So when Test Replay is enabled, the Runner UI does not render during `cypress run` + +If you'd like to render the Runner UI, you can turn on this behavior by passing `--runner-ui` as a CLI flag. +You may encounter a slower performance when turning this on, especially in lower resourced machines. +Learn more about the `--runner-ui` [command line option](/app/references/command-line#cypress-run-runner-ui). + + ## Opt-out of Test Replay :::caution @@ -124,7 +232,7 @@ Test Replays, including the test data leveraged within your tests, are visible t ::: -If you do not want to capture test data for replay and debugging purposes, simply "shut off" Test Replay in project settings. +If you do not want to capture test data for replay and debugging purposes, you can "shut off" Test Replay in project settings.