feat: add testIsolation option and support describe-only overrides #23040

emilyrohrbough · 2022-08-01T13:44:57Z

Closes Add new testIsolation configuration option #22279
add testIsolation option - when experimentalSessionAndOrigin=false, the default value is legacy. When experimentalSessionAndOrigin=true, this the default value is strict.
- legacy - this is the current test isolation behavior in cypress - clock, viewport, cookies, local storage, etc continue to be reset before each tests
- strict - all rests from legacy plus clearing the page between each test by visiting about:blank. This ensures app state is reset/clean for each test.
add support for config override levels - this allows each option to determine what override levels are supported. The override levels are:
any - can be set from any level which includes:
- code - when either the support file or spec file is loaded and config is altered with Cypress.config()
- test:before:run - config altered during event handler with Cypress.config()
- test:before:run:async - config altered during event handler with Cypress.config()
- suite - configuration provided as suite override describe('...', { ...configOverides }, ()=>{})
- test- configuration provided as test override it('...', { ...configOverides }, ()=>{})
- runtime - configuration altered from runnable (hook or test execution) using Cypress.config()
suite - configuration can only be set from the suite level
never - configuration is read-only and cannot be altered without restarting Cypress

User facing changelog

Ability new configuration option, called testIsolation, has been added to allow you to configured the test isolation mode. This configuration option can be enabled by setting the experimentalSessionAndOrigin flag to true in the Cypress config and defaults to strict mode. Read more about test isolation in Cypress to learn more.

Additional details

discovered test:before:run does not handle thrown errors correctly ([driver] errors thrown from test:before:run are incorrectly handled #23039) so config validation is currently skipped for this hook.

How has the user experience changed?

Improved error messaging
Ability to set the testIsolation intensity when experiementalSessionAndOrigin=true. this preps for when this goes GA and users don't want to update tests.

PR Tasks

Have tests been added/updated?
Has the original issue (or this PR, if no issue exists) been tagged with a release in ZenHub? (user-facing changes only)
Has a PR for user-facing changes been opened in cypress-documentation? add docs on new testIsolation configuration option. cypress-documentation#4634
Have API changes been updated in the type definitions?

…fig override levels during run-time

cypress-bot · 2022-08-01T13:45:00Z

Thanks for taking the time to open a PR!

Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

packages/config/src/options.ts

cypress · 2022-08-01T15:02:53Z

Test summary

4961 • 0 • 59 • 0 • 1

Run details


Project	cypress
Status	Passed
Commit	`2291a29`
Started	Aug 12, 2022 4:46 PM
Ended	Aug 12, 2022 4:59 PM
Duration	12:29 💡
OS	Linux Debian - 11.3
Browser	Electron 102

View run in Cypress Dashboard ➡️

Flakiness

	cypress/e2e/cypress/proxy-logging.cy.ts		1
1	Proxy Logging > request logging > xhr log has response body/status code when xhr response is logged second

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

emilyrohrbough · 2022-08-01T17:51:45Z

packages/config/src/options.ts

    requireRestartOnChange: 'server',
  }, {
    name: 'excludeSpecPattern',
    defaultValue: (options: Record<string, any> = {}) => options.testingType === 'component' ? ['**/__snapshots__/*', '**/__image_snapshots__/*'] : '*.hot-update.js',
    validation: validate.isStringOrArrayOfStrings,
-    canUpdateDuringTestTime: true,


looking at this....why can this update at test time?

I don't think it should be able to - probably incorrect.

thats funny, what would overriding this even do at test time? Probably nothing.

emilyrohrbough · 2022-08-01T17:52:34Z

packages/config/src/options.ts

  }, {
    name: 'numTestsKeptInMemory',
    defaultValue: 50,
    validation: validate.isNumber,
-    canUpdateDuringTestTime: true,
+    overrideLevels: ALL_OVERRIDE_LEVELS,


Same with this opt...this shouldn't be able to update. 😕

I think we do something in our support file to set this dynamically if in run mode to test snapshots. We probably could configure it differently

Maybe there's not a great reason to change this during a test, but it's still technically possible, right? I think that's the main consideration, whether you can instead of whether you should.

emilyrohrbough · 2022-08-01T17:52:35Z

packages/config/src/options.ts

  }, {
    name: 'port',
    defaultValue: null,
    validation: validate.isNumber,
-    canUpdateDuringTestTime: true,
+    overrideLevels: ALL_OVERRIDE_LEVELS,


Same with this opt...this shouldn't be able to update. 😕

emilyrohrbough · 2022-08-01T17:52:54Z

packages/config/src/options.ts

  }, {
    name: 'projectId',
    defaultValue: null,
    validation: validate.isString,
-    canUpdateDuringTestTime: true,
+    overrideLevels: ALL_OVERRIDE_LEVELS,


lmiller1990

Left some misc comments/questions.

cli/types/cypress.d.ts

packages/config/src/browser.ts

lmiller1990 · 2022-08-02T01:15:40Z

packages/server/lib/config.ts

@@ -215,6 +214,19 @@ export function mergeDefaults (
    throw makeConfigError(errors.get(err, ...args))
  }, testingType)

+  // TODO: testIsolation should equal 'strict' by default when experimentalSessionAndOrigin=true


Will we eventually apply strict to everything? My understanding is we have strict for cy.origin to make it more reliable (clean slate). Since this seems ideal for all tests, is the intention to make this the default in Cypress 11?

Great question! When Origin & Sessions go GA, we will default to strict. Long-term this would be the only "mode", however we've received some concerns/pushback in user-feedback outreach that enforcing strict mode without an option to remove the "clear page" before each test would cause some large re-writes for some and is might be a show-stopper for future upgrades. Overall I think strict is what users should want to leverage, but there are some scenarios where this would be nice to turn off to allow more modular tests if the tests aren't altering page/app state.

cy.origin is tied to and dependent on the sessions feature because it enables clearing cookies/localstorage/etc for all origins, whereas historically it's only been cleared for the primary origin. Test isolation is a separate feature that, as you intuited, should benefit all e2e tests regardless of whether or not they use cy.origin.

lmiller1990 · 2022-08-02T01:17:09Z

system-tests/test/testConfigOverrides_spec.ts

@@ -33,6 +33,23 @@ describe('testConfigOverrides', () => {
    },
  })

+  // FIXME: bug in runner causes browser to hang in run mode when test:before:run throws an exception


Nice, the tests throughout this PR look very thorough 💯

Is there an issue for following up on this?

I think this one #23039

packages/config/test/index.spec.ts

Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com>

…ess-io/cypress into test-isolation-config-opt-2

chrisbreiding · 2022-08-11T17:28:30Z

packages/config/src/options.ts

@@ -27,31 +29,35 @@ export type BreakingOptionErrorKey =
  | 'RENAMED_CONFIG_OPTION'
  | 'TEST_FILES_RENAMED'

-type TestingType = 'e2e' | 'component'
+// The test-time override levels
+export const ALL_OVERRIDE_LEVELS = ['code', 'event', 'suite', 'test', 'runtime'] as const


I think it would be useful to have the comment above this explain what each level means

I'm having trouble discerning the practical difference between 'code' and 'runtime'. It seems that 'code' is when setting config at the top/global level and 'runtime' is when it's inside a test, but it seems like the result is the same either way, since they're handled the same way during validation. The only difference seems to be when crafting the error message, but it seems like that's just a nicety that could apply to the 'code' error as well. Is there a case where a config value would only be overrideable at 'code' level and not 'runtime' or vice versa? If not, it seems like they could be merged into the same value.

So the code level is either when the support file or spec file is loaded before mocha has processed the spec to determine the suites/tests/runnable in the code. Originally I had named it "supportFile" but found it could be when either support file or spec file was loaded into the AUT.

I do think from a messaging standpoint the delineation is helpful since runTime is when a mocha runnable itself is executing, but maybe it's not? 🤷🏻‍♀️ are you confused on the difference between the two and/or are you wanting "slimmer" override levels and messaging?

I understand the difference now - thanks for the clarification. If there's a use-case for having them separate, that's fine, but I would suggest renaming them to something like runtime-global and runtime-test. However, if there's not a use-case, I'd say YAGNI 😅 It's an internal API, so we can always change it later if a use-case emerges.

chrisbreiding · 2022-08-11T17:45:10Z

packages/config/src/browser.ts

@@ -172,16 +193,24 @@ export const validateNoBreakingTestingTypeConfig = (cfg: any, testingType: keyof
  return validateNoBreakingOptions(options, cfg, onWarning, onErr, testingType)
 }

-export const validateNoReadOnlyConfig = (config: any, onErr: (property: string) => void) => {
-  let errProperty
+export const validateOverridableAtTestTest = (config: any, overrideLevel: Exclude<OverrideLevel, 'never'>, onErr: (result: InvalidTestOverrideResult) => void) => {


Suggested change

export const validateOverridableAtTestTest = (config: any, overrideLevel: Exclude<OverrideLevel, 'never'>, onErr: (result: InvalidTestOverrideResult) => void) => {

export const validateOverridableAtTestTime = (config: any, overrideLevel: Exclude<OverrideLevel, 'never'>, onErr: (result: InvalidTestOverrideResult) => void) => {

Or validateOverridableAtRuntime?

oh yes. good call out. I wanted to call this testTime, since that's what it in in my head, but I went with runtime to align with the resolve level of 'runtime' we use to pass to the cloud

chrisbreiding · 2022-08-11T17:57:56Z

packages/config/src/options.ts

@@ -27,31 +29,35 @@ export type BreakingOptionErrorKey =
  | 'RENAMED_CONFIG_OPTION'
  | 'TEST_FILES_RENAMED'

-type TestingType = 'e2e' | 'component'
+// The test-time override levels
+export const ALL_OVERRIDE_LEVELS = ['code', 'event', 'suite', 'test', 'runtime'] as const


I'm having trouble discerning the practical difference between 'code' and 'runtime'. It seems that 'code' is when setting config at the top/global level and 'runtime' is when it's inside a test, but it seems like the result is the same either way, since they're handled the same way during validation. The only difference seems to be when crafting the error message, but it seems like that's just a nicety that could apply to the 'code' error as well. Is there a case where a config value would only be overrideable at 'code' level and not 'runtime' or vice versa? If not, it seems like they could be merged into the same value.

cli/types/cypress.d.ts

mjhenkes · 2022-08-12T12:58:29Z

packages/config/src/options.ts

  }, {
    name: 'arch',
    defaultValue: () => os.arch(),
    validation: validate.isString,
-    canUpdateDuringTestTime: false,


i like that this defaults to false now, much cleaner

chrisbreiding · 2022-08-12T13:10:43Z

packages/driver/src/util/config.ts

+// The run-time override levels (listed in order applied):
+//   fileLoad -  config override via Cypress.config() when either loading the supportFile or specFile in the
+//                 browser (this is before mocha as process the spec
+//   suite    - config override via describe('', {...}, () => {})
+//   test     - config override via it('', {...}, () => {})
+//   event    - config override via Cypress.config() in test:before:runner or test:before:runner:async event
+//   runtime  - config override via Cypress.config() when the test callback is executed


This comment should be updated

I left it intentionally to document the order these overrides are applied since the aren't documented and not easy to find in the code base without stepping through the runner code.

Co-authored-by: Matt Henkes <mjhenkes@gmail.com>

packages/server/lib/util/spec_writer.ts

Co-authored-by: Matt Henkes <mjhenkes@gmail.com>

AtofStryker · 2022-08-12T15:52:20Z

system-tests/test/testConfigOverrides_spec.ts

@@ -61,7 +61,7 @@ describe('testConfigOverrides', () => {
      spec: 'testConfigOverrides/invalid.js',
      snapshot: true,
      browser: browserList,
-      expectedExitCode: 8,
+      expectedExitCode: 14,


dare I ask what exit code 14 is?

Yup! there are 14 failing tests in the invalid.js file

emilyrohrbough added 7 commits July 29, 2022 08:46

feat: add testIsolation config option and add support for varying con…

28380a5

…fig override levels during run-time

clean up based on tests

8677ec7

fix unit tests & types

174b50c

system tests

3ca878d

and fix

819f71f

fix tests & align runTime with runtime value passed to the dashboard

41fb01b

fix snapshots

6226550

emilyrohrbough requested review from a team as code owners August 1, 2022 13:44

emilyrohrbough requested review from AtofStryker, chrisbreiding, mjhenkes and mschile and removed request for a team August 1, 2022 13:45

emilyrohrbough self-assigned this Aug 1, 2022

emilyrohrbough commented Aug 1, 2022

View reviewed changes

packages/config/src/options.ts Outdated Show resolved Hide resolved

emilyrohrbough added 2 commits August 1, 2022 08:53

add types

7b62033

Merge branch 'develop' into test-isolation-config-opt-2

68fd5b1

fix

15ec8d8

emilyrohrbough commented Aug 1, 2022

View reviewed changes

emilyrohrbough requested a review from debrisapron August 1, 2022 19:10

lmiller1990 reviewed Aug 2, 2022

View reviewed changes

emilyrohrbough and others added 2 commits August 2, 2022 09:05

Update cli/types/cypress.d.ts

dec1da8

Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com>

Merge branch 'develop' into test-isolation-config-opt-2

9de9028

emilyrohrbough and others added 6 commits August 11, 2022 09:51

fix failing test

8446984

fix snapshot

3248797

overruled....changing lax back to legacy

073de5a

Merge branch 'develop' into test-isolation-config-opt-2

623ef3b

fix

28a7140

Merge branch 'test-isolation-config-opt-2' of https://github.com/cypr…

fb14105

…ess-io/cypress into test-isolation-config-opt-2

chrisbreiding reviewed Aug 11, 2022

View reviewed changes

chrisbreiding self-requested a review August 11, 2022 17:28

chrisbreiding requested changes Aug 11, 2022

View reviewed changes

emilyrohrbough added 2 commits August 11, 2022 14:25

change code -> fileLoad

2b94621

okayyyy another round of feedback applied

586ca74

mjhenkes reviewed Aug 12, 2022

View reviewed changes

cli/types/cypress.d.ts Show resolved Hide resolved

mjhenkes reviewed Aug 12, 2022

View reviewed changes

cli/types/cypress.d.ts Outdated Show resolved Hide resolved

mjhenkes reviewed Aug 12, 2022

View reviewed changes

chrisbreiding requested changes Aug 12, 2022

View reviewed changes

Update cli/types/cypress.d.ts

9449f40

Co-authored-by: Matt Henkes <mjhenkes@gmail.com>

emilyrohrbough requested review from chrisbreiding and mjhenkes August 12, 2022 14:06

chrisbreiding approved these changes Aug 12, 2022

View reviewed changes

Merge branch 'develop' into test-isolation-config-opt-2

368ff71

mjhenkes reviewed Aug 12, 2022

View reviewed changes

packages/server/lib/util/spec_writer.ts Outdated Show resolved Hide resolved

mjhenkes approved these changes Aug 12, 2022

View reviewed changes

emilyrohrbough and others added 2 commits August 12, 2022 10:23

Update packages/server/lib/util/spec_writer.ts

f34fdb3

Co-authored-by: Matt Henkes <mjhenkes@gmail.com>

Merge branch 'develop' into test-isolation-config-opt-2

2291a29

AtofStryker approved these changes Aug 12, 2022

View reviewed changes

emilyrohrbough changed the title ~~feat: add testIsolation option and support config override levels~~ feat: add testIsolation option and support describe-only overrides Aug 12, 2022

emilyrohrbough merged commit 0729a68 into develop Aug 12, 2022

emilyrohrbough deleted the test-isolation-config-opt-2 branch August 12, 2022 16:56

cypress-bot bot mentioned this pull request Aug 12, 2022

Add new testIsolation configuration option #22279

Closed

richard-cox mentioned this pull request Aug 24, 2023

fix support links tests rancher/dashboard#9421

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add testIsolation option and support describe-only overrides #23040

feat: add testIsolation option and support describe-only overrides #23040

emilyrohrbough commented Aug 1, 2022 •

edited

Loading

cypress-bot bot commented Aug 1, 2022

cypress bot commented Aug 1, 2022 •

edited

Loading

emilyrohrbough Aug 1, 2022

lmiller1990 Aug 2, 2022

mjhenkes Aug 12, 2022 •

edited

Loading

emilyrohrbough Aug 1, 2022

AtofStryker Aug 2, 2022

chrisbreiding Aug 3, 2022

emilyrohrbough Aug 1, 2022

emilyrohrbough Aug 1, 2022

lmiller1990 left a comment

lmiller1990 Aug 2, 2022

emilyrohrbough Aug 2, 2022

chrisbreiding Aug 3, 2022

lmiller1990 Aug 2, 2022

chrisbreiding Aug 3, 2022

lmiller1990 Aug 3, 2022

chrisbreiding Aug 11, 2022

chrisbreiding Aug 11, 2022

emilyrohrbough Aug 11, 2022 •

edited

Loading

chrisbreiding Aug 11, 2022

chrisbreiding Aug 11, 2022

emilyrohrbough Aug 11, 2022

chrisbreiding Aug 11, 2022

mjhenkes Aug 12, 2022

chrisbreiding Aug 12, 2022

emilyrohrbough Aug 12, 2022 •

edited

Loading

AtofStryker Aug 12, 2022

emilyrohrbough Aug 12, 2022 •

edited

Loading

	export const validateOverridableAtTestTest = (config: any, overrideLevel: Exclude<OverrideLevel, 'never'>, onErr: (result: InvalidTestOverrideResult) => void) => {
	export const validateOverridableAtTestTime = (config: any, overrideLevel: Exclude<OverrideLevel, 'never'>, onErr: (result: InvalidTestOverrideResult) => void) => {

feat: add testIsolation option and support describe-only overrides #23040

feat: add testIsolation option and support describe-only overrides #23040

Conversation

emilyrohrbough commented Aug 1, 2022 • edited Loading

User facing changelog

Additional details

How has the user experience changed?

PR Tasks

cypress-bot bot commented Aug 1, 2022

cypress bot commented Aug 1, 2022 • edited Loading

Test summary

Run details

Flakiness

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjhenkes Aug 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmiller1990 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilyrohrbough Aug 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilyrohrbough Aug 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilyrohrbough Aug 12, 2022 • edited Loading

Choose a reason for hiding this comment

emilyrohrbough commented Aug 1, 2022 •

edited

Loading

cypress bot commented Aug 1, 2022 •

edited

Loading

mjhenkes Aug 12, 2022 •

edited

Loading

emilyrohrbough Aug 11, 2022 •

edited

Loading

emilyrohrbough Aug 12, 2022 •

edited

Loading

emilyrohrbough Aug 12, 2022 •

edited

Loading