Add Cypress e2e test for Posit Workbench #2984

cgraham-rs · 2025-09-03T15:39:13Z

Intent

Add initial e2e support and test for Publisher deployments on Posit Workbench using Positron.

Fixes #2911

Type of Change

- Bug Fix
- New Feature
- Breaking Change
- Documentation
- Refactor
- Tooling

Approach

Leverage previous work that implemented manual Workbench testings in chore(workbench): enable Posit Workbench for local manual testing #2909
Leverage existing VS Code commands and workflows as much as possible
Enabled the following plugins:
- cypress-wait-until this was already in the package.json but not enabled
- cypress-network-idle extremely handy tool for pacing tests by waiting for specific network traffic to idle/stop
Add new Workbench-specific commands to a new workbench support file and name them as clearly as possible to reduce potential confusion
Added some new commands to CI to save Workbench related logs as artifacts which deviates from the current method of printing them to the CLI
Also deviated from the existing method of post-test DEBUG where we print out the installed extensions, etc. as most of that is captured during the Workbench extensions install command
In order to ensure a clean Workbench environment for each test leverage our justfile commands to stop, remove, and start a fresh Workbench container. This adds approximately 25 seconds to each test but is the only way I could find that would ensure a completely clean environment
- Will create a ticket for future work that can try maintaining test isolation by creating new users for each test, though this may be challenging since we're using the public Workbench container
Implement the following tests leveraged directly from test/e2e/tests/deployments.cy.js:
- static deployment test that I used to get us started and prove that all of this works locally and in CI
Updated CONTRIBUTING for all of the Workbench related changes in this and the previous PR

KNOWN ISSUES

The first test that runs, does the first visit to Workbench, then silently errors and restarts, then works fine after.
- I've chased this down to overriding the baseUrl in the spec, which is a valid thing to do. But when the baseUrl in Cypress config does not match when we do our first visit() Cypress silently restarts the whole spec again. If they do match this restart is not observed. Publisher is currently using an older Cypress version so maybe a version upgrade would correct this behavior, though I see nothing obvious mentioned in the Cypress changelog. We do have a paid contract with Cypress and they are responsive if we can provide a reproducible issue.
Positron session is successfully started though it takes ~60s. This seems related to running Workbench in Cypress as the Positron sessions start quickly in my local browser from the same Workbench container. This occurs when running via Cypress locally or in CI.

User Impact

None

Automated Tests

New Workbench-specific Cypress e2e spec.

Directions for Reviewers

Follow the existing local just dev workflow and ensure you can run the Workbench spec
Verify the Workbench spec ran and passed in CI
Verify the new Workbench-specific command names make sense and are clear without having to dive deeply into the code

Checklist

I have updated CHANGELOG.md to cover notable changes.

…container readiness

…d documented

…Workbench()

This reverts commit 4b54c55.

This reverts commit 65154c4.

This reverts commit 0398b8e.

cgraham-rs · 2025-12-01T21:13:10Z

.github/workflows/e2e.yaml

  cypress:
    name: e2e tests (cypress)
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-latest-8x


I don't have strong notes indicating why this change was made, but minimally to get Workbench to start quicker. We can explore reducing this to a 4x runner or back down to the standard runner, and remove any related container debug output.

agreed it might be worth a try to drop it down and see how it does (or if there's any negative impact).

Tests run fine with the standard runner, though slightly slower. Will leave it at standard so we are not needlessly consuming the larger runners.

Runner E2E Job Time Difference

ubuntu-latest-8x (8 cores) ~11m 38s Baseline

ubuntu-latest (2 cores) ~13m 14s +1m 36s slower (+13.7%)

And finally discovered one likely reason the 8x runner was chosen.

failed to register layer: write /usr/lib/rstudio-server/bin/quarto/share/deno_std/cache/gen/https/jsr.io/766911eda918ff2de41436f0cd1bd07e1f75dc019f40fdb27f776b8d6c7b3239.js: no space left on device

Have added a script we use in other repos that can tidy the runners to ensure adequate disk space without requiring a larger runner.

Before
Used: 54G/72G (75% full)
After
Used: 38G/72G (53% full)

.github/workflows/e2e.yaml

test/e2e/support/sequences.js

test/e2e/workbench-entrypoint.sh

cgraham-rs · 2025-12-01T23:14:09Z

Current failure with the Workbench test.

…te script

cgraham-rs · 2025-12-02T21:22:51Z

Current failure with the Workbench test.

And finally fixed with some retooling of setting permissions for the rstudio Workbench user with the content-workspace folder

tyatposit

A bit of a rough session startup time length, but overall looks good to me and passing locally. Just 1 issue i ran into building and that possible issue with clicking the input box selection. Plus a few minor suggestions.

test/e2e/docker-compose.yml

tyatposit · 2025-12-04T18:24:10Z

.github/workflows/e2e.yaml

  cypress:
    name: e2e tests (cypress)
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-latest-8x


agreed it might be worth a try to drop it down and see how it does (or if there's any negative impact).

tyatposit · 2025-12-04T18:29:50Z

test/e2e/support/workbench.js

+  cy.log("Installing Publisher extension in Workbench");
+  cy.exec(`just install-workbench-extension release`, {
+    failOnNonZeroExit: false,
+    timeout: 30_000,


suggestion: could potentially extract these out into variables with naming that helps explain the need for the waits. might also help with future updates if that ever happens though i think this is the only timeout that is repeated. 🤷‍♂️

timeout: 30_000 is repeated three times in this file but they're used for very different functions so I'm not confident the same value will continue to be identical in all cases forever.

tyatposit · 2025-12-04T18:32:23Z

test/e2e/support/workbench.js

+  cy.log("Starting Workbench Positron session");
+
+  // Start a Positron session
+  // TODO remove this workaround for "All types of sessions are disabled" error after Workbench 2025.12.0 is released


if there's a gh issue tracking this, would be helpful to link it here.

Do we have guidance for adding links to private repos here?

good point with this being a public project. I wasn't thinking about that. maybe a question for @jonkeane ?

test/scripts/install-workbench-extension.sh

tyatposit · 2025-12-04T18:51:48Z

test/e2e/support/workbench.js

+
+/**
+ * Checks if an interpreter is ready without attempting to start one
+ * @returns {Cypress.Chainable<boolean>} - Returns a chainable boolean indicating if interpreter is ready


liking these @returns
good idea and nice to have. 👍

test/e2e/support/workbench.js

…fallback

…Workbench extension

… ensure selection registers before clicking "OK"

…equate free space on the runner

cgraham-rs force-pushed the cgraham/e2e-workbench-positron-cypress branch from b998480 to e737281 Compare September 3, 2025 16:13

cgraham-rs force-pushed the cgraham/e2e-workbench-positron-cypress branch 2 times, most recently from 2394956 to 572f95f Compare September 15, 2025 14:49

cgraham-rs force-pushed the cgraham/e2e-workbench-positron-cypress branch 2 times, most recently from 374ae2c to 643d9f1 Compare September 26, 2025 17:53

cgraham-rs force-pushed the cgraham/e2e-workbench-positron-cypress branch from 643d9f1 to 3cbb1c2 Compare November 21, 2025 17:59

cgraham-rs added 24 commits December 1, 2025 14:28

Create GitHub App token as first step in job

0398b8e

Initial workbench docker container work

c528334

Streamline workbench justfile commands, use docker compose, wait for …

e6f79d9

…container readiness

Final justfile command polishing to get everything working locally an…

9612f61

…d documented

Remove .gitignore

f9ff59a

Mount content-workspace in Workbench container

185ae96

Update Workbench image with entrypoint script to disable workspace trust

3bd9028

Update .gitignore to exclude Workbench state directories

c130d7d

Fix workbench-entrypoint script

4139865

cypress-wait-until was installed but not imported

cbb1faa

Add more Workbench exclusions to .gitignore

d9e9646

Initial Workbench deployments test file

d94f625

Extend Workbench project name and fix project cleanup

23ead6c

Define Workbench base URL in config for reuse

d737f20

Move Workbench project prefex to cypress config

a4b2f33

Move Workbench commands to support file

551ca23

Build Workbench during build-images

eb024a3

Automatically install Publisher into Workbench during Cypress tests

471beda

Fail when running certain Workbench commands during cleanupAndRestart…

9064882

…Workbench()

Get Workbench tests working with latest UI changes

d584adb

Ensure Workbench is built and has license available in CI

908a58c

Update CONTRIBUTING with Workbench instructions

d8aff0b

Try mounting the dist directory and installing from there

aaaa306

Fix Positron folder permissions in Workbench for installing extensions

a443694

cgraham-rs added 4 commits December 1, 2025 15:09

Revert "Ensure cypress-timestamps is installed for e2e"

9c41f63

This reverts commit 4b54c55.

Revert "Implement cypress-timestamps"

a079534

This reverts commit 65154c4.

Fix deployment assertion

5d0fabb

Revert "Create GitHub App token as first step in job"

242a5b8

This reverts commit 0398b8e.

cgraham-rs commented Dec 1, 2025

View reviewed changes

.github/workflows/e2e.yaml Outdated Show resolved Hide resolved

cgraham-rs commented Dec 1, 2025

View reviewed changes

.github/workflows/e2e.yaml Outdated Show resolved Hide resolved

cgraham-rs commented Dec 1, 2025

View reviewed changes

test/e2e/support/sequences.js Outdated Show resolved Hide resolved

cgraham-rs commented Dec 1, 2025

View reviewed changes

test/e2e/workbench-entrypoint.sh Outdated Show resolved Hide resolved

cgraham-rs added 3 commits December 1, 2025 16:32

Ensure scripts fail properly

5620212

Revert only running Workbench e2e tests

3ad8240

Add Positron logs to artifacts

f55e134

cgraham-rs added 2 commits December 2, 2025 12:24

Refactor obtaining Positron logs from Workbench container into separa…

53f5722

…te script

Fix content-workspace permissions to allow rstudio user to write files

2af3527

cgraham-rs requested a review from tyatposit December 2, 2025 21:33

cgraham-rs marked this pull request as ready for review December 2, 2025 21:39

cgraham-rs requested review from dotNomad, marcosnav and sagerb as code owners December 2, 2025 21:39

tyatposit approved these changes Dec 4, 2025

View reviewed changes

cgraham-rs added 8 commits December 5, 2025 09:54

Red the expected Publisher extension version from package.json, with …

e239929

…fallback

Rename scripts and command to refer to Positron extension instead of …

70f5dde

…Workbench extension

Improve pacing when selecting project directory in workbench tests to…

abcad84

… ensure selection registers before clicking "OK"

Add platform to Workbench container definitions

a4eb7e2

Revert so runs-on: ubuntu-latest

00025d2

Remove unnecessary system and container debug

40e8877

Set deployment timeout back to 60 seconds

be80f2e

Add tidy-github-actions-runner.sh script to e2e workflow to ensure ad…

b1514e0

…equate free space on the runner

Runner	E2E Job Time	Difference
ubuntu-latest-8x (8 cores)	~11m 38s	Baseline
ubuntu-latest (2 cores)	~13m 14s	+1m 36s slower (+13.7%)

Add Cypress e2e test for Posit Workbench #2984

Are you sure you want to change the base?

Add Cypress e2e test for Posit Workbench #2984

Conversation

cgraham-rs commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

Type of Change

Approach

KNOWN ISSUES

User Impact

Automated Tests

Directions for Reviewers

Checklist

Uh oh!

cgraham-rs Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgraham-rs Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgraham-rs commented Dec 1, 2025

Uh oh!

cgraham-rs commented Dec 2, 2025

Uh oh!

tyatposit left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cgraham-rs commented Sep 3, 2025 •

edited

Loading

cgraham-rs Dec 1, 2025 •

edited

Loading

cgraham-rs Dec 5, 2025 •

edited

Loading