Integration tests

Integration and end to end tests are located in the integration-tests directory. These tests use Puppeteer to interact with Terra UI in a browser.

Running the tests

Run tests (from within the integration-tests folder) against a local UI:

cd integration-tests
yarn test-local

(alternatively, run within the integration-tests workspace with yarn workspace terra-integration-tests run test-local)

To rerun failed tests:

yarn test-local --onlyFailures

To run a specific test, pass the test name as an argument (for example: yarn test-local workspace-dashboard).

The test-local script wraps test and automatically reads service account keys from Google Secret Manager. Alternatively, those keys can be provided as environment variables and yarn test run directly.

TERRA_SA_KEY=$(...) \
LYLE_SA_KEY=$(...) \
yarn test

By default, tests run locally will be run against dev. To run against a different environment (for example, staging), the UI's configuration must match the environment used for testing. For example:

cp config/staging.json public/config.json
yarn start

cd integration-tests
ENVIRONMENT=staging yarn test-local

Configuration

Azure Configuration

To run Azure integration tests, in addition to the below configurations, the terra-envs file must include a parameter billingProjectAzure under the appropriate environment's key. This Azure billing project must already exist on the target environment, and the test user must have a reader or owner role on the billing project.

The following environment variables are parsed by the tests:

• Service account credentials (required):

  • TERRA_SA_KEY: service account key for creating access tokens for test user.
  • LYLE_SA_KEY: service account key to access Lyle.

• ENVIRONMENT: Terra UI instance to test. Options: dev, alpha, perf, staging

  • Default dev, which sets:
    • BILLING_PROJECT: used for workspace creation.
      • Default saturn-integration-test-dev
    • TEST_URL: URL for the ui.
      • Default https://bvdp-saturn-dev.appspot.com
    • WORKFLOW_NAME: workflow/method used for tests. Expects published config named [name]-configured.
      • Default echo_to_file

• HEADLESS: By default, the browser will run in headless mode. Set this to false to show a browser window.

• SLOW: Set to true for the test to pause briefly in between executing commands.

• LOG_DIR: Directory where to save test logs.

  • Default /tmp/test-results

• LYLE_URL: URL for the service account allocator.

  • Default https://terra-lyle.appspot.com

• SCREENSHOT_DIR: without this, screenshots won't be saved on test failure.

• TERRA_USER_EMAIL: account that already has access to BILLING_PROJECT

  • Default Scarlett.Flowerpicker@test.firecloud.org

Debugging

To open a debug port into the jest tests run the following in the integration-tests folder

TERRA_SA_KEY=$(...) \
LYLE_SA_KEY=$(...) \
node --inspect-brk node_modules/.bin/jest [test name] --runInBand

To disable headless mode while debugging

HEADLESS=false \
TERRA_SA_KEY=$(...) \
LYLE_SA_KEY=$(...) \
node --inspect-brk node_modules/.bin/jest [test name] --runInBand

Using your IDE you can connect to the debug port and set breakpoints. More info here.

Detecting Flakiness

You can run:

yarn test-flakes [test name]

By default this will run your test 100 times and display the stack trace of every failure encountered.

You can tweak this with the following settings:

Setting	Default	Description
RUNS	100	The number of test runs you want to execute (Default: 100)
CONCURRENCY	10	The size of the browser pool / the number of simultaneous tests to run. 25 seems to be the upper limit for a mac with an M1 max chip
CLUSTER_TIMEOUT_MINUTES	120	The number of minutes before the overall test times out. 120 is overkill. If you want to run hundreds of iterations, it is best to overestimate this value

Creating a new environment (e.g., dev, alpha, etc)

To set up an environment, run node scripts/initializeEnvironment.js.

Creating a test

1. Update slack-notify-channels.json in slack/ directory: Add new test name and Slack channel for test result notification.