Skip to content

Commit 8f3bfe0

Browse files
andrii-ipre-commit-ci[bot]3coins
authored
Add E2E tests (#350)
* add basic e2e testing setup * adjust execute test step name * test sidebar chat icon, add testing class * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add sidebar snapshot * test chat sidepanel, extend helper class * adjust welcome message test, add snapshot * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adjust naming * removeempty line * move ui-tests to packages/jupyter-ai/ * update e2e ci workflow for ui-tests folder move * update ui-tests folder location for yarn.lock hash * run lint locally * Add "Update Playwright Snapshots" CI workflow * change if clause * specify npm client * remove report and artifact specifiers * Update README.md to have correct commands and folders * update e2e/integration test README * Add Integration / E2E testing section to the docs * update wording of docs on snapshots * Ignore all non-linux snapshots * Update packages/jupyter-ai/ui-tests/README.md Co-authored-by: Piyush Jain <piyushjain@duck.com> * remove cd command that would return users back to root * remove cd ../../../ * Remove repeating setup instructions * Add suggestion to generate snapshots before the 1st run * remove unnecessary link anchor * remove rudimentary jlpm build --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Piyush Jain <piyushjain@duck.com>
1 parent 640f36c commit 8f3bfe0

File tree

13 files changed

+4649
-91
lines changed

13 files changed

+4649
-91
lines changed

‎.github/workflows/e2e-tests.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: E2E Tests
2+
3+
# suppress warning raised by https://github.com/jupyter/jupyter_core/pull/292
4+
env:
5+
JUPYTER_PLATFORM_DIRS: '1'
6+
7+
on:
8+
push:
9+
branches: main
10+
pull_request:
11+
branches: '*'
12+
13+
jobs:
14+
e2e-tests:
15+
name: Linux
16+
runs-on: ubuntu-latest
17+
18+
env:
19+
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/pw-browsers
20+
21+
steps:
22+
- name: Checkout
23+
uses: actions/checkout@v2
24+
25+
- name: Base Setup
26+
uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1
27+
28+
- name: Install extension dependencies and build the extension
29+
run: ./scripts/install.sh
30+
31+
- name: Install ui-tests dependencies
32+
working-directory: packages/jupyter-ai/ui-tests
33+
env:
34+
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1
35+
run: jlpm install
36+
37+
- name: Set up browser cache
38+
uses: actions/cache@v2
39+
with:
40+
path: |
41+
${{ github.workspace }}/pw-browsers
42+
key: ${{ runner.os }}-${{ hashFiles('packages/jupyter-ai/ui-tests/yarn.lock') }}
43+
44+
- name: Install browser
45+
working-directory: packages/jupyter-ai/ui-tests
46+
run: jlpm install-chromium
47+
48+
- name: Execute e2e tests
49+
working-directory: packages/jupyter-ai/ui-tests
50+
run: jlpm test
51+
52+
- name: Upload Playwright Test report
53+
if: always()
54+
uses: actions/upload-artifact@v2
55+
with:
56+
name: jupyter-ai-playwright-tests-linux
57+
path: |
58+
packages/jupyter-ai/ui-tests/test-results
59+
packages/jupyter-ai/ui-tests/playwright-report

‎.github/workflows/update-snapshots.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
name: Update Playwright Snapshots
2+
3+
on:
4+
issue_comment:
5+
types: [created, edited]
6+
workflow_dispatch:
7+
inputs:
8+
number:
9+
description: 'PR number'
10+
required: true
11+
12+
permissions:
13+
contents: write
14+
pull-requests: write
15+
16+
jobs:
17+
update-snapshots:
18+
if: ${{ github.event.inputs || (github.event.issue.pull_request && contains(github.event.comment.body, 'please update playwright snapshots')) }}
19+
runs-on: ubuntu-latest
20+
21+
steps:
22+
- name: Checkout
23+
uses: actions/checkout@v3
24+
with:
25+
token: ${{ secrets.GITHUB_TOKEN }}
26+
27+
- name: Configure git to use https
28+
run: git config --global hub.protocol https
29+
30+
- name: Checkout the branch from the PR that triggered the job
31+
run: hub pr checkout ${{ github.event.inputs.number || github.event.issue.number }}
32+
env:
33+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
34+
35+
- name: Base Setup
36+
uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1
37+
38+
- name: Install extension dependencies and build the extension
39+
run: ./scripts/install.sh
40+
41+
- uses: jupyterlab/maintainer-tools/.github/actions/update-snapshots@v1
42+
with:
43+
github_token: ${{ secrets.GITHUB_TOKEN }}
44+
start_server_script: 'null'
45+
test_folder: packages/jupyter-ai/ui-tests
46+
npm_client: jlpm
47+
48+
- name: Comment back on the PR
49+
run: |
50+
hub api repos/${{ github.repository }}/issues/${{ github.event.inputs.number || github.event.issue.number }}/comments --raw-field 'body=Playwright snapshots updated.'
51+
env:
52+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎docs/source/contributors/index.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,3 +117,38 @@ To uninstall your Jupyter AI development environment, deactivate and remove the
117117
conda deactivate
118118
conda env remove -n jupyter-ai
119119
```
120+
121+
## Testing
122+
123+
### Integration / E2E tests
124+
125+
This extension uses Playwright for the integration / E2E tests (user-level tests).
126+
More precisely, the JupyterLab helper
127+
[Galata](https://github.com/jupyterlab/jupyterlab/tree/master/galata) is used to
128+
test the extension in JupyterLab.
129+
130+
Install test dependencies (needed only once):
131+
132+
```sh
133+
cd ./packages/jupyter-ai/ui-tests/
134+
jlpm install
135+
jlpm playwright install
136+
```
137+
138+
Tests involve snapshot comparisons against a reference snapshots generated by the Github CI. If you are using an OS other than Linux, you will need to generate local snapshots before running the tests locally for the first time. To do this, execute the command:
139+
140+
```sh
141+
cd ./packages/jupyter-ai/ui-tests/
142+
jlpm test:update
143+
```
144+
145+
To execute tests, run:
146+
147+
```sh
148+
cd ./packages/jupyter-ai/ui-tests/
149+
jlpm test
150+
```
151+
152+
You can find more information in the
153+
[ui-tests](https://github.com/jupyterlab/jupyter-ai/tree/main/packages/jupyter-ai/ui-tests)
154+
README.

‎packages/jupyter-ai/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,3 +125,7 @@ dmypy.json
125125
# Coverage reports
126126
coverage/*
127127
junit.xml
128+
129+
# Ignore all non-linux snapshots
130+
ui-tests/tests/jupyter-ai.spec.ts-snapshots/*
131+
!ui-tests/tests/jupyter-ai.spec.ts-snapshots/*-linux.png

‎packages/jupyter-ai/ui-tests/README.md‎

Lines changed: 30 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -12,35 +12,37 @@ in [jupyter_server_test_config.py](./jupyter_server_test_config.py).
1212

1313
The default configuration will produce video for failing tests and an HTML report.
1414

15-
## Run the tests
15+
> There is a new experimental UI mode that you may fall in love with; see [that video](https://www.youtube.com/watch?v=jF0yA-JLQW0).
1616
1717
> All commands are assumed to be executed from the root directory
1818
19+
## Run the tests
20+
1921
To run the tests, you need to:
2022

2123
1. Compile the extension:
2224

2325
```sh
24-
jlpm install
25-
jlpm build:prod
26+
./scripts/install.sh
2627
```
2728

2829
> Check the extension is installed in JupyterLab.
2930
3031
2. Install test dependencies (needed only once):
3132

3233
```sh
33-
cd ./ui-tests
34+
cd ./packages/jupyter-ai/ui-tests/
3435
jlpm install
3536
jlpm playwright install
36-
cd ..
3737
```
3838

39+
> Tests involve snapshot comparisons against a reference snapshots generated by the Github CI. If you are using an OS other than Linux, you will need to generate local snapshots before running the tests locally for the first time. See [Update the tests snapshots](#update-the-tests-snapshots) section for the instructions.
40+
3941
3. Execute the [Playwright](https://playwright.dev/docs/intro) tests:
4042

4143
```sh
42-
cd ./ui-tests
43-
jlpm playwright test
44+
cd ./packages/jupyter-ai/ui-tests/
45+
jlpm test
4446
```
4547

4648
Test results will be shown in the terminal. In case of any test failures, the test report
@@ -50,34 +52,12 @@ for configuring that behavior.
5052

5153
## Update the tests snapshots
5254

53-
> All commands are assumed to be executed from the root directory
54-
5555
If you are comparing snapshots to validate your tests, you may need to update
56-
the reference snapshots stored in the repository. To do that, you need to:
57-
58-
1. Compile the extension:
59-
60-
```sh
61-
jlpm install
62-
jlpm build:prod
63-
```
64-
65-
> Check the extension is installed in JupyterLab.
66-
67-
2. Install test dependencies (needed only once):
56+
the reference snapshots stored in the repository. To do that, you need to execute the [Playwright](https://playwright.dev/docs/intro) command:
6857

6958
```sh
70-
cd ./ui-tests
71-
jlpm install
72-
jlpm playwright install
73-
cd ..
74-
```
75-
76-
3. Execute the [Playwright](https://playwright.dev/docs/intro) command:
77-
78-
```sh
79-
cd ./ui-tests
80-
jlpm playwright test -u
59+
cd ./packages/jupyter-ai/ui-tests/
60+
jlpm test:update
8161
```
8262

8363
> Some discrepancy may occurs between the snapshots generated on your computer and
@@ -87,39 +67,24 @@ jlpm playwright test -u
8767
8868
## Create tests
8969

90-
> All commands are assumed to be executed from the root directory
91-
9270
To create tests, the easiest way is to use the code generator tool of playwright:
9371

94-
1. Compile the extension:
72+
1. Start the server:
9573

9674
```sh
97-
jlpm install
98-
jlpm build:prod
75+
cd ./packages/jupyter-ai/ui-tests/
76+
jlpm start
9977
```
10078

101-
> Check the extension is installed in JupyterLab.
102-
103-
2. Install test dependencies (needed only once):
79+
2. Execute the [Playwright code generator](https://playwright.dev/docs/codegen) in **another terminal**:
10480

10581
```sh
106-
cd ./ui-tests
107-
jlpm install
108-
jlpm playwright install
109-
cd ..
110-
```
111-
112-
3. Execute the [Playwright code generator](https://playwright.dev/docs/codegen):
113-
114-
```sh
115-
cd ./ui-tests
82+
cd ./packages/jupyter-ai/ui-tests/
11683
jlpm playwright codegen localhost:8888
11784
```
11885

11986
## Debug tests
12087

121-
> All commands are assumed to be executed from the root directory
122-
12388
To debug tests, a good way is to use the inspector tool of playwright:
12489

12590
1. Compile the extension:
@@ -134,15 +99,24 @@ jlpm build:prod
13499
2. Install test dependencies (needed only once):
135100

136101
```sh
137-
cd ./ui-tests
102+
cd ./packages/jupyter-ai/ui-tests/
138103
jlpm install
139104
jlpm playwright install
140-
cd ..
141105
```
142106

143107
3. Execute the Playwright tests in [debug mode](https://playwright.dev/docs/debug):
144108

145109
```sh
146-
cd ./ui-tests
147-
PWDEBUG=1 jlpm playwright test
110+
cd ./packages/jupyter-ai/ui-tests/
111+
jlpm playwright test --debug
112+
```
113+
114+
## Upgrade Playwright and the browsers
115+
116+
To update the web browser versions, you must update the package `@playwright/test`:
117+
118+
```sh
119+
cd ./packages/jupyter-ai/ui-tests/
120+
jlpm up "@playwright/test"
121+
jlpm playwright install
148122
```

‎packages/jupyter-ai/ui-tests/jupyter_server_test_config.py‎

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,9 @@
44
opens the server to the world and provide access to JupyterLab
55
JavaScript objects through the global window variable.
66
"""
7-
from tempfile import mkdtemp
7+
from jupyterlab.galata import configure_jupyter_server
88

9-
c.ServerApp.port = 8888
10-
c.ServerApp.port_retries = 0
11-
c.ServerApp.open_browser = False
12-
13-
c.ServerApp.root_dir = mkdtemp(prefix="galata-test-")
14-
c.ServerApp.token = ""
15-
c.ServerApp.password = ""
16-
c.ServerApp.disable_check_xsrf = True
17-
c.LabApp.expose_app_in_browser = True
9+
configure_jupyter_server(c)
1810

1911
# Uncomment to set server log level to debug level
2012
# c.ServerApp.log_level = "DEBUG"

‎packages/jupyter-ai/ui-tests/package.json‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,16 @@
11
{
2-
"name": "jupyter_ai-ui-tests",
2+
"name": "jupyter-ai-e2e-tests",
33
"version": "1.0.0",
4-
"description": "JupyterLab jupyter_ai Integration Tests",
4+
"description": "Jupyter AI E2E tests",
55
"private": true,
66
"scripts": {
77
"start": "jupyter lab --config jupyter_server_test_config.py",
8-
"test": "jlpm playwright test"
8+
"install-chromium": "jlpm playwright install chromium",
9+
"test": "jlpm playwright test",
10+
"test:update": "jlpm playwright test --update-snapshots"
911
},
1012
"devDependencies": {
11-
"@jupyterlab/galata": "^4.3.0"
13+
"@jupyterlab/galata": "^5.0.5",
14+
"@playwright/test": "^1.37.0"
1215
}
1316
}

0 commit comments

Comments
 (0)