Playwright suite on CI + me/todo/autocomplete specs #666

trilopin · 2023-04-05T17:35:50Z

Summary

Key points:

Ordered execution
Videos recorded always
Screenshots are taken when there are failures (instead of always)
Shared auth state, login just once (through UI)
Rate limit bypassed (3 searches per entire suite consumed)
eslint added for playwright folder, extends webapp eslintrc.json and override some rules
tests are done against mattermost-enterprise image at master

Still 1/5 on the way tests are structured (unconnected/connected/nosetup map). A record would be nice in the future.

Ticket Link

Fixes https://mattermost.atlassian.net/browse/MM-51379
Fixes https://mattermost.atlassian.net/browse/MM-51368

…se github repo

trilopin · 2023-04-24T19:06:56Z

e2e/playwright/support/utils.ts


 import type {Page} from '@playwright/test';

 import {ChannelsPage} from '@e2e-support/ui/pages';
 import {UserProfile} from '@mattermost/types/users';
 import Client4 from '@mattermost/client/client4';

-const SCREENSHOTS_DIR = path.join(__dirname, '../screenshots');
+export const waitForNewMessages = async (page: Page) => {
+    await page.waitForTimeout(1000);


the smart wait for New-messages does not work (yet) in CI for some tests.

I'll research more, but I see some value in having green e2e until then.

trilopin · 2023-04-24T19:08:55Z

e2e/playwright/.eslintrc.json

@@ -0,0 +1,18 @@
+{
+    "extends": [
+        "../../webapp/.eslintrc.json"


I tried to extend from mattermost-server/e2e-tests/.eslintrc.json but has no rules at all, and missed most of the points I wanted to cover. That's why I extended from plugin/webapp/.eslintrc.json

trilopin · 2023-04-24T19:09:34Z

e2e/playwright/.eslintrc.json

+            "plugins": ["import"],
+            "parser": "@typescript-eslint/parser",
+            "rules": {
+                "import/no-unresolved": "off",


this shouldn't be here but fails with custom imports mapped in tsconfig.json

Does this cause other issues with linting? e.g. something that has to do with the type that was imported. This may be caught by the typescript compiler instead, though CI is not running checks with tsc in the e2e folder

Is this a problem only in CI? If so, I think we'll need to clone the webapp repo during linting

added tsc to CI

tsc/lint are done after cloning webapp repo in setup-environment. It happened on CI and locally

trilopin · 2023-04-26T10:54:39Z

@mickmister I'd want to finish and merge this before I go through your create-issue PR. The main reason is that I've changed the structure a lot. I'd be willing to adapt your PR after merging this one if you want.

trilopin · 2023-04-27T09:45:02Z

build/setup.mk

 # Ensure that go is installed. Note that this is independent of whether or not a server is being
 # built, since the build script itself uses go.
 ifeq ($(GO),)
    $(error "go is not available: see https://golang.org/doc/install")
 endif

 # Ensure that the build tools are compiled. Go's caching makes this quick.
-$(shell cd build/manifest && $(GO) build -o ../bin/manifest)
+$(shell cd build/manifest && $(GO) build -buildvcs=false $(GO_BUILD_FLAGS) -o ../bin/manifest)


@mickmister, this comes from your changes, is it needed? what's the goal?

This was originally to have the workflow runner be able to build the plugin in a docker container. This can be removed as that's not how we're running the tests.

mickmister

LGTM 🎉

mickmister · 2023-04-26T20:44:35Z

.github/workflows/playwright.yml

@@ -78,6 +78,7 @@ jobs:
          MM_SERVICESETTINGS_SITEURL: http://localhost:8065
          MM_PLUGINSETTINGS_AUTOMATICPREPACKAGEDPLUGINS: false
          MM_ANNOUNCEMENTSETTINGS_ADMINNOTICESENABLED: false
+          MM_SERVICESETTINGS_ENABLEONBOARDINGFLOW: false


Note that moving this logic here could make the test fail on a local machine I think

it was already handled in patchConfig, removed from here

.gitignore

mickmister · 2023-04-26T20:47:14Z

e2e/playwright/.eslintrc.json

+            "plugins": ["import"],
+            "parser": "@typescript-eslint/parser",
+            "rules": {
+                "import/no-unresolved": "off",


Is this a problem only in CI? If so, I think we'll need to clone the webapp repo during linting

mickmister · 2023-04-27T21:36:43Z

build/setup.mk

 # Ensure that go is installed. Note that this is independent of whether or not a server is being
 # built, since the build script itself uses go.
 ifeq ($(GO),)
    $(error "go is not available: see https://golang.org/doc/install")
 endif

 # Ensure that the build tools are compiled. Go's caching makes this quick.
-$(shell cd build/manifest && $(GO) build -o ../bin/manifest)
+$(shell cd build/manifest && $(GO) build -buildvcs=false $(GO_BUILD_FLAGS) -o ../bin/manifest)


This was originally to have the workflow runner be able to build the plugin in a docker container. This can be removed as that's not how we're running the tests.

mickmister · 2023-04-27T21:37:35Z

e2e/playwright/mock_oauth_server/index.ts

@@ -24,5 +27,6 @@ const app = makeOAuthServer({

 const port = process.env.OAUTH_SERVER_PORT || 8080;
 app.listen(port, () => {
+    // eslint-disable-next-line no-console


We can probably just get rid of this file. It's used to stand up the oauth server separate from the tests, but it's not ever used like that

mickmister · 2023-04-27T21:41:57Z

e2e/playwright/tests/command/autocomplete.spec.ts

+                // * Assert suggestions are visible
+                await slash.toBeVisible();


This reads weird without expect being on the same line

Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>

mickmister

LGTM 🎉

e2e/playwright/mock_oauth_server/index.ts

e2e/playwright/support/init_test.ts

srkgupta · 2023-05-02T09:39:47Z

e2e/playwright/tests/command/me.spec.ts

+
+export default {
+    connected: () => {
+        test.describe('/github me', () => {


Is this test dependent on any other test to be able to complete the pre-requisite steps of connecting their github account? I am sorry, but I am not seeing any references here.

EDIT: I see the sequence of tests defined in test.list.ts file. This might just be a little confusing if we dont see this file or read the code. Should we update somewhere in Playwright README that the tests are executed in the order defined in e2e/playwright/tests/test.list.ts file (or) do you think this is not required and is a standard Playwright practice of writing E2E tests?

unfortunately, tests are dependent and ran serially in a specific order specified in test.list.ts

in the future, we will explore better approaches to prepare the entire env for each test, due to the complexity, the first full iteration is like this.

srkgupta · 2023-05-02T09:53:32Z

e2e/playwright/tests/command/todo.spec.ts

+                await expect(todo.getDesc(GithubRHSCategory.OPEN_PR)).toHaveText('You have 1 open pull requests:');
+                await expect(todo.getDesc(GithubRHSCategory.ASSIGNMENTS)).toHaveText('You have 19 assignments:');
+                await expect(todo.getDesc(GithubRHSCategory.REVIEW_PR)).toHaveText('You have 1 pull requests awaiting your review:');
+                await expect(todo.getDesc(GithubRHSCategory.UNREAD)).toHaveText('You have 1 unread messages:');


Should we have these numbers defined in a separate file, so its easy to edit in future (if these value changes due to any other situations) and ensure assertions passes, instead of requiring the test to be updated in multiple places? What do you think?

💯 to that, I'll add that change in my tests for sidebar/coutners since I need to use them too.

srkgupta

Thanks @trilopin. The tests and it's dependencies looks great overall. Just left a few minor comments.

srkgupta

Approving the changes based on previous comments and discussions. LGTM 🎉

* master: Playwright suite on CI + me/todo/autocomplete specs (#666)

mickmister added 30 commits March 9, 2023 06:12

initial config for playwright tests

1ba332e

move files

9206110

add test for /github setup

336f6cb

remove comment

e13a256

change test name

712f2cb

point to webapp repo, and automatically adjust playwright config to u…

d50de72

…se github repo

reference mm-webapp directly via tsconfig.json, and npm runn commands

835c014

add await for completeness

0f7a987

fix plugin upload

ab59cd6

add assertion for connect link URL

516784a

use cypress comment conventions

7a37280

add show-report command

37ee3de

move code around

6059bfb

setup and connect work with mock oauth server

5341060

add functionality for mock oauth server

1a4e957

finish setup and connect tests

3a1cf0f

rename env var

aa04abd

change oauth token env var name

2e09f01

rearrange gitignore

4baacde

run oauth server inside of playwright tests

9755ff6

add base github action

80da990

changes for running playwright

5c72c6b

ci env var

4c04fb1

run ci unconditionally

e50225e

more ci changes

cfb010f

change ci docker image

e3412ac

remove mm-webapp dependency

2ec71ad

split up ci steps

e875642

fix npm dep cache path

74c6af4

use mickmister fork for webapp repo

d45dcee

trilopin added 2 commits April 24, 2023 20:44

use the right path

3d0473e

use the right order

e20eed4

trilopin requested a review from mickmister April 24, 2023 19:00

trilopin added 2: Dev Review Requires review by a core committer and removed Work In Progress Not yet ready for review labels Apr 24, 2023

trilopin requested a review from srkgupta April 24, 2023 19:00

trilopin commented Apr 24, 2023

View reviewed changes

npm i with the right version

ffb9ad7

trilopin changed the title ~~WIP Playwright specs~~ Playwright suite on CI + me/todo/autocomplete specs Apr 25, 2023

fix lint warning + introduce tsc for playwright in CI

c9dc7ca

trilopin commented Apr 27, 2023

View reviewed changes

mickmister approved these changes Apr 27, 2023

View reviewed changes

trilopin and others added 2 commits April 28, 2023 09:39

Update .gitignore

aea58c9

Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>

address feedback

7f8324a

mickmister approved these changes Apr 28, 2023

View reviewed changes

srkgupta reviewed May 2, 2023

View reviewed changes

e2e/playwright/mock_oauth_server/index.ts Show resolved Hide resolved

srkgupta reviewed May 2, 2023

View reviewed changes

e2e/playwright/support/init_test.ts Show resolved Hide resolved

srkgupta reviewed May 2, 2023

View reviewed changes

e2e/playwright/support/init_test.ts Show resolved Hide resolved

srkgupta reviewed May 2, 2023

View reviewed changes

srkgupta suggested changes May 2, 2023

View reviewed changes

srkgupta self-requested a review May 2, 2023 10:00

srkgupta approved these changes May 2, 2023

View reviewed changes

trilopin merged commit 9bfa67c into master May 2, 2023

trilopin deleted the pw-specs branch May 2, 2023 10:22

trilopin added a commit that referenced this pull request May 2, 2023

Add counter e2e tests, with and without refresh

edb4d7a

* master: Playwright suite on CI + me/todo/autocomplete specs (#666)

avas27JTG mentioned this pull request Nov 9, 2023

Bump version to 2.2.0. #709

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Playwright suite on CI + me/todo/autocomplete specs #666

Playwright suite on CI + me/todo/autocomplete specs #666

trilopin commented Apr 5, 2023 •

edited

Loading

trilopin Apr 24, 2023 •

edited

Loading

trilopin Apr 24, 2023 •

edited

Loading

trilopin Apr 24, 2023

mickmister Apr 26, 2023

mickmister Apr 26, 2023

trilopin Apr 27, 2023

trilopin Apr 28, 2023

trilopin commented Apr 26, 2023 •

edited

Loading

trilopin Apr 27, 2023

mickmister Apr 27, 2023

mickmister left a comment

mickmister Apr 26, 2023

trilopin Apr 28, 2023

mickmister Apr 26, 2023

mickmister Apr 27, 2023

mickmister Apr 27, 2023

mickmister Apr 27, 2023

mickmister left a comment

srkgupta May 2, 2023 •

edited

Loading

trilopin May 2, 2023

srkgupta May 2, 2023

trilopin May 2, 2023

srkgupta left a comment

srkgupta left a comment

		// * Assert suggestions are visible
		await slash.toBeVisible();

Playwright suite on CI + me/todo/autocomplete specs #666

Playwright suite on CI + me/todo/autocomplete specs #666

Conversation

trilopin commented Apr 5, 2023 • edited Loading

Summary

Ticket Link

trilopin Apr 24, 2023 • edited Loading

Choose a reason for hiding this comment

trilopin Apr 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trilopin commented Apr 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mickmister left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mickmister left a comment

Choose a reason for hiding this comment

srkgupta May 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srkgupta left a comment

Choose a reason for hiding this comment

srkgupta left a comment

Choose a reason for hiding this comment

trilopin commented Apr 5, 2023 •

edited

Loading

trilopin Apr 24, 2023 •

edited

Loading

trilopin Apr 24, 2023 •

edited

Loading

trilopin commented Apr 26, 2023 •

edited

Loading

srkgupta May 2, 2023 •

edited

Loading