Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Playwright suite on CI + me/todo/autocomplete specs #666

Merged
merged 119 commits into from
May 2, 2023
Merged
Show file tree
Hide file tree
Changes from 117 commits
Commits
Show all changes
119 commits
Select commit Hold shift + click to select a range
1ba332e
initial config for playwright tests
mickmister Mar 9, 2023
9206110
move files
mickmister Mar 9, 2023
336f6cb
add test for `/github setup`
mickmister Mar 9, 2023
e13a256
remove comment
mickmister Mar 9, 2023
712f2cb
change test name
mickmister Mar 9, 2023
d50de72
point to webapp repo, and automatically adjust playwright config to u…
mickmister Mar 9, 2023
835c014
reference mm-webapp directly via tsconfig.json, and npm runn commands
mickmister Mar 9, 2023
0f7a987
add await for completeness
mickmister Mar 9, 2023
ab59cd6
fix plugin upload
mickmister Mar 10, 2023
516784a
add assertion for connect link URL
mickmister Mar 10, 2023
7a37280
use cypress comment conventions
mickmister Mar 10, 2023
37ee3de
add show-report command
mickmister Mar 10, 2023
6059bfb
move code around
mickmister Mar 15, 2023
5341060
setup and connect work with mock oauth server
mickmister Mar 15, 2023
1a4e957
add functionality for mock oauth server
mickmister Mar 16, 2023
3a1cf0f
finish setup and connect tests
mickmister Mar 16, 2023
aa04abd
rename env var
mickmister Mar 16, 2023
2e09f01
change oauth token env var name
mickmister Mar 16, 2023
4baacde
rearrange gitignore
mickmister Mar 16, 2023
9755ff6
run oauth server inside of playwright tests
mickmister Mar 17, 2023
80da990
add base github action
mickmister Mar 17, 2023
5c72c6b
changes for running playwright
mickmister Mar 17, 2023
4c04fb1
ci env var
mickmister Mar 17, 2023
e50225e
run ci unconditionally
mickmister Mar 17, 2023
cfb010f
more ci changes
mickmister Mar 17, 2023
e3412ac
change ci docker image
mickmister Mar 17, 2023
2ec71ad
remove mm-webapp dependency
mickmister Mar 17, 2023
e875642
split up ci steps
mickmister Mar 17, 2023
74c6af4
fix npm dep cache path
mickmister Mar 17, 2023
d45dcee
use mickmister fork for webapp repo
mickmister Mar 17, 2023
828be6a
fix setup-environment
mickmister Mar 17, 2023
ad2aecf
fix webapp repo checkout
mickmister Mar 17, 2023
6c3f477
change setup-environment to have webapp repo fetch its deps
mickmister Mar 17, 2023
ae1826b
move env var to step
mickmister Mar 17, 2023
5fb1dea
move env var to test step
mickmister Mar 17, 2023
1b7f42e
run ci headless
mickmister Mar 17, 2023
7d6d978
add longer waits
mickmister Mar 17, 2023
4024d9b
add more screenshot steps for debugging ci
mickmister Mar 20, 2023
d7234d9
map oauth server port in ci config
mickmister Mar 21, 2023
c1e7c95
use docker network-alias to connect to oauth server
mickmister Mar 21, 2023
90257f8
change image version
mickmister Mar 21, 2023
8f36cd2
typo
mickmister Mar 21, 2023
b840c2e
use user-defined bridge network aliases
mickmister Mar 21, 2023
d40e17e
add go build flag for compiling plugin
mickmister Mar 21, 2023
11554b6
change ci container image
mickmister Mar 21, 2023
b963909
use go 1.18 to build plugin
mickmister Mar 21, 2023
6c68e87
use go 1.18 for build folder go.mod
mickmister Mar 21, 2023
266f1ee
use GO_BUILD_FLAGS when compiling build tools
mickmister Mar 21, 2023
678a1aa
bump ci go version to make vcs error show
mickmister Mar 21, 2023
7c7ec4f
force buildvcs=false go build flag
mickmister Mar 21, 2023
8fdf456
change database connection string
mickmister Mar 21, 2023
b4b02eb
temporarily hardcode db host because env var doesn't apply in playwri…
mickmister Mar 21, 2023
ea8a188
fix db host
mickmister Mar 21, 2023
6a0b226
set db host manually
mickmister Mar 21, 2023
1808905
npx playwright install --with-deps
mickmister Mar 21, 2023
4df5cc9
use playwright docker image
mickmister Mar 21, 2023
9a8a92a
revert changes to make it so we build on runner process again
mickmister Mar 21, 2023
33e6251
use docker host ubuntu ip
mickmister Mar 21, 2023
11c4034
test must pass
mickmister Mar 21, 2023
08f8a20
dirty code tests from michael s poc
trilopin Mar 13, 2023
a25c3c2
add debug
trilopin Mar 13, 2023
e50edcd
first test for /github me command
trilopin Mar 15, 2023
7a9abdc
improved autocomplete tests, slash helper local
trilopin Mar 16, 2023
5c1e13d
todo tests + helpers
trilopin Mar 17, 2023
3812056
changes for monorepo
mickmister Mar 24, 2023
1ca9377
remove import of webapp test setup in webapp tests
mickmister Mar 24, 2023
050dc15
Merge branch 'e2e-playwright' into e2e-playwright-tests
trilopin Mar 24, 2023
848d9ff
use testBrowser.login()
mickmister Mar 24, 2023
dafbb06
try ci with just iphone platform
mickmister Mar 24, 2023
db20a9c
upload test-results artifacts
mickmister Mar 24, 2023
e1d34a2
include test results in artifact
mickmister Mar 24, 2023
f6bcbe9
mv files
mickmister Mar 24, 2023
bf533e1
upload artifacts unconditionally
mickmister Mar 24, 2023
d7e4591
run test for all devices
mickmister Mar 24, 2023
90200b5
use ldflags to communicate with mock oauth server
mickmister Mar 29, 2023
ae77466
use PLUGIN_GITHUB_E2E_OAUTH_TOKEN
mickmister Mar 29, 2023
6832c74
use custom playwright config
mickmister Mar 29, 2023
0c8e871
fail when oauth token is not provided
mickmister Mar 29, 2023
09c62fb
change artifact logic since we're running config in cwd now
mickmister Mar 29, 2023
3b5f584
serial and ordered tests
trilopin Mar 28, 2023
7d4e7c8
Merge branch 'e2e-playwright' into e2e-playwright-tests
trilopin Mar 30, 2023
393ffa7
Merge branch 'e2e-playwright-tests' into pw-specs
trilopin Apr 5, 2023
e499d35
force close and fix sleep
trilopin Apr 5, 2023
a652681
get github_setup improvements
trilopin Apr 5, 2023
4f7fafe
green suite (no rate limit fix yet)
trilopin Apr 12, 2023
57eefdb
green suite (no rate limit fix yet) again
trilopin Apr 12, 2023
07fdaef
shared auth, bypass rate limit
trilopin Apr 17, 2023
ec8ba8a
Merge branch 'master' into pw-specs
trilopin Apr 17, 2023
765577a
force npm i with npm6 to get back to lockfileversion 1
trilopin Apr 18, 2023
d5ba685
update mattermost-redux to avoid redux-offline
trilopin Apr 18, 2023
a110f33
increase timeout + move artifacts
trilopin Apr 18, 2023
b81b834
small cleanup
trilopin Apr 18, 2023
f30d99a
try waitfornewmessages approach
trilopin Apr 19, 2023
6f4c34f
remove unnecesary code + keep wait
trilopin Apr 19, 2023
8b44986
try with 4 cores
trilopin Apr 19, 2023
824faca
again with 2 cores
trilopin Apr 19, 2023
1a9c03d
rely on a new env var injected on webpacl instead of enableDeveloper
trilopin Apr 20, 2023
2ddf651
try 4 cores
trilopin Apr 20, 2023
f34ce51
chase race condition
trilopin Apr 20, 2023
eb78faf
disable me test
trilopin Apr 20, 2023
9ca62af
iter 1
trilopin Apr 20, 2023
751eb04
add debug
trilopin Apr 20, 2023
7ba0bb2
try some changes on setup
trilopin Apr 21, 2023
d24ed76
try some changes on setup 2
trilopin Apr 21, 2023
e148813
skip assertiong on bot name
trilopin Apr 21, 2023
68d9910
wait for ts approach
trilopin Apr 24, 2023
7d54e9a
try with server at master
trilopin Apr 24, 2023
8dcb658
disable onbaordingflow
trilopin Apr 24, 2023
545af9e
avoid tutorials
trilopin Apr 24, 2023
0ab6398
try again the smart New-messages detection
trilopin Apr 24, 2023
4e7cc75
adress feedback
trilopin Apr 24, 2023
d0df667
add eslint
trilopin Apr 24, 2023
fb7a48a
extend eslint + run eslint in ci
trilopin Apr 24, 2023
3d0473e
use the right path
trilopin Apr 24, 2023
e20eed4
use the right order
trilopin Apr 24, 2023
ffb9ad7
npm i with the right version
trilopin Apr 25, 2023
c9dc7ca
fix lint warning + introduce tsc for playwright in CI
trilopin Apr 27, 2023
aea58c9
Update .gitignore
trilopin Apr 28, 2023
7f8324a
address feedback
trilopin Apr 28, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions .github/workflows/playwright.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
ports:
- 9200:9200
mattermost-server:
image: mattermost/mattermost-enterprise-edition:release-7.7
image: mattermost/mattermost-enterprise-edition:master
env:
DB_HOST: postgres
DB_PORT_NUMBER: 5432
Expand All @@ -78,6 +78,7 @@ jobs:
MM_SERVICESETTINGS_SITEURL: http://localhost:8065
MM_PLUGINSETTINGS_AUTOMATICPREPACKAGEDPLUGINS: false
MM_ANNOUNCEMENTSETTINGS_ADMINNOTICESENABLED: false
MM_SERVICESETTINGS_ENABLEONBOARDINGFLOW: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that moving this logic here could make the test fail on a local machine I think

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was already handled in patchConfig, removed from here

MM_FEATUREFLAGS_AppsEnabled: true

ports:
Expand Down Expand Up @@ -126,6 +127,7 @@ jobs:
run: make deploy
env:
PLUGIN_E2E_MOCK_OAUTH_SERVER_URL: http://172.17.0.1:8080
E2E_TESTING: true

- name: ci/setup-node
uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0
Expand All @@ -139,6 +141,16 @@ jobs:
cd e2e/playwright
./setup-environment

- name: ci/tsc
run: |
cd e2e/playwright
npm run tsc

- name: ci/lint
run: |
cd e2e/playwright
npm run lint

- name: ci/run-playwright-tests
run: |
cd e2e/playwright
Expand All @@ -152,8 +164,6 @@ jobs:
run: |
cd e2e/playwright
mkdir results

mv screenshots results
mv playwright-report results

- uses: actions/upload-artifact@v3
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,5 @@ dist/

# Jetbrains
.idea/

_debug
trilopin marked this conversation as resolved.
Show resolved Hide resolved
5 changes: 4 additions & 1 deletion build/custom.mk
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,8 @@ endif
GO_BUILD_FLAGS += -ldflags '-X "github.com/mattermost/mattermost-plugin-api/experimental/telemetry.rudderWriteKey=$(RUDDER_WRITE_KEY)"'

ifdef PLUGIN_E2E_MOCK_OAUTH_SERVER_URL
GO_BUILD_FLAGS += -ldflags '-X "github.com/mattermost/mattermost-plugin-github/server/plugin.testOAuthServerURL=$(PLUGIN_E2E_MOCK_OAUTH_SERVER_URL)"'
GO_BUILD_FLAGS += -ldflags '-X "github.com/mattermost/mattermost-plugin-github/server/plugin.testOAuthServerURL=$(PLUGIN_E2E_MOCK_OAUTH_SERVER_URL)"'
endif
.PHONY: deploy-e2e
deploy-e2e: dist
E2E_TESTING=true PLUGIN_E2E_MOCK_OAUTH_SERVER_URL=http://localhost:8080 ./build/bin/pluginctl deploy $(PLUGIN_ID) dist/$(BUNDLE_NAME)
6 changes: 4 additions & 2 deletions build/setup.mk
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
GO_BUILD_FLAGS ?=

# Ensure that go is installed. Note that this is independent of whether or not a server is being
# built, since the build script itself uses go.
ifeq ($(GO),)
$(error "go is not available: see https://golang.org/doc/install")
endif

# Ensure that the build tools are compiled. Go's caching makes this quick.
$(shell cd build/manifest && $(GO) build -o ../bin/manifest)
$(shell cd build/manifest && $(GO) build -buildvcs=false $(GO_BUILD_FLAGS) -o ../bin/manifest)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mickmister, this comes from your changes, is it needed? what's the goal?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was originally to have the workflow runner be able to build the plugin in a docker container. This can be removed as that's not how we're running the tests.


# Ensure that the deployment tools are compiled. Go's caching makes this quick.
$(shell cd build/pluginctl && $(GO) build -o ../bin/pluginctl)
$(shell cd build/pluginctl && $(GO) build -buildvcs=false $(GO_BUILD_FLAGS) -o ../bin/pluginctl)

# Extract the plugin id from the manifest.
PLUGIN_ID ?= $(shell build/bin/manifest id)
Expand Down
18 changes: 18 additions & 0 deletions e2e/playwright/.eslintrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"extends": [
"../../webapp/.eslintrc.json"
Copy link
Contributor Author

@trilopin trilopin Apr 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to extend from mattermost-server/e2e-tests/.eslintrc.json but has no rules at all, and missed most of the points I wanted to cover. That's why I extended from plugin/webapp/.eslintrc.json

],
"overrides": [
{
"files": ["**/*.ts"],
"plugins": ["import"],
"parser": "@typescript-eslint/parser",
"rules": {
"import/no-unresolved": "off",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this shouldn't be here but fails with custom imports mapped in tsconfig.json

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this cause other issues with linting? e.g. something that has to do with the type that was imported. This may be caught by the typescript compiler instead, though CI is not running checks with tsc in the e2e folder

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a problem only in CI? If so, I think we'll need to clone the webapp repo during linting

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added tsc to CI

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tsc/lint are done after cloning webapp repo in setup-environment. It happened on CI and locally

"max-nested-callbacks": "off",
"no-shadow": "off",
"@typescript-eslint/no-shadow": "error"
}
}
]
}
2 changes: 2 additions & 0 deletions e2e/playwright/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,5 @@
screenshots
playwright-report
test-results
.auth-user.json
.eslintcache
2 changes: 1 addition & 1 deletion e2e/playwright/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ What this script does:
- Install Playwright test dependencies - `cd ../e2e-tests/playwright && npm i`
- Install Playwright - `npx install playwright`
- Install GitHub plugin e2e dependencies - `cd ../../../mattermost-plugin-github/e2e/playwright && npm i`
- Build and deploy plugin with mock OAuth server URL env var - `PLUGIN_E2E_MOCK_OAUTH_SERVER_URL=http://localhost:8080 make deploy`
- Build and deploy plugin with e2e support - `make deploy-e2e`

-----

Expand Down
5 changes: 5 additions & 0 deletions e2e/playwright/mock_oauth_server/.env.sample
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# export MOCK_OAUTH_ACCESS_TOKEN=

# export MM_SERVICESETTINGS_SITEURL=
# export MM_PLUGIN_ID=
# export OAUTH_SERVER_PORT=
6 changes: 5 additions & 1 deletion e2e/playwright/mock_oauth_server/index.ts
srkgupta marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
/* eslint-disable no-console */
/* eslint-disable no-process-exit */
/* eslint-disable no-process-env */
require('dotenv').config();

import {ExpiryAlgorithms, makeOAuthServer} from './mock_oauth_server';
Expand All @@ -8,7 +11,7 @@ if (!mockOAuthAccessToken) {
process.exit(0);
}

const defaultAuthorizePrefix = '/login/oauth' // Used by GitHub
const defaultAuthorizePrefix = '/login/oauth'; // Used by GitHub
const authorizeURLPrefix = process.env.OAUTH_AUTHORIZE_URL_PREFIX || defaultAuthorizePrefix;

const mattermostSiteURL = process.env.MM_SERVICESETTINGS_SITEURL || 'http://localhost:8065';
Expand All @@ -24,5 +27,6 @@ const app = makeOAuthServer({

const port = process.env.OAUTH_SERVER_PORT || 8080;
app.listen(port, () => {
// eslint-disable-next-line no-console
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably just get rid of this file. It's used to stand up the oauth server separate from the tests, but it's not ever used like that

console.log(`Mock OAuth server listening on port ${port}`);
});
9 changes: 5 additions & 4 deletions e2e/playwright/mock_oauth_server/mock_oauth_server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ export const makeOAuthServer = ({
expiryAlgorithm,
}: OAuthServerOptions): express.Express => {
if (!mockOAuthAccessToken) {
throw new Error(`MockOAuthServer: Please provide an OAuth access token to use`);
throw new Error('MockOAuthServer: Please provide an OAuth access token to use');
}

if (expiryAlgorithm !== ExpiryAlgorithms.NO_EXPIRY) {
Expand All @@ -29,14 +29,15 @@ export const makeOAuthServer = ({

const app = express();

// eslint-disable-next-line new-cap
const oauthRouter = express.Router();

oauthRouter.get('/authorize', function (req, res) {
oauthRouter.get('/authorize', (req, res) => {
const query = req.url.split('?')[1];
res.redirect(`${mattermostSiteURL}/plugins/${pluginId}/oauth/complete?${query}&code=1234`);
});

oauthRouter.post('/access_token', function (req, res) {
oauthRouter.post('/access_token', (req, res) => {
const token = {
access_token: mockOAuthAccessToken,
token_type: 'bearer',
Expand All @@ -49,4 +50,4 @@ export const makeOAuthServer = ({
app.use(authorizeURLPrefix, oauthRouter);

return app;
}
};
Loading