Performance Tests for Login and Home Pages by using Playwright #119681

suchcodemuchwow · 2021-11-24T23:04:57Z

This PR aims to experiment different libraries, tools and options for performance test environment which we aim to see stable results in each CI build.

…ts by using synthetics

suchcodemuchwow · 2021-11-26T17:00:41Z

@elasticmachine merge upstream

…me-page

…hetics-perf-test-login-and-home-page

… github.com:suchcodemuchwow/kibana into 2021-11-25-synthetics-perf-test-login-and-home-page

x-pack/test/performance/config.synthetics.ts

lizozom · 2021-11-30T13:26:51Z

I'd love to track this PR in the main meta issue
Is this and #117861 two different tests?

…hetics-perf-test-login-and-home-page

…-6 to c2-60 change machine types running performance tests

.buildkite/pipelines/performance/nightly.yml

test/functional/services/remote/webdriver.ts

x-pack/test/performance/config.playwright.ts

tylersmalley

Some minor changes, overall looking good!

… github.com:suchcodemuchwow/kibana into 2021-11-25-synthetics-perf-test-login-and-home-page

LeeDr · 2022-01-13T14:26:18Z

@suchcodemuchwow could you guide me to where the results of the performance tests are?

dmlemeshko · 2022-01-13T14:55:03Z

Run performance tests by running ES and Kibana separately

Hey @suchcodemuchwow Cool stuff! I can see that you start them separately, but ES and Kibana still share the same machine/resources, right? What is the reason to keep ES running longer?

suchcodemuchwow · 2022-01-13T16:11:27Z

Run performance tests by running ES and Kibana separately

Hey @suchcodemuchwow Cool stuff! I can see that you start them separately, but ES and Kibana still share the same machine/resources, right? What is the reason to keep ES running longer?

Hey @dmlemeshko, we wanted to keep ES running and have two phases of kibana tests running. We have 1 WARMUP and 1 TEST phase. We found out that in first (WARMUP) phase we have longer durations compared to subsequent runs. 2,3,4 ... is quite similar and we decided to only have 1 TEST phase.
Here where we are running these tests.

suchcodemuchwow · 2022-01-13T16:15:15Z

Hey @LeeDr, we had bunch of builds in this PR because of experimentation, but below one is the most recent one:
APM UI
Performance Dashboard

suchcodemuchwow · 2022-01-17T10:56:57Z

@LeeDr, @dmlemeshko could you please check my comments. We would like to merge this PR so we can run our daily jobs.

dmlemeshko

LGTM

suchcodemuchwow · 2022-01-18T09:39:15Z

@elasticmachine merge upstream

…me-page

kibana-ci · 2022-01-18T10:58:45Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #16931 succeeded 4470336
💛 Build #16628 was flaky 2441674
💔 Build #15623 failed 4778a08
💚 Build #15440 succeeded 16032a0
💔 Build #15410 failed 6d1460e
💔 Build #15232 failed 2a81b1a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @suchcodemuchwow

kibanamachine · 2022-01-18T11:03:39Z

💔 All backports failed

Status	Branch	Result
❌	8.0	Backport failed because of merge conflicts
❌	7.17	Backport failed because of merge conflicts

How to fix

Re-run the backport manually:

node scripts/backport --pr 119681

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2022-01-20T11:06:19Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 119681 or prevent reminders by adding the backport:skip label.

kibanamachine · 2022-01-21T11:16:18Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 119681 or prevent reminders by adding the backport:skip label.

kibanamachine · 2022-01-24T11:26:17Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 119681 or prevent reminders by adding the backport:skip label.

…) (#123619) add performance tests for Login and Home pages - [x] Rename and reduce the number of environment variables for performance test CI job - [x] ITERATION_COUNT_ENV => PERF_TEST_COUNT - [x] DISABLE_CACHE, TEST_RUNNER, TEST_THROTTLE_NETWORK removed - [x] Use c2-16 queue for performance tests - [x] Run performance tests by running ES and Kibana separately - [x] Add warmup phase to performance tests (Not reported to APM). APM Label: PERF_TEST_PHASE - [x] Add playwright dependency - [x] Throttle network speed ( downloadThroughput: 750.000 B/s, uploadThroughput: 750.000 B/s, latency: 100 ms ) - [x] Disable cache by using - page.route (‘**’, ….) - [x] Login and Home pages performance tests. (cherry picked from commit eeb5efc) # Conflicts: # x-pack/test/performance/config.playwright.ts

…) (#123620) add performance tests for Login and Home pages - [x] Rename and reduce the number of environment variables for performance test CI job - [x] ITERATION_COUNT_ENV => PERF_TEST_COUNT - [x] DISABLE_CACHE, TEST_RUNNER, TEST_THROTTLE_NETWORK removed - [x] Use c2-16 queue for performance tests - [x] Run performance tests by running ES and Kibana separately - [x] Add warmup phase to performance tests (Not reported to APM). APM Label: PERF_TEST_PHASE - [x] Add playwright dependency - [x] Throttle network speed ( downloadThroughput: 750.000 B/s, uploadThroughput: 750.000 B/s, latency: 100 ms ) - [x] Disable cache by using - page.route (‘**’, ….) - [x] Login and Home pages performance tests. (cherry picked from commit eeb5efc) # Conflicts: # x-pack/test/performance/config.playwright.ts

matschaffer · 2022-06-24T07:25:23Z

test/functional/services/remote/network_profiles.ts

-  CABLE_5MBPS: { DOWNLOAD: 5 * kB * sec, UPLOAD: 1 * kB * sec, LATENCY: 0.28 * sec },
-  CABLE_8MBPS: { DOWNLOAD: 8 * kB * sec, UPLOAD: 2 * kB * sec, LATENCY: 0.1 * sec },
-  WIFI: { DOWNLOAD: 30 * kB * sec, UPLOAD: 15 * kB * sec, LATENCY: 0.002 * sec },
+  CLOUD_USER: { DOWNLOAD: 6 * MBps, UPLOAD: 6 * MBps, LATENCY: 0.1 * sec },


Anyone know why we removed these? I was looking for a way to throttle during a functional test to confirm a loading theory. Seems like this might have worked and I was about to add 3G_SLOW back. But maybe I shouldn't?

We removed them because they were inaccurate and finding information that accurately describes the speeds used by the browser profiles was really hard, so we punted on it and only worried about the profile that we actually use.

Gotcha. #135103 has the snippet I ended up using instead (to slow a test down enough to fail). Maybe this sort of thing would be better as a helper rather than webdriver-init-time setting 🤔

feat(performance tests for login and home pages): add performance tes…

025c0d9

…ts by using synthetics

suchcodemuchwow added chore v8.0.0 release_note:skip Skip the PR/issue when compiling release notes auto-backport Deprecated - use backport:version if exact versions are needed v7.16.0 v8.1.0 labels Nov 24, 2021

kibanamachine and others added 9 commits November 26, 2021 12:00

Merge branch 'main' into 2021-11-25-synthetics-perf-test-login-and-ho…

ee2579a

…me-page

Merge branch 'main' of github.com:elastic/kibana into 2021-11-25-synt…

ade404f

…hetics-perf-test-login-and-home-page

fix(functional_tests.js): add synthetics config import

6eba35d

Merge branch '2021-11-25-synthetics-perf-test-login-and-home-page' of…

d8e7a31

… github.com:suchcodemuchwow/kibana into 2021-11-25-synthetics-perf-test-login-and-home-page

fix test start

a441885

fix test config

3c0323b

fix: increase timeout of Login to Kibana step

2518fd4

fix: change ftr config to let kibana server ask login

2954966

fix: ftr tests config fix

e0dbfe0

suchcodemuchwow marked this pull request as ready for review November 29, 2021 16:11

suchcodemuchwow requested a review from a team as a code owner November 29, 2021 16:11

tylersmalley reviewed Nov 29, 2021

View reviewed changes

x-pack/test/performance/config.synthetics.ts Outdated Show resolved Hide resolved

Cavit Baturalp Gurdin added 7 commits November 30, 2021 16:31

refactor: remove labels, move test runner to separate file

67d0d6c

[Perf] move performance test scripts to own files

b15fcb5

[Perf] add sleeps to server starts

d2bb2cc

[Perf] Fix run tests in parallel

aca36a8

Merge branch 'main' of github.com:elastic/kibana into 2021-11-25-synt…

93ebbc2

…hetics-perf-test-login-and-home-page

Merge branch 'main' of github.com:elastic/kibana into 2021-11-25-synt…

5704c08

…hetics-perf-test-login-and-home-page

ci(.buildkite/pipelines/performance/nightly.yml): swap queue ci-group…

f3e6cb7

…-6 to c2-60 change machine types running performance tests

suchcodemuchwow commented Dec 2, 2021

View reviewed changes

.buildkite/pipelines/performance/nightly.yml Outdated Show resolved Hide resolved

fix: add c2-60 instance types to sub performance tests

f76fa49

tylersmalley reviewed Jan 12, 2022

View reviewed changes

test/functional/services/remote/webdriver.ts Outdated Show resolved Hide resolved

tylersmalley reviewed Jan 12, 2022

View reviewed changes

x-pack/test/performance/config.playwright.ts Outdated Show resolved Hide resolved

tylersmalley suggested changes Jan 12, 2022

View reviewed changes

Cavit Baturalp Gurdin added 2 commits January 12, 2022 14:41

[refactor] prefix environment variables with TEST

8f334a0

Merge branch '2021-11-25-synthetics-perf-test-login-and-home-page' of…

4470336

… github.com:suchcodemuchwow/kibana into 2021-11-25-synthetics-perf-test-login-and-home-page

tylersmalley approved these changes Jan 12, 2022

View reviewed changes

suchcodemuchwow changed the title ~~Performance Tests for Login and Home Pages by using Synthetics~~ Performance Tests for Login and Home Pages by using Playwright Jan 13, 2022

suchcodemuchwow self-assigned this Jan 13, 2022

dmlemeshko approved these changes Jan 17, 2022

View reviewed changes

Merge branch 'main' into 2021-11-25-synthetics-perf-test-login-and-ho…

1730c8d

…me-page

suchcodemuchwow merged commit eeb5efc into elastic:main Jan 18, 2022

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jan 20, 2022

This was referenced Jan 24, 2022

[8.0] Performance Tests for Login and Home Pages by using Playwright (#119681) #123619

Merged

[7.17] Performance Tests for Login and Home Pages by using Playwright (#119681) #123620

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jan 24, 2022

matschaffer reviewed Jun 24, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Performance Tests for Login and Home Pages by using Playwright #119681

Performance Tests for Login and Home Pages by using Playwright #119681

suchcodemuchwow commented Nov 24, 2021 •

edited

Loading

suchcodemuchwow commented Nov 26, 2021

lizozom commented Nov 30, 2021

tylersmalley left a comment

LeeDr commented Jan 13, 2022

dmlemeshko commented Jan 13, 2022 •

edited

Loading

suchcodemuchwow commented Jan 13, 2022 •

edited

Loading

suchcodemuchwow commented Jan 13, 2022

suchcodemuchwow commented Jan 17, 2022

dmlemeshko left a comment

suchcodemuchwow commented Jan 18, 2022

kibana-ci commented Jan 18, 2022

kibanamachine commented Jan 18, 2022

kibanamachine commented Jan 20, 2022

kibanamachine commented Jan 21, 2022

kibanamachine commented Jan 24, 2022

matschaffer Jun 24, 2022

spalger Jun 24, 2022 •

edited

Loading

matschaffer Jun 24, 2022

Performance Tests for Login and Home Pages by using Playwright #119681

Performance Tests for Login and Home Pages by using Playwright #119681

Conversation

suchcodemuchwow commented Nov 24, 2021 • edited Loading

suchcodemuchwow commented Nov 26, 2021

lizozom commented Nov 30, 2021

tylersmalley left a comment

Choose a reason for hiding this comment

LeeDr commented Jan 13, 2022

dmlemeshko commented Jan 13, 2022 • edited Loading

suchcodemuchwow commented Jan 13, 2022 • edited Loading

suchcodemuchwow commented Jan 13, 2022

suchcodemuchwow commented Jan 17, 2022

dmlemeshko left a comment

Choose a reason for hiding this comment

suchcodemuchwow commented Jan 18, 2022

kibana-ci commented Jan 18, 2022

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Jan 18, 2022

💔 All backports failed

How to fix

Questions ?

kibanamachine commented Jan 20, 2022

kibanamachine commented Jan 21, 2022

kibanamachine commented Jan 24, 2022

matschaffer Jun 24, 2022

Choose a reason for hiding this comment

spalger Jun 24, 2022 • edited Loading

Choose a reason for hiding this comment

matschaffer Jun 24, 2022

Choose a reason for hiding this comment

suchcodemuchwow commented Nov 24, 2021 •

edited

Loading

dmlemeshko commented Jan 13, 2022 •

edited

Loading

suchcodemuchwow commented Jan 13, 2022 •

edited

Loading

spalger Jun 24, 2022 •

edited

Loading