[Reporting] Improve functional test steps #63259

tsullivan · 2020-04-10T16:23:52Z

Summary

This PR streamlines Reporting functional tests by using ES-Archiver to load saved dashboards to report against, instead of building a dashboard on-the-fly in the test.

Timing results:
BEFORE

           ?- ? pass  (18.9s) "reporting app Reporting Dashboard Print PDF button is not available if new"
           ?- ? pass  (10.8s) "reporting app Reporting Dashboard Print PDF button becomes available when saved"
           ?- ? pass  (1.0m) "reporting app Reporting Dashboard Print Layout downloads a PDF file"
           ?- ? pass  (15.6s) "reporting app Reporting Dashboard Print PNG button is not available if new"
           ?- ? pass  (11.7s) "reporting app Reporting Dashboard Print PNG button becomes available when saved"
           ?- ? pass  (2.0m) "reporting app Reporting Dashboard Preserve Layout matches baseline report"
           ?- ? pass  (3.7s) "reporting app Reporting Discover Generate CSV button is not available if new"
           ?- ? pass  (5.8s) "reporting app Reporting Discover Generate CSV button becomes available when saved"
           ?- ? pass  (25.3s) "reporting app Reporting Discover Generate CSV button generates a report with data"
           ?- ? pass  (15.3s) "reporting app Reporting Discover Generate CSV button generates a report with no data"
           ?- ? pass  (11.0s) "reporting app Reporting Visualize Print PDF button is not available if new"
           ?- ? pass  (33.2s) "reporting app Reporting Visualize Print PDF button becomes available when saved"
           ?- ? pass  (19.3s) "reporting app Reporting Visualize Print PDF button matches baseline report"
13 passing (8.0m)

AFTER

           ?- ? pass  (25.6s) "reporting app Reporting Dashboard Print PDF button is not available if new"
           ?- ? pass  (11.6s) "reporting app Reporting Dashboard Print PDF button becomes available when saved"
           ?- ? pass  (44.6s) "reporting app Reporting Dashboard Print Layout downloads a PDF file"
           ?- ? pass  (16.0s) "reporting app Reporting Dashboard Print PNG button is not available if new"
           ?- ? pass  (11.4s) "reporting app Reporting Dashboard Print PNG button becomes available when saved"
           ?- ? pass  (36.7s) "reporting app Reporting Dashboard Preserve Layout matches baseline report"
           ?- ? pass  (3.7s) "reporting app Reporting Discover Generate CSV button is not available if new"
           ?- ? pass  (5.9s) "reporting app Reporting Discover Generate CSV button becomes available when saved"
           ?- ? pass  (25.3s) "reporting app Reporting Discover Generate CSV button generates a report with data"
           ?- ? pass  (25.2s) "reporting app Reporting Discover Generate CSV button generates a report with no data"
           ?- ? pass  (10.9s) "reporting app Reporting Visualize Print PDF button is not available if new"
           ?- ? pass  (34.2s) "reporting app Reporting Visualize Print PDF button becomes available when saved"
           ?- ? pass  (35.2s) "reporting app Reporting Visualize Print PDF button downloaded PDF has OK status"
13 passing (5.0m)

NOTE:

'reporting app Reporting Dashboard Print Layout downloads a PDF file' runs as a standalone test now
The PDF files in x-pack/test/reporting/functional/reports/baseline were not used, so I removed them in this PR.

elasticmachine · 2020-04-10T16:23:55Z

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

tsullivan · 2020-04-10T16:39:54Z

x-pack/test/functional/page_objects/reporting_page.js

reporting/historic was not used for anything

tsullivan · 2020-04-10T16:40:34Z

x-pack/test/functional/page_objects/reporting_page.js

A default index is provided by the ES Archive now

tsullivan · 2020-04-10T17:39:26Z

I've added the 7.7.0 version label, as all of these tests are still being skipped in the 7.7 branch and we'd like to re-enable them.

cc @liza-mae

tsullivan · 2020-04-13T16:47:18Z

@elasticmachine merge upstream

kibanamachine · 2020-04-13T18:28:10Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 74439bd

History

💚 Build #40189 succeeded 1c553ac
💚 Build #40170 succeeded e3b81b099b9e3c4d242b456bf03d3a1521cf75ae

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

dmlemeshko

Code LGTM. I started flaky test runner with 30 threads to check if it is stable:
https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/361/
Let's wait for it before merge

* Start of refactoring * remove unused fixtuers * fix the refactoring bugs * update archive with better dashboard * important comment Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* master: (29 commits) Add test:jest_integration npm script (elastic#62938) [data.search.aggs] Remove service getters from agg types (AggConfig part) (elastic#62548) [Discover] Fix broken setting of bucketInterval (elastic#62939) Disable adding conditions when in alert management context. (elastic#63514) [Alerting] fixes to allow pre-configured actions to be executed (elastic#63432) adding useMemo (elastic#63504) [Maps] fix double fetch when filter pill is added (elastic#63024) [Lens] Fix missing formatting bug in "break down by" (elastic#63288) [SIEM] [Cases] Removed double pasted line (elastic#63507) [Reporting] Improve functional test steps (elastic#63259) [SIEM][CASE] Tests for server's configuration API (elastic#63099) [SIEM] [Cases] Case container unit tests (elastic#63376) [ML] Improving parsing of large uploaded files (elastic#62970) [ML] Listing global calendars on the job management page (elastic#63124) [Ingest][Endpoint] Add Ingest rest api response types for use in Endpoint (elastic#63373) Add help text to form fields (elastic#63165) [ML] Converts utils Mocha tests to Jest (elastic#63132) [Metrics UI] Refactor With* containers to hooks (elastic#59503) [NP] Migrate logstash server side code to NP (elastic#63135) Clicking cancel in saved query save modal doesn't close it (elastic#62774) ...

* Start of refactoring * remove unused fixtuers * fix the refactoring bugs * update archive with better dashboard * important comment Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* Revert "FTR: add chromium-based Edge browser support (#61684) (#63068)" This reverts commit e8caa06. * update lock file * [Reporting] Improve functional test steps (#63259) * Start of refactoring * remove unused fixtuers * fix the refactoring bugs * update archive with better dashboard * important comment Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> * get tests to pass * get more tests to pass * reduce branch diffs * reduce loc change Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* resolve conflicts * get tests to pass * get more tests to pass * reduce branch diffs Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

tsullivan added review v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Reporting Services v7.8.0 labels Apr 10, 2020

tsullivan requested a review from joelgriffith April 10, 2020 16:23

tsullivan added 4 commits April 10, 2020 09:37

Start of refactoring

19f6e6c

remove unused fixtuers

348cb8a

fix the refactoring bugs

9258566

update archive with better dashboard

a155e75

tsullivan force-pushed the reporting/fn-test-better branch from e3b81b0 to 2d45a68 Compare April 10, 2020 16:38

tsullivan commented Apr 10, 2020

View reviewed changes

x-pack/test/functional/page_objects/reporting_page.js Outdated

Copy link

Member Author

tsullivan Apr 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reporting/historic was not used for anything

tsullivan commented Apr 10, 2020

View reviewed changes

x-pack/test/functional/page_objects/reporting_page.js Outdated

Copy link

Member Author

tsullivan Apr 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A default index is provided by the ES Archive now

important comment

1c553ac

tsullivan force-pushed the reporting/fn-test-better branch from 2d45a68 to 1c553ac Compare April 10, 2020 16:41

tsullivan added the v7.7.0 label Apr 10, 2020

Merge branch 'master' into reporting/fn-test-better

74439bd

tsullivan requested a review from liza-mae April 13, 2020 16:52

dmlemeshko approved these changes Apr 14, 2020

View reviewed changes

tsullivan merged commit 29aba88 into elastic:master Apr 14, 2020

This was referenced Apr 14, 2020

[7.x] [Reporting] Improve functional test steps (#63259) #63519

Merged

[7.7] [Reporting] Improve functional test steps (#63259) #63520

Merged

tsullivan deleted the reporting/fn-test-better branch June 25, 2020 18:08

sophiec20 added the zDeprecated Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead label Aug 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Reporting] Improve functional test steps #63259

[Reporting] Improve functional test steps #63259

Uh oh!

tsullivan commented Apr 10, 2020 •

edited

Loading

Uh oh!

elasticmachine commented Apr 10, 2020

Uh oh!

tsullivan Apr 10, 2020

Uh oh!

tsullivan Apr 10, 2020

Uh oh!

tsullivan commented Apr 10, 2020

Uh oh!

tsullivan commented Apr 13, 2020

Uh oh!

kibanamachine commented Apr 13, 2020

Uh oh!

dmlemeshko left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Reporting] Improve functional test steps #63259

[Reporting] Improve functional test steps #63259

Uh oh!

Conversation

tsullivan commented Apr 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

elasticmachine commented Apr 10, 2020

Uh oh!

tsullivan Apr 10, 2020

Choose a reason for hiding this comment

Uh oh!

tsullivan Apr 10, 2020

Choose a reason for hiding this comment

Uh oh!

tsullivan commented Apr 10, 2020

Uh oh!

tsullivan commented Apr 13, 2020

Uh oh!

kibanamachine commented Apr 13, 2020

💚 Build Succeeded

History

Uh oh!

dmlemeshko left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tsullivan commented Apr 10, 2020 •

edited

Loading