Comprehensive dashboard a11y tests #57821

bhavyarm · 2020-02-17T21:22:19Z

Adding more a11y tests to dashboard app for menu options in edit and view mode

kibanamachine · 2020-02-20T01:20:51Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 056fb5a

Test Failures

Kibana Pipeline / kibana-oss-agent / Accessibility Tests.test/accessibility/apps/dashboard·ts.Dashboard add a saved search

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:03:18]         └-: Dashboard
[00:03:18]           └-> "before all" hook
[00:03:18]           └-> "before all" hook
[00:03:18]             │ debg Navigate to: http://localhost:61131/app/kibana#/home/tutorial_directory/sampleData
[00:03:18]             │ debg browser[INFO] http://localhost:61131/app/kibana?_t=1582155741697#/home/tutorial_directory/sampleData 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:03:18]             │
[00:03:18]             │ debg browser[INFO] http://localhost:61131/bundles/app/kibana/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:03:18]             │ debg currentUrl = http://localhost:61131/app/kibana#/home/tutorial_directory/sampleData
[00:03:18]             │          appUrl = http://localhost:61131/app/kibana#/home/tutorial_directory/sampleData
[00:03:18]             │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:03:22]             │ debg TestSubjects.exists(addSampleDataSetflights)
[00:03:22]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=2500
[00:03:22]             │ debg browser[INFO] http://localhost:61131/bundles/plugin/data/data.plugin.js 67:139970 "INFO: 2020-02-19T23:42:24Z
[00:03:22]             │        Adding connection to http://localhost:61131/elasticsearch
[00:03:22]             │
[00:03:22]             │      "
[00:03:22]             │ debg TestSubjects.click(addSampleDataSetflights)
[00:03:22]             │ debg Find.clickByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=10000
[00:03:22]             │ debg Find.findByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=10000
[00:03:23]             │ debg TestSubjects.find(sampleDataSetCardflights)
[00:03:23]             │ debg Find.findByCssSelector('[data-test-subj="sampleDataSetCardflights"]') with timeout=10000
[00:03:23]             │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1582153959667121426] [kibana_sample_data_flights] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:03:23]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1582153959667121426] updating number_of_replicas to [0] for indices [kibana_sample_data_flights]
[00:03:23]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1582153959667121426] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[kibana_sample_data_flights][0]]]).
[00:03:27]             │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-tests-xl-1582153959667121426] [.kibana_2/-4xWwYPBRuuC6Mj63HpIfQ] update_mapping [_doc]
[00:03:27]             │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-tests-xl-1582153959667121426] [.kibana_2/-4xWwYPBRuuC6Mj63HpIfQ] update_mapping [_doc]
[00:03:29]           └-> dashboard
[00:03:29]             └-> "before each" hook: global before each
[00:03:29]             │ debg navigating to dashboard url: http://localhost:61131/app/kibana#/dashboards
[00:03:29]             │ debg Navigate to: http://localhost:61131/app/kibana#/dashboards
[00:03:29]             │ debg ... sleep(700) start
[00:03:29]             │ debg browser[INFO] http://localhost:61131/app/kibana?_t=1582155752836#/dashboards 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:03:29]             │
[00:03:29]             │ debg browser[INFO] http://localhost:61131/bundles/app/kibana/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:03:30]             │ debg ... sleep(700) end
[00:03:30]             │ debg returned from get, calling refresh
[00:03:30]             │ debg browser[INFO] http://localhost:61131/app/kibana?_t=1582155752836#/dashboards 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:03:30]             │
[00:03:30]             │ debg browser[INFO] http://localhost:61131/bundles/app/kibana/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:03:30]             │ debg currentUrl = http://localhost:61131/app/kibana#/dashboards
[00:03:30]             │          appUrl = http://localhost:61131/app/kibana#/dashboards
[00:03:30]             │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:03:32]             │ debg TestSubjects.find(kibanaChrome)
[00:03:32]             │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=10000
[00:03:32]             │ debg browser[INFO] http://localhost:61131/bundles/plugin/data/data.plugin.js 67:139970 "INFO: 2020-02-19T23:42:35Z
[00:03:32]             │        Adding connection to http://localhost:61131/elasticsearch
[00:03:32]             │
[00:03:32]             │      "
[00:03:32]             │ debg ... sleep(501) start
[00:03:33]             │ debg ... sleep(501) end
[00:03:33]             │ debg in navigateTo url = http://localhost:61131/app/kibana#/dashboards?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))
[00:03:33]             │ debg --- retry.try error: URL changed, waiting for it to settle
[00:03:33]             │ debg ... sleep(501) start
[00:03:34]             │ debg ... sleep(501) end
[00:03:34]             │ debg in navigateTo url = http://localhost:61131/app/kibana#/dashboards?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))
[00:03:34]             │ debg TestSubjects.exists(statusPageContainer)
[00:03:34]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:03:36]             │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:03:37]             └- ✓ pass  (8.2s) "Dashboard dashboard"
[00:03:37]           └-> create dashboard button
[00:03:37]             └-> "before each" hook: global before each
[00:03:37]             │ debg TestSubjects.exists(newItemButton)
[00:03:37]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="newItemButton"]') with timeout=2500
[00:03:37]             │ debg TestSubjects.click(newItemButton)
[00:03:37]             │ debg Find.clickByCssSelector('[data-test-subj="newItemButton"]') with timeout=10000
[00:03:37]             │ debg Find.findByCssSelector('[data-test-subj="newItemButton"]') with timeout=10000
[00:03:38]             └- ✓ pass  (941ms) "Dashboard create dashboard button"
[00:03:38]           └-> save empty dashboard
[00:03:38]             └-> "before each" hook: global before each
[00:03:38]             │ debg TestSubjects.click(dashboardSaveMenuItem)
[00:03:38]             │ debg Find.clickByCssSelector('[data-test-subj="dashboardSaveMenuItem"]') with timeout=10000
[00:03:38]             │ debg Find.findByCssSelector('[data-test-subj="dashboardSaveMenuItem"]') with timeout=10000
[00:03:38]             │ debg TestSubjects.find(savedObjectSaveModal)
[00:03:38]             │ debg Find.findByCssSelector('[data-test-subj="savedObjectSaveModal"]') with timeout=10000
[00:03:38]             │ debg entering new title
[00:03:38]             │ debg TestSubjects.setValue(savedObjectTitle, Dashboard Listing A11y)
[00:03:38]             │ debg TestSubjects.click(savedObjectTitle)
[00:03:38]             │ debg Find.clickByCssSelector('[data-test-subj="savedObjectTitle"]') with timeout=10000
[00:03:38]             │ debg Find.findByCssSelector('[data-test-subj="savedObjectTitle"]') with timeout=10000
[00:03:38]             │ debg DashboardPage.clickSave
[00:03:38]             │ debg TestSubjects.click(confirmSaveSavedObjectButton)
[00:03:38]             │ debg Find.clickByCssSelector('[data-test-subj="confirmSaveSavedObjectButton"]') with timeout=10000
[00:03:38]             │ debg Find.findByCssSelector('[data-test-subj="confirmSaveSavedObjectButton"]') with timeout=10000
[00:03:39]             │ debg Find.waitForElementStale with timeout=10000
[00:03:39]             │ debg TestSubjects.exists(saveDashboardSuccess)
[00:03:39]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="saveDashboardSuccess"]') with timeout=120000
[00:03:39]             │ debg Find.findByCssSelector('.euiToast') with timeout=20000
[00:03:39]             │ debg Find.findByCssSelector('.euiToastHeader__title') with timeout=10000
[00:03:39]             │ debg Toast title: Dashboard 'Dashboard Listing A11y' was saved
[00:03:39]             │ debg Find.clickByCssSelector('.euiToast__closeButton') with timeout=10000
[00:03:39]             │ debg Find.findByCssSelector('.euiToast__closeButton') with timeout=10000
[00:03:39]             │ debg isGlobalLoadingIndicatorVisible
[00:03:39]             │ debg TestSubjects.exists(globalLoadingIndicator)
[00:03:39]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:03:41]             │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:03:41]             │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:03:41]             │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:03:41]             │ debg Waiting for save modal to close
[00:03:41]             │ debg TestSubjects.exists(savedObjectSaveModal)
[00:03:41]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="savedObjectSaveModal"]') with timeout=2500
[00:03:44]             │ debg --- retry.tryForTime error: [data-test-subj="savedObjectSaveModal"] is not displayed
[00:03:45]             └- ✓ pass  (6.5s) "Dashboard save empty dashboard"
[00:03:45]           └-> Open Edit mode
[00:03:45]             └-> "before each" hook: global before each
[00:03:45]             │ debg Switching to edit mode
[00:03:45]             │ debg TestSubjects.click(dashboardEditMode)
[00:03:45]             │ debg Find.clickByCssSelector('[data-test-subj="dashboardEditMode"]') with timeout=10000
[00:03:45]             │ debg Find.findByCssSelector('[data-test-subj="dashboardEditMode"]') with timeout=10000
[00:03:45]             │ debg Waiting up to 20000ms for in edit mode...
[00:03:45]             │ debg TestSubjects.findAll(embeddablePanel)
[00:03:45]             │ debg Find.allByCssSelector('[data-test-subj="embeddablePanel"]') with timeout=2500
[00:03:47]             │ debg TestSubjects.findAll(embeddablePanelToggleMenuIcon)
[00:03:47]             │ debg Find.allByCssSelector('[data-test-subj="embeddablePanelToggleMenuIcon"]') with timeout=2500
[00:03:50]             └- ✓ pass  (5.6s) "Dashboard Open Edit mode"
[00:03:50]           └-> Open add panel
[00:03:50]             └-> "before each" hook: global before each
[00:03:50]             │ debg DashboardAddPanel.clickOpenAddPanel
[00:03:50]             │ debg TestSubjects.click(dashboardAddPanelButton)
[00:03:50]             │ debg Find.clickByCssSelector('[data-test-subj="dashboardAddPanelButton"]') with timeout=10000
[00:03:50]             │ debg Find.findByCssSelector('[data-test-subj="dashboardAddPanelButton"]') with timeout=10000
[00:03:50]             │ debg ... sleep(500) start
[00:03:51]             │ debg ... sleep(500) end
[00:03:51]             └- ✓ pass  (1.2s) "Dashboard Open add panel"
[00:03:51]           └-> add a visualization
[00:03:51]             └-> "before each" hook: global before each
[00:03:51]             │ debg TestSubjects.click(savedObjectTitle[Flights]-Delay-Buckets)
[00:03:51]             │ debg Find.clickByCssSelector('[data-test-subj="savedObjectTitle[Flights]-Delay-Buckets"]') with timeout=10000
[00:03:51]             │ debg Find.findByCssSelector('[data-test-subj="savedObjectTitle[Flights]-Delay-Buckets"]') with timeout=10000
[00:03:52]             └- ✓ pass  (669ms) "Dashboard add a visualization"
[00:03:52]           └-> add a saved search
[00:03:52]             └-> "before each" hook: global before each
[00:03:52]             │ debg DashboardAddPanel.addEmbeddable, name: [Flights] Flight Log, type: search
[00:03:52]             │ debg DashboardAddPanel.ensureAddPanelIsShowing
[00:03:52]             │ debg DashboardAddPanel.isAddPanelOpen
[00:03:52]             │ debg TestSubjects.exists(dashboardAddPanel)
[00:03:52]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="dashboardAddPanel"]') with timeout=2500
[00:03:52]             │ debg DashboardAddPanel.addToFilter(search)
[00:03:52]             │ debg Find.waitForDeletedByCssSelector('[data-test-subj="savedObjectFinderLoadingIndicator"]') with timeout=10000
[00:03:53]             │ debg DashboardAddPanel.toggleFilter
[00:03:53]             │ debg TestSubjects.click(savedObjectFinderFilterButton)
[00:03:53]             │ debg Find.clickByCssSelector('[data-test-subj="savedObjectFinderFilterButton"]') with timeout=10000
[00:03:53]             │ debg Find.findByCssSelector('[data-test-subj="savedObjectFinderFilterButton"]') with timeout=10000
[00:03:53]             │ debg TestSubjects.click(savedObjectFinderFilter-search)
[00:03:53]             │ debg Find.clickByCssSelector('[data-test-subj="savedObjectFinderFilter-search"]') with timeout=10000
[00:03:53]             │ debg Find.findByCssSelector('[data-test-subj="savedObjectFinderFilter-search"]') with timeout=10000
[00:03:53]             │ debg DashboardAddPanel.toggleFilter
[00:03:53]             │ debg TestSubjects.click(savedObjectFinderFilterButton)
[00:03:53]             │ debg Find.clickByCssSelector('[data-test-subj="savedObjectFinderFilterButton"]') with timeout=10000
[00:03:53]             │ debg Find.findByCssSelector('[data-test-subj="savedObjectFinderFilterButton"]') with timeout=10000
[00:03:53]             │ debg Find.waitForDeletedByCssSelector('[data-test-subj="savedObjectFinderLoadingIndicator"]') with timeout=10000
[00:03:53]             │ debg TestSubjects.setValue(savedObjectFinderSearchInput, "[Flights] Flight Log")
[00:03:53]             │ debg TestSubjects.click(savedObjectFinderSearchInput)
[00:03:53]             │ debg Find.clickByCssSelector('[data-test-subj="savedObjectFinderSearchInput"]') with timeout=10000
[00:03:53]             │ debg Find.findByCssSelector('[data-test-subj="savedObjectFinderSearchInput"]') with timeout=10000
[00:03:54]             │ debg Find.waitForDeletedByCssSelector('[data-test-subj="savedObjectFinderLoadingIndicator"]') with timeout=10000
[00:03:54]             │ debg TestSubjects.click(savedObjectTitle[Flights]-Flight-Log)
[00:03:54]             │ debg Find.clickByCssSelector('[data-test-subj="savedObjectTitle[Flights]-Flight-Log"]') with timeout=10000
[00:03:54]             │ debg Find.findByCssSelector('[data-test-subj="savedObjectTitle[Flights]-Flight-Log"]') with timeout=10000
[00:03:54]             │ debg TestSubjects.exists(addObjectToDashboardSuccess)
[00:03:54]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="addObjectToDashboardSuccess"]') with timeout=2500
[00:03:57]             │ debg --- retry.tryForTime error: [data-test-subj="addObjectToDashboardSuccess"] is not displayed
[00:03:57]             │ debg TestSubjects.exists(dashboardAddPanel)
[00:03:57]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="dashboardAddPanel"]') with timeout=1000
[00:03:57]             │ debg Closing flyout dashboardAddPanel
[00:03:57]             │ debg TestSubjects.find(dashboardAddPanel)
[00:03:57]             │ debg Find.findByCssSelector('[data-test-subj="dashboardAddPanel"]') with timeout=10000
[00:03:58]             │ debg Waiting up to 20000ms for flyout closed...
[00:03:58]             │ debg TestSubjects.exists(dashboardAddPanel)
[00:03:58]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="dashboardAddPanel"]') with timeout=1000
[00:03:59]             │ debg --- retry.tryForTime error: [data-test-subj="dashboardAddPanel"] is not displayed
[00:04:00]             │ info Taking screenshot "/dev/shm/workspace/kibana/test/functional/screenshots/failure/Dashboard add a saved search.png"
[00:04:00]             │ info Current URL is: http://localhost:61131/app/kibana#/dashboard/858c0f60-5371-11ea-a38a-77f739ab448b?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(description:%27%27,filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),panels:!((embeddableConfig:(),gridData:(h:15,i:%2772b4cbb1-85d5-42ab-adfd-a8f56420d17b%27,w:24,x:0,y:0),id:%279886b410-4c8b-11e8-b3d7-01146121b73d%27,panelIndex:%2772b4cbb1-85d5-42ab-adfd-a8f56420d17b%27,type:visualization,version:%278.0.0-SNAPSHOT%27),(embeddableConfig:(),gridData:(h:15,i:a5cdd14c-79f1-41ae-843e-f61e1b3feb26,w:24,x:24,y:0),id:%27571aaf70-4c88-11e8-b3d7-01146121b73d%27,panelIndex:a5cdd14c-79f1-41ae-843e-f61e1b3feb26,type:search,version:%278.0.0-SNAPSHOT%27)),query:(language:kuery,query:%27%27),timeRestore:!f,title:%27Dashboard%20Listing%20A11y%27,viewMode:edit)
[00:04:01]             │ info Saving page source to: /dev/shm/workspace/kibana/test/functional/failure_debug/html/Dashboard add a saved search.html
[00:04:01]             └- ✖ fail: "Dashboard add a saved search"
[00:04:01]             │

Stack Trace

Error: a11y report:

VIOLATION
  [aria-hidden-focus]: Ensures aria-hidden elements do not contain focusable elements
    Help: https://dequeuniversity.com/rules/axe/3.3/aria-hidden-focus?application=axeAPI
    Elements:
      - .visChart

VIOLATION
  [button-name]: Ensures buttons have discernible text
    Help: https://dequeuniversity.com/rules/axe/3.3/button-name?application=axeAPI
    Elements:
      - .kbnDocTable__bar > .kuiBarSection > tool-bar-pager-buttons > .kuiButtonGroup > .kuiButton.kuiButton--basic[data-test-subj="btnPrevPage"]
      - .kbnDocTable__bar > .kuiBarSection > tool-bar-pager-buttons > .kuiButtonGroup > .kuiButton.kuiButton--basic[data-test-subj="btnNextPage"]
      - .kbnDocTable__bar--footer > .kuiBarSection > tool-bar-pager-buttons > .kuiButtonGroup > .kuiButton.kuiButton--basic[data-test-subj="btnPrevPage"]
      - .kbnDocTable__bar--footer > .kuiBarSection > tool-bar-pager-buttons > .kuiButtonGroup > .kuiButton.kuiButton--basic[data-test-subj="btnNextPage"]
    at Accessibility.testAxeReport (test/accessibility/services/a11y/a11y.ts:85:15)
    at Accessibility.testAppSnapshot (test/accessibility/services/a11y/a11y.ts:58:18)
    at process._tickCallback (internal/process/next_tick.js:68:7)

Kibana Pipeline / kibana-xpack-agent / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/rollup_job/tsvb·js.rollup app tsvb integration create rollup tsvb

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 10 times on tracked branches: https://github.com/elastic/kibana/issues/56816

[00:00:00]       │
[00:36:11]         └-: rollup app
[00:36:11]           └-> "before all" hook
[00:37:42]           └-: tsvb integration
[00:37:42]             └-> "before all" hook
[00:37:42]             └-> "before all" hook
[00:37:42]               │ info [visualize/default] Loading "mappings.json"
[00:37:42]               │ info [visualize/default] Loading "data.json"
[00:37:42]               │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana_2/DauEaoDiSLCE_eXFczcdtw] deleting index
[00:37:42]               │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana_1/LunmgIghRrSzeRhuFJkRXw] deleting index
[00:37:42]               │ info [visualize/default] Deleted existing index [".kibana_2",".kibana_1"]
[00:37:42]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana] creating index, cause [api], templates [], shards [1]/[0], mappings [_doc]
[00:37:42]               │ info [visualize/default] Created index ".kibana"
[00:37:42]               │ debg [visualize/default] ".kibana" settings {"index":{"number_of_shards":"1","auto_expand_replicas":"0-1","number_of_replicas":"0"}}
[00:37:42]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [metricbeat-7] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[00:37:42]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [metricbeat-7/Joe4IfksQya2HwhbgpgZoQ] create_mapping
[00:37:42]               │ info [visualize/default] Indexed 8 docs into ".kibana"
[00:37:42]               │ info [visualize/default] Indexed 1 docs into "metricbeat-7"
[00:37:43]               │ info Creating index .kibana_2.
[00:37:43]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana_2] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:37:43]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] updating number_of_replicas to [0] for indices [.kibana_2]
[00:37:43]               │ info Reindexing .kibana to .kibana_1
[00:37:43]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:37:43]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] updating number_of_replicas to [0] for indices [.kibana_1]
[00:37:43]               │ info [o.e.t.LoggingTaskListener] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] 71880 finished with response BulkByScrollResponse[took=99ms,timed_out=false,sliceId=null,updated=0,created=8,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[00:37:44]               │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana/9D2daRDHTdWA0AwuVBucFA] deleting index
[00:37:44]               │ info Migrating .kibana_1 saved objects to .kibana_2
[00:37:44]               │ debg Migrating saved objects space:default, index-pattern:metricbeat-*, custom-space:index-pattern:metricbeat-*, index-pattern:logstash-*, custom_space:index-pattern:logstash-*, visualization:i-exist, custom_space:visualization:i-exist, query:okjpgs
[00:37:44]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana_2/C5PfZGLiQ-WqTkwgNh2LhA] update_mapping [_doc]
[00:37:44]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana_2/C5PfZGLiQ-WqTkwgNh2LhA] update_mapping [_doc]
[00:37:44]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana_2/C5PfZGLiQ-WqTkwgNh2LhA] update_mapping [_doc]
[00:37:44]               │ info Pointing alias .kibana to .kibana_2.
[00:37:44]               │ info Finished in 835ms.
[00:37:44]               │ debg applying update to kibana config: {"accessibility:disableAnimations":true,"dateFormat:tz":"UTC"}
[00:37:44]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana_2/C5PfZGLiQ-WqTkwgNh2LhA] update_mapping [_doc]
[00:37:45]             └-> create rollup tsvb
[00:37:45]               └-> "before each" hook: global before each
[00:37:45]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [rollup-source-data] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[00:37:45]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [rollup-source-data/G1Zz0OtXSEif50UmOVy3VA] create_mapping
[00:37:45]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [rollup-target-data] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:37:45]               │ debg navigating to visualize url: http://localhost:6111/app/kibana#/visualize
[00:37:45]               │ debg Navigate to: http://localhost:6111/app/kibana#/visualize
[00:37:45]               │ info [o.e.x.r.j.RollupJobTask] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] Rollup job [tsvb-test-rollup-job-1582156283921] created.
[00:37:45]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [rollup-target-data/S8EiqPwaTfKsk5-DHDW4dA] update_mapping [_doc]
[00:37:46]               │ debg ... sleep(700) start
[00:37:46]               │ debg browser[INFO] http://localhost:6111/app/kibana?_t=1582158550811#/visualize 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:37:46]               │
[00:37:46]               │ debg browser[INFO] http://localhost:6111/bundles/app/kibana/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:37:46]               │ debg ... sleep(700) end
[00:37:46]               │ debg returned from get, calling refresh
[00:37:47]               │ debg browser[INFO] http://localhost:6111/app/kibana?_t=1582158550811#/visualize 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:37:47]               │
[00:37:47]               │ debg browser[INFO] http://localhost:6111/bundles/app/kibana/bootstrap.js 9:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:37:48]               │ debg currentUrl = http://localhost:6111/app/kibana#/visualize
[00:37:48]               │          appUrl = http://localhost:6111/app/kibana#/visualize
[00:37:48]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:37:57]               │ debg TestSubjects.find(kibanaChrome)
[00:37:57]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=10000
[00:37:57]               │ debg browser[INFO] http://localhost:6111/bundles/plugin/data/data.plugin.js 67:139970 "INFO: 2020-02-20T00:29:19Z
[00:37:57]               │        Adding connection to http://localhost:6111/elasticsearch
[00:37:57]               │
[00:37:57]               │      "
[00:37:57]               │ debg ... sleep(501) start
[00:37:57]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1582153959657891803] [.kibana_2/C5PfZGLiQ-WqTkwgNh2LhA] update_mapping [_doc]
[00:37:58]               │ debg ... sleep(501) end
[00:37:58]               │ debg in navigateTo url = http://localhost:6111/app/kibana#/visualize?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))
[00:37:58]               │ debg TestSubjects.exists(statusPageContainer)
[00:37:58]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:38:00]               │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:38:01]               │ debg TestSubjects.exists(newItemButton)
[00:38:01]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="newItemButton"]') with timeout=2500
[00:38:01]               │ debg TestSubjects.click(newItemButton)
[00:38:01]               │ debg Find.clickByCssSelector('[data-test-subj="newItemButton"]') with timeout=10000
[00:38:01]               │ debg Find.findByCssSelector('[data-test-subj="newItemButton"]') with timeout=10000
[00:38:01]               │ debg TestSubjects.find(visNewDialogTypes)
[00:38:01]               │ debg Find.findByCssSelector('[data-test-subj="visNewDialogTypes"]') with timeout=10000
[00:38:01]               │ debg TestSubjects.click(visType-metrics)
[00:38:01]               │ debg Find.clickByCssSelector('[data-test-subj="visType-metrics"]') with timeout=10000
[00:38:01]               │ debg Find.findByCssSelector('[data-test-subj="visType-metrics"]') with timeout=10000
[00:38:01]               │ debg isGlobalLoadingIndicatorVisible
[00:38:01]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:38:01]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:38:03]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:38:04]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:38:04]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:38:04]               │ debg TestSubjects.exists(tvbVisEditor)
[00:38:04]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="tvbVisEditor"]') with timeout=10000
[00:38:04]               │ debg openQuickSelectTimeMenu
[00:38:04]               │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:38:04]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=2500
[00:38:06]               │ debg --- retry.tryForTime error: [data-test-subj="superDatePickerQuickMenu"] is not displayed
[00:38:07]               │ debg opening quick select menu
[00:38:07]               │ debg TestSubjects.click(superDatePickerToggleQuickMenuButton)
[00:38:07]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:38:07]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:38:07]               │ debg TestSubjects.click(superDatePickerCommonlyUsed_Last_24 hours)
[00:38:07]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerCommonlyUsed_Last_24 hours"]') with timeout=10000
[00:38:07]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerCommonlyUsed_Last_24 hours"]') with timeout=10000
[00:38:07]               │ debg TestSubjects.find(metricTsvbTypeBtn)
[00:38:07]               │ debg Find.findByCssSelector('[data-test-subj="metricTsvbTypeBtn"]') with timeout=10000
[00:38:07]               │ debg TestSubjects.exists(tsvbMetricValue)
[00:38:07]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="tsvbMetricValue"]') with timeout=10000
[00:38:08]               │ debg TestSubjects.click(metricEditorPanelOptionsBtn)
[00:38:08]               │ debg Find.clickByCssSelector('[data-test-subj="metricEditorPanelOptionsBtn"]') with timeout=10000
[00:38:08]               │ debg Find.findByCssSelector('[data-test-subj="metricEditorPanelOptionsBtn"]') with timeout=10000
[00:38:08]               │ debg isGlobalLoadingIndicatorVisible
[00:38:08]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:38:08]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:38:10]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:38:10]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:38:10]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:38:10]               │ debg TestSubjects.find(metricsIndexPatternInput)
[00:38:10]               │ debg Find.findByCssSelector('[data-test-subj="metricsIndexPatternInput"]') with timeout=10000
[00:38:11]               │ debg isGlobalLoadingIndicatorVisible
[00:38:11]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:38:11]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:38:12]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:38:13]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:38:13]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:38:13]               │ debg TestSubjects.find(metricsIndexPatternInterval)
[00:38:13]               │ debg Find.findByCssSelector('[data-test-subj="metricsIndexPatternInterval"]') with timeout=10000
[00:38:13]               │ debg isGlobalLoadingIndicatorVisible
[00:38:13]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:38:13]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:38:14]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:38:15]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:38:15]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:38:15]               │ debg TestSubjects.find(metricsDropLastBucket-no)
[00:38:15]               │ debg Find.findByCssSelector('[data-test-subj="metricsDropLastBucket-no"]') with timeout=10000
[00:38:15]               │ debg isGlobalLoadingIndicatorVisible
[00:38:15]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:38:15]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:38:17]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:38:17]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:38:17]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:38:17]               │ debg ... sleep(3000) start
[00:38:20]               │ debg ... sleep(3000) end
[00:38:20]               │ debg Waiting up to 20000ms for rendering count to stabilize...
[00:38:20]               │ debg TestSubjects.find(visualizationLoader)
[00:38:20]               │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
[00:38:20]               │ debg -- firstCount=7
[00:38:20]               │ debg ... sleep(1000) start
[00:38:21]               │ debg ... sleep(1000) end
[00:38:21]               │ debg TestSubjects.find(visualizationLoader)
[00:38:21]               │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
[00:38:21]               │ debg -- secondCount=7
[00:38:21]               │ debg Find.findByCssSelector('.tvbVisMetric__value--primary') with timeout=10000
[00:38:21]               │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional/screenshots/failure/rollup app tsvb integration create rollup tsvb.png"
[00:38:21]               │ info Current URL is: http://localhost:6111/app/kibana#/visualize/create?type=metrics&_g=(refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now))&_a=(filters:!(),linked:!f,query:(language:kuery,query:%27%27),uiState:(),vis:(aggs:!(),params:(axis_formatter:number,axis_position:left,axis_scale:normal,background_color_rules:!((id:%2710853e10-5378-11ea-99c3-c146095111e7%27)),default_index_pattern:%27metricbeat-*%27,default_timefield:%27@timestamp%27,drop_last_bucket:0,id:%2761ca57f0-469d-11e7-af02-69e470af7417%27,index_pattern:rollup-target-data,interval:%271d%27,isModelInvalid:!f,series:!((axis_position:right,chart_type:line,color:%2368BC00,fill:0.5,formatter:number,id:%2761ca57f1-469d-11e7-af02-69e470af7417%27,line_width:1,metrics:!((id:%2761ca57f2-469d-11e7-af02-69e470af7417%27,type:count)),point_size:1,separate_axis:0,split_mode:everything,stacked:none)),show_grid:1,show_legend:1,time_field:%27%27,type:metric),title:%27%27,type:metrics))
[00:38:22]               │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional/failure_debug/html/rollup app tsvb integration create rollup tsvb.html
[00:38:22]               └- ✖ fail: "rollup app tsvb integration create rollup tsvb"
[00:38:22]               │

Stack Trace

{ Error: expected '0' to sort of equal '3'
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.it (test/functional/apps/rollup_job/tsvb.js:90:27)
    at process._tickCallback (internal/process/next_tick.js:68:7) actual: '0', expected: '3', showDiff: true }

History

💔 Build #27229 failed d81a0fc

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

majagrubic · 2020-02-20T08:13:25Z

test/accessibility/apps/dashboard.ts

-        defaultIndex: 'logstash-*',
-      });
-      await PageObjects.common.navigateToApp('dashboard');
+      // await esArchiver.loadIfNeeded('logstash_functional');


Why is this commented out?

Earlier round of phase 1 tests used logstash data. I removed it and using sample data. Removed the commented out lines. Thanks @majagrubic

majagrubic · 2020-02-20T09:32:51Z

I had a quick look at the failures, and a lot of them are due to elements in visualize/discover/eui. I can fix some of those, but want to make sure someone is not already tackling that, so we don't duplicate the work.

dmlemeshko

Awesome work! I left a small comment, otherwise good to go.

dmlemeshko · 2020-02-20T12:52:07Z

test/accessibility/apps/dashboard.ts

+    it('Exit out of full screen mode', async () => {
+      const logoButton = await PageObjects.dashboard.getExitFullScreenLogoButton();
+      await logoButton.moveMouseTo();
+      await PageObjects.dashboard.clickExitFullScreenTextButton();


I have seen the same 3 lines of code in one of the full_screen_mode.js tests.
How about we wrap it up in a function placing in dashboard_page.ts:

async function exitFullScreenMode() { const logoButton = await this.getExitFullScreenLogoButton(); await logoButton.moveMouseTo(); await this.clickExitFullScreenTextButton(); }

If you agree, please update the other test as well

bhavyarm · 2020-02-21T17:55:50Z

@majagrubic @dmlemeshko thank you. I have addressed the review comments!

bhavyarm · 2020-02-27T12:25:50Z

@elasticmachine merge upstream

myasonik · 2020-03-02T21:14:06Z

@bhavyarm, @majagrubic merged her PR (#58122) which I think superseded this one. Will you take a look to make sure everything you wanted is in master?

If so, I think you can close this PR.

bhavyarm · 2020-03-03T21:43:32Z

@myasonik thanks. @majagrubic pr is missing one of the test changes I made in a separate test because of Dima's comment(test/functional/apps/dashboard/full_screen_mode.js ). I am going to close this PR - raise a separate one to address it. Thanks!

adding comprehensive dashboard tests

d81a0fc

bhavyarm self-assigned this Feb 17, 2020

bhavyarm requested review from dmlemeshko and myasonik February 17, 2020 21:22

bhavyarm mentioned this pull request Feb 17, 2020

[meta] Accessibility tests #51456

Open

77 tasks

bhavyarm added Project:Accessibility release_note:skip Skip the PR/issue when compiling release notes v8.0.0 labels Feb 17, 2020

bhavyarm added 2 commits February 19, 2020 15:10

Merge branch 'master' into a11y2Dashboard

45715b0

fixing delete and adding dima changes

056fb5a

bhavyarm marked this pull request as ready for review February 19, 2020 23:13

majagrubic reviewed Feb 20, 2020

View reviewed changes

dmlemeshko reviewed Feb 20, 2020

View reviewed changes

majagrubic mentioned this pull request Feb 20, 2020

Dashboard a11y tests #58122

Merged

7 tasks

addressing review comments

cab78c9

myasonik mentioned this pull request Feb 21, 2020

(Accessibility) Skip links to main content #38980

Closed

Merge branch 'master' into a11y2Dashboard

47980f9

bhavyarm closed this Mar 3, 2020

bhavyarm deleted the a11y2Dashboard branch March 3, 2020 22:25

bhavyarm mentioned this pull request Mar 3, 2020

Calling exitFullScreenMode() from fullscreenmode test #59238

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comprehensive dashboard a11y tests #57821

Comprehensive dashboard a11y tests #57821

bhavyarm commented Feb 17, 2020

kibanamachine commented Feb 20, 2020

Standard Out

Stack Trace

Standard Out

Stack Trace

majagrubic Feb 20, 2020

bhavyarm Feb 21, 2020

majagrubic commented Feb 20, 2020

dmlemeshko left a comment

dmlemeshko Feb 20, 2020 •

edited

Loading

bhavyarm commented Feb 21, 2020

bhavyarm commented Feb 27, 2020

myasonik commented Mar 2, 2020

bhavyarm commented Mar 3, 2020

Comprehensive dashboard a11y tests #57821

Comprehensive dashboard a11y tests #57821

Conversation

bhavyarm commented Feb 17, 2020

kibanamachine commented Feb 20, 2020

💔 Build Failed

Test Failures

Standard Out

Stack Trace

Standard Out

Stack Trace

History

majagrubic Feb 20, 2020

Choose a reason for hiding this comment

bhavyarm Feb 21, 2020

Choose a reason for hiding this comment

majagrubic commented Feb 20, 2020

dmlemeshko left a comment

Choose a reason for hiding this comment

dmlemeshko Feb 20, 2020 • edited Loading

Choose a reason for hiding this comment

bhavyarm commented Feb 21, 2020

bhavyarm commented Feb 27, 2020

myasonik commented Mar 2, 2020

bhavyarm commented Mar 3, 2020

dmlemeshko Feb 20, 2020 •

edited

Loading