Skip to content

Conversation

@justinkambic
Copy link
Contributor

@justinkambic justinkambic commented Jul 23, 2020

Summary

Code is WIP.

Fixes #72994.

We have a few skipped a11y tests in master. At least one of these is resulting of an underlying Uptime helper function, rather than actual availability-related problems. This patch should fix that.

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@justinkambic justinkambic added bug Fixes for quality problems that affect the customer experience blocker Project:Accessibility v8.0.0 Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability v7.10.0 v7.9.0 labels Jul 23, 2020
@justinkambic justinkambic self-assigned this Jul 23, 2020
@justinkambic justinkambic force-pushed the uptime_unskip-accessibility-test branch from 3541ae4 to 526f3d3 Compare July 23, 2020 15:48
@justinkambic
Copy link
Contributor Author

Passed a round of flaky test runner: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/653/

@justinkambic justinkambic force-pushed the uptime_unskip-accessibility-test branch from 526f3d3 to 933cda2 Compare July 23, 2020 18:42
@justinkambic justinkambic force-pushed the uptime_unskip-accessibility-test branch from 933cda2 to 9d77f97 Compare July 23, 2020 19:12
@justinkambic justinkambic added the release_note:skip Skip the PR/issue when compiling release notes label Jul 23, 2020
@justinkambic
Copy link
Contributor Author

Second round of flaky test runner passed: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/659/

@justinkambic justinkambic marked this pull request as ready for review July 23, 2020 21:57
@elasticmachine
Copy link
Contributor

Pinging @elastic/uptime (Team:uptime)

@justinkambic
Copy link
Contributor Author

Third round of flaky test runner passed x100 executions: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/662/

@justinkambic
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Copy link
Contributor

@shahzad31 shahzad31 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !!

Copy link
Contributor

@spalger spalger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any problems with this, though my instincts say that overriding the default timeouts isn't a good idea. I have no idea how those timeouts are chosen, but it seems like a tricky game to play...

@justinkambic
Copy link
Contributor Author

justinkambic commented Jul 27, 2020

@spalger do you recommend removing the custom timeout altogether? This test was just failing because 2kms was too short, but I'm fine with removing that and simply relying on the default. I don't think it will impact anything.

I only added the greater value as a quick fix, I didn't really give it much thought.

@spalger
Copy link
Contributor

spalger commented Jul 27, 2020

@spalger do you recommend removing the custom timeout altogether?

In general, yes. I think trying to manage timeouts manually is tricky business and will lead to a great deal of flakiness unless folks are managing them pretty carefully and with the considerations of the potentially very surprising delays we see in CI caused by us running so much at one time in VMs. I'm not necessarily recommending doing it right now, but I think in general it's an anti-pattern to override the defaults unless there's a clear reason for it.

@justinkambic
Copy link
Contributor Author

Ok thanks. I can bring this up in my team's next technical sync that we should aim to remove these unless we can provide a justification for each one.

@justinkambic justinkambic merged commit 1c690c6 into elastic:master Jul 27, 2020
justinkambic added a commit to justinkambic/kibana that referenced this pull request Jul 27, 2020
…c#73078)

* Increase timeout in attempt to fix skipped a11y test.

* Temporarily only run uptime tests for faster flaky testing.

* Uncomment other test suites.

* Unskip test and delete comment.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
justinkambic added a commit to justinkambic/kibana that referenced this pull request Jul 27, 2020
…c#73078)

* Increase timeout in attempt to fix skipped a11y test.

* Temporarily only run uptime tests for faster flaky testing.

* Uncomment other test suites.

* Unskip test and delete comment.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
justinkambic added a commit that referenced this pull request Jul 27, 2020
#73346)

* Increase timeout in attempt to fix skipped a11y test.

* Temporarily only run uptime tests for faster flaky testing.

* Uncomment other test suites.

* Unskip test and delete comment.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
justinkambic added a commit that referenced this pull request Jul 27, 2020
#73345)

* Increase timeout in attempt to fix skipped a11y test.

* Temporarily only run uptime tests for faster flaky testing.

* Uncomment other test suites.

* Unskip test and delete comment.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@justinkambic
Copy link
Contributor Author

@justinkambic justinkambic deleted the uptime_unskip-accessibility-test branch July 27, 2020 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blocker bug Fixes for quality problems that affect the customer experience Project:Accessibility release_note:skip Skip the PR/issue when compiling release notes Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability v7.9.0 v7.10.0 v8.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failing test: X-Pack Accessibility Tests.x-pack/test/accessibility/apps/uptime·ts - uptime settings page

5 participants