Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes flakiness on timelion suggestions #89538

Merged
merged 5 commits into from
Feb 1, 2021

Conversation

stratoula
Copy link
Contributor

@stratoula stratoula commented Jan 28, 2021

Summary

Closes #89352
Flaky cloud tester run for 15 times https://internal-ci.elastic.co/job/elastic+estf-cloud-kibana-flaky-test-runner/
This is going to remove the flakiness (I hope). What we want is to test that the suggestions are returned.

Checklist

@stratoula
Copy link
Contributor Author

@elasticmachine merge upstream

@stratoula stratoula added v8.0.0 v7.11.1 v7.12.0 release_note:skip Skip the PR/issue when compiling release notes test-cloud Feature:Timelion Timelion app and visualization labels Jan 29, 2021
@stratoula stratoula marked this pull request as ready for review January 29, 2021 14:29
@stratoula stratoula added the Team:Visualizations Visualization editors, elastic-charts and infrastructure label Jan 29, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app (Team:KibanaApp)

@stratoula
Copy link
Contributor Author

@elasticmachine merge upstream

@flash1293
Copy link
Contributor

@stratoula Can you explain what the original issue was? It seems like it's showing one additional suggestion?

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@stratoula
Copy link
Contributor Author

@flash1293 yes. When it failed for the first time I noticed this:
timelion
I thought that there was a latency on the suggestion click so on this PR I increased the timeout #87273

But this was not the problem. I used a 7.11 cloud instance and I saw at first that the suggestions loaded for the split on the cloud are not the same as locally (locally never fails).

And an additional note here: The way that these timelion tests work is: update the expression, click a suggestion from the list, then update again the expression, clicking another suggestion. Imho, it seems to be very flaky for cloud.

@flash1293
Copy link
Contributor

Is it an issue the expression in the screenshot is broken?

@stratoula
Copy link
Contributor Author

The expression brokes when the test fails, so it fails on expect(suggestions.length).to.eql(51); and it gets confused and then the steps are not followed correctly and the expression brokes. But the problem starts with the number of the suggestions that on 7.11 (haven't checked other versions tbh) are 53. Does it answer your question? Sorry, not sure if I got it correctly 🙂

Copy link
Contributor

@flash1293 flash1293 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, makes sense, thanks. LGTM to fix the test for now, but as it's loosening our test coverage a little, it would be cool to add back a better test for this later.

If I understood correctly, the suggestion list didn't updated yet and that's why the number is wrong, so we could also add a retry loop for the 51:

await retry.try(async () => {
  const suggestions = await PageObjects.timelion.getSuggestionItemsText();
  expect(suggestions.length).to.eql(51);
});

@stratoula
Copy link
Contributor Author

Thanx Joe! I tried it but retry doesn't fix it. The suggestion list always returns a wrong number so I will create an issue to investigate it and also fix the coverage

@stratoula stratoula merged commit 19b1f46 into elastic:master Feb 1, 2021
stratoula added a commit to stratoula/kibana that referenced this pull request Feb 1, 2021
* Fixes flakiness on timelion suggestions

* Improvements

* Remove flakiness

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
# Conflicts:
#	test/functional/apps/timelion/_expression_typeahead.js
stratoula added a commit to stratoula/kibana that referenced this pull request Feb 1, 2021
* Fixes flakiness on timelion suggestions

* Improvements

* Remove flakiness

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
# Conflicts:
#	test/functional/apps/timelion/_expression_typeahead.js
stratoula added a commit that referenced this pull request Feb 1, 2021
* Fixes flakiness on timelion suggestions

* Improvements

* Remove flakiness

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
# Conflicts:
#	test/functional/apps/timelion/_expression_typeahead.js
stratoula added a commit that referenced this pull request Feb 1, 2021
* Fixes flakiness on timelion suggestions

* Improvements

* Remove flakiness

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
# Conflicts:
#	test/functional/apps/timelion/_expression_typeahead.js
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Timelion Timelion app and visualization release_note:skip Skip the PR/issue when compiling release notes Team:Visualizations Visualization editors, elastic-charts and infrastructure test-cloud v7.11.1 v7.12.0 v8.0.0
Projects
None yet
4 participants