Flaky tests: changeSize and discoverWork in test-resources.js #16124
Comments
cathyxz
changed the title
|
I've noticed the following happening occasionally: DESCRIBE => Resources discoverWork [23:54:04] Safari 11.1.0 (Mac OS X 10.13.4): Executed 3227 of 3553 (Skipped 301) 1 FAILED https://travis-ci.org/ampproject/amphtml/jobs/396641161 DESCRIBE => Resources discoverWork [23:44:56] Safari 11.1.0 (Mac OS X 10.13.4): Executed 3280 of 3583 (Skipped 303) 1 FAILED |
|
These have also been acting up a bit today. cc/ @rsimha https://travis-ci.org/ampproject/amphtml/jobs/431103300 |
|
Can we disable this test while it's fixed? It's currently blocking several unrelated PRs from being merged. |
|
Looks like different tests in the same |
|
Good observation, @choumx. Here are some failures on |
dreamofabear
changed the title
|
Aaron, would you mind taking a crack at these two flaky tests? I've noticed them a lot recently on master and PR builds. Recent failure on master: https://travis-ci.org/ampproject/amphtml/jobs/477890970 |
|
This is still a top cause of flakiness. From today: https://travis-ci.org/ampproject/amphtml/jobs/480964219#L1524 Meanwhile, can we skip the flaky tests to keep /cc @alanorozco who is build cop |
|
I haven't looked at this yet but will by end of day today. Sorry for the delay. Thanks. |
|
I am in the process of upgrading my sauce lab account so I can run this in the environment the logs indicate it is failing in. @choumx posted a failure in a non sauce lab environment but given it's flakiness, I am unable to reproduce after several attempts. Looking at the failures, I think the easy fix (not necessarily the right fix) could just be removing the number of calls expected from the call viewport.getRect(). Some of the logs from the test failures provided indicate the state machine throwing this error so it seems like there is some unexpected state transition triggering this? I have a PR to test on saucelabs with the modified expectations which passes locally. Once I can reproduce the failures in sauce labs and run my changes and verify that my changes reduces or fixes the flakiness then I will send out the PR. I think that immediate fix is ok as it'll allow us to run the tests with hopefully reduced flakiness and test the resources functionality without having to comment them out. Investigating why getRect is getting called more than expected is difficult when I can't reproduce. Thanks |
|
cc\ @estherkim |
|
This is a high priority issue but it hasn't been updated in awhile. @alabiaga Do you have any updates? |
Remove call count expectations on getRect method call which is the culprit of the flakiness of the tests mentioned. This should be fine as it doesn't take away from what's being primarily tested as it occurs in the afterEach call. Note that this is not a permanent fix but a temporary one to alleviate the flakiness. I will keep #16124 open. Tested locally and ran it several times in saucelabs.
commit 08466820bf2c8c13e21abc5c26159d228edb345c
Author: Esther Kim <esth@google.com>
Date: Fri Feb 15 10:47:56 2019 -0500
Whitespace
commit cdfd719
Author: Esther Kim <esth@google.com>
Date: Fri Feb 15 10:34:13 2019 -0500
Stop timer logging
commit 4b63cd7
Author: Esther Kim <esth@google.com>
Date: Fri Feb 15 10:08:59 2019 -0500
Typos
commit 7db4b28
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 18:12:37 2019 -0500
Review comments
commit ced9c5d
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 18:01:12 2019 -0500
Review comments
commit 8fa0e88
Merge: e65dd15 893ff3f
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 17:29:46 2019 -0500
Merge branch 'master' into travis-build-matrix
commit e65dd15
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 17:28:35 2019 -0500
Review comments
commit d69c38a
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 17:08:20 2019 -0500
Update logging
commit 7e4f701
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 17:00:38 2019 -0500
Review changes
commit 893ff3f
Author: Aaron Turner <aaron@aaronthedev.com>
Date: Thu Feb 14 13:49:16 2019 -0800
Return 400 when recaptcha example fails (ampproject#20668)
commit 3f0d6c6
Author: Enrique Marroquin <5449100+Enriqe@users.noreply.github.com>
Date: Thu Feb 14 16:37:15 2019 -0500
[amp-story] Allow more parameters for pan/zoom animations ✨ (ampproject#20778)
* allow more parameters in for pan/zoom animations.
* change attribute names
* types
* fix test
* assert values for params, change attribute naming.
* change attribute names do scale-start, scale-end.
* hoist defaults to constants
commit 35d6030
Author: Justin Ridgewell <justin@ridgewell.name>
Date: Thu Feb 14 16:02:00 2019 -0500
🐛 Assert that values are never returned from vsync callbacks (ampproject#20836)
* Assert that values are never returned from vsync callbacks
Vsync does not propogate those values in any way. This is specifically targetting code like:
```js
vsync.measure(() => {
return el.getBoundingClientRect();
});
// or
vsync.mutatePromise(() => {
return someOtherPromise;
});
```
* Fix dev call
* Lint
* Don't use devAssert so this stays in production
* Type checks
commit b9ba432
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 15:52:54 2019 -0500
Fix nit, hopefully fix chrome start
commit 3cb87fa
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 15:08:42 2019 -0500
bash
commit 5d9844e
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 14:54:40 2019 -0500
rm
commit b027eb0
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 14:39:52 2019 -0500
Review comments
commit fe27d76
Author: sohanirao <35511558+sohanirao@users.noreply.github.com>
Date: Thu Feb 14 11:38:43 2019 -0800
SwG Release 0.1.22.44 (ampproject#20834)
commit 736d506
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 14:25:37 2019 -0500
Install chrome for dist test job, add EXTENSIONS_CSS_MAP to build artifact zip
commit a36ee5d
Author: delima02 <47327855+delima02@users.noreply.github.com>
Date: Thu Feb 14 14:17:42 2019 -0500
🐛 Guard against getBoundingClientRect to prevent unspecified errors in IE. (ampproject#20731)
* Guard against `Element.getBoundingClientRect`.
* Refactor solution.
* Check if element exists in the DOM.
* Fix solution using `Node.isConnected`.
* Correct the default value of the desired DOMRect
* Patch getBoundingClientRect for unsupported browsers. Alternative solution
* Refactor patch.
* Fix - use assigned variable rect, previously unused.
* Fix imports.
* Minor fixes.
* Make the return type of getBoundingClientRect be consistent with usages
* File overview comment and a minor fix.
commit e9acbd2
Author: Cathy Zhu <cathyxz@gmail.com>
Date: Fri Feb 15 03:17:37 2019 +0800
Undo space reservation after successful height change (ampproject#20856)
commit ccea3e2
Author: Nikita Beloglazov <nikelandjelo@gmail.com>
Date: Thu Feb 14 11:12:16 2019 -0800
Export few data types from content-recommendation.js (ampproject#20854)
It allows users of that file to use the types. Given that we return objects of these types from exported functions - exporting types themselves makes sense.
Tested:
presubmit
commit 80f9fa9
Author: Sepand Parhami <sparhami@google.com>
Date: Thu Feb 14 11:10:11 2019 -0800
🐛Do not perform static layout for cloned nodes with static layout. (ampproject#20724)
commit a751cc7
Author: keithwrightbos <keithwrightbos@users.noreply.github.com>
Date: Thu Feb 14 13:47:48 2019 -0500
AdSense Fast Fetch delay 3 viewport experiment - respect loading strategy (ampproject#20866)
* initial commit
* modify dep-check
* truly fix dep issue
commit 806db59
Author: Carlos Vializ <cvializ@users.noreply.github.com>
Date: Thu Feb 14 13:21:26 2019 -0500
✨Implement autoexpand for amp-form textarea element (ampproject#20772)
* Add expanding textarea feature to amp-form
* Add autoshrink behavior in addition to autoexpand
* reset
* reset2 it works
* Does not stay at top
* Prevent scrollbar from flashing before max-height
* default autoshrink. Prevent shrinking if the textarea hasn't yet expanded
* Use class style
* Add manual test
* Cleanup
* Review feedback. Remove autoshrink attr. Lazy-load feature.
* Encapsulate all textarea logic, including installation.
* Mark expensive property accesses as OK
* Warn and forbid textarea with initial overflow
* Add tests for confidence. Fix presubmit errors.
commit cb95582
Author: Esther Kim <esth@google.com>
Date: Thu Feb 14 11:51:37 2019 -0500
Separate PR jobs
commit 4679a4c
Author: Gabriel Majoulet <gmajoulet@google.com>
Date: Thu Feb 14 10:52:06 2019 -0500
Displaying more specific messages when users have to resize their window, + refactoring. (ampproject#20835)
commit b9814f5
Author: keithwrightbos <keithwrightbos@users.noreply.github.com>
Date: Thu Feb 14 10:43:42 2019 -0500
initial commit (ampproject#20865)
commit e570d46
Author: Hongfei Ding <lannka@users.noreply.github.com>
Date: Wed Feb 13 22:15:49 2019 -0800
amp-ad-exit: use HostService if exists. (ampproject#20357)
* amp-ad-exit: use HostService if exists.
* naming & docs
* rename host-api to host-service
* lint
commit ecc3daa
Author: Cathy Zhu <cathyxz@gmail.com>
Date: Thu Feb 14 13:34:08 2019 +0800
Make is-my-pr-in-production-yet more prominent in our docs (ampproject#20857)
commit 12d441c
Author: Alan Orozco <alanorozco@users.noreply.github.com>
Date: Wed Feb 13 13:44:50 2019 -0800
✨auto-lightbox: Discard tap actions with invalid references (ampproject#20789)
Exposes `ActionService.hasResolvableAction` to:
- Discard tap action in auto-lightbox criteria when it does not resolve to a valid target.
- Fix `amp-lightbox-gallery` so it will properly traverse up the tree to discard targets, and not to discard those dynamically set.
commit dd4037d
Author: Nikita Beloglazov <nikelandjelo@gmail.com>
Date: Wed Feb 13 13:25:13 2019 -0800
Improve Google matched content logic to reflect adsbygoogle.js. (ampproject#20624)
The commit introduces core-shared.js file which supposed to be shared between AMP and internal Google adsbygoogle.js codebase. The file has no dependencies on environment to make importing these files in other projects easier.
In particular this commit adds support for publisher settings as described in https://support.google.com/adsense/answer/7533385
Tested:
unit tests
manually on fast festh and delayed fetch
commit 76a7c0f
Author: keithwrightbos <keithwrightbos@users.noreply.github.com>
Date: Wed Feb 13 16:10:58 2019 -0500
initial commit (ampproject#20837)
commit f1fb33b
Author: keithwrightbos <keithwrightbos@users.noreply.github.com>
Date: Wed Feb 13 15:37:39 2019 -0500
initial commit (ampproject#20833)
commit 6831b63
Author: Nikita Beloglazov <nikelandjelo@gmail.com>
Date: Wed Feb 13 12:37:10 2019 -0800
Cleanup adsense amp-auto-ads holdout experiment. Close ampproject#9247. (ampproject#20774)
It has been running for a while and we no longer need it. If someone decides to look at results in future - they can rollback this commit and restart experiment.
Tested:
presubmit
commit 4f3cd0d
Author: William Chou <willchou@google.com>
Date: Wed Feb 13 15:28:26 2019 -0500
Don't send amp-bind state to untrusted viewers (ampproject#20822)
* Only send state for history updates to trusted viewers.
* Add and fix tests.
* Fix type check.
* Fix presubmit.
commit ab467bc
Author: Carlos Vializ <cvializ@users.noreply.github.com>
Date: Wed Feb 13 14:26:35 2019 -0500
✅ Resolve some flakiness in e2e tests (ampproject#20794)
* Resolve some flakiness in e2e tests
* Unskip passing test
commit e13a531
Author: Aaron Labiaga <alabiaga@google.com>
Date: Wed Feb 13 13:35:59 2019 -0500
Make test-resource.js not flaky (ampproject#20514)
Remove call count expectations on getRect method call which is the culprit of the flakiness of the tests mentioned. This should be fine as it doesn't take away from what's being primarily tested as it occurs in the afterEach call. Note that this is not a permanent fix but a temporary one to alleviate the flakiness. I will keep ampproject#16124 open.
Tested locally and ran it several times in saucelabs.
commit 4a7282f
Author: Carlos Vializ <cvializ@users.noreply.github.com>
Date: Wed Feb 13 12:01:23 2019 -0500
Skip date picker visual diff tests for flakes (ampproject#20809)
commit 703ab02
Author: William Chou <willchou@google.com>
Date: Wed Feb 13 11:53:05 2019 -0500
Clean up 'user-error-reporting' (ampproject#20779)
* Clean up user-error-reporting.
* Fix lint.
commit 229d909
Author: Alan Orozco <alanorozco@users.noreply.github.com>
Date: Wed Feb 13 02:44:33 2019 -0800
🛑 <amp-video-iframe> integration safeguards (ampproject#20826)
- Prefix all frame-level errors with `<amp-video-iframe>` to let authors know source of integration mistakes.
- Fail to `adopt` window twice, notify that script tag should only be included once.
- Display valid listener types when trying to `listenTo('invalidFramework')`.
- Don't let author `listenTo` twice.
commit 84f701f
Author: Ali Ghassemi <aghassemi@google.com>
Date: Tue Feb 12 23:21:22 2019 -0800
✅ amp-youtube: add amp=1 to the url (ampproject#20821)
commit b58b6fa
Author: Alan Orozco <alanorozco@users.noreply.github.com>
Date: Tue Feb 12 23:03:58 2019 -0800
✨ Set #amp=1 fragment in amp-video-iframe src (ampproject#20823)
Docs say we do it, but we don't, so let's do it!
Also cleaned up some tests to be clearer and to use `async/await`.
commit f92f627
Author: Justin Ridgewell <justin@ridgewell.name>
Date: Tue Feb 12 20:24:26 2019 -0500
Optimize scopeSelector (ampproject#20819)
https://jsbench.github.io/#e00c46505b5b875d2dcd5894d6e46adc
commit 0298883
Author: Alan Orozco <alanorozco@users.noreply.github.com>
Date: Tue Feb 12 16:57:41 2019 -0800
🐛Reset min dimensions on undock (ampproject#20817)
Otherwise the video can be oversized and clipped when docked and resizing the window.
commit bcce892
Author: Caleb Cordry <ccordry@google.com>
Date: Tue Feb 12 16:33:05 2019 -0800
📖amp story ads RTC & Targeting docs (ampproject#20818)
|
Here's another data point (I'm build cop this week): #21151 (comment) (I haven't skipped any tests yet, will hold off until this issue is fixed.) |
|
#20514 should've fixed the traces in OP but looks like there are new flakes. |
|
Yup, will take another look. Thanks
…On Mon, Mar 4, 2019, 11:16 AM William Chou ***@***.***> wrote:
#20514 <#20514> should've fixed
the traces in OP but looks like there are new flakes.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#16124 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAeKUAc9TanLcOICQ32MighqlG2lv-Inks5vTUbOgaJpZM4UsknV>
.
|
|
adding flake: https://travis-ci.org/ampproject/amphtml/jobs/503826554 |
Remove call count expectations on getRect method call which is the culprit of the flakiness of the tests mentioned. This should be fine as it doesn't take away from what's being primarily tested as it occurs in the afterEach call. Note that this is not a permanent fix but a temporary one to alleviate the flakiness. I will keep ampproject#16124 open. Tested locally and ran it several times in saucelabs.
Remove call count expectations on getRect method call which is the culprit of the flakiness of the tests mentioned. This should be fine as it doesn't take away from what's being primarily tested as it occurs in the afterEach call. Note that this is not a permanent fix but a temporary one to alleviate the flakiness. I will keep ampproject#16124 open. Tested locally and ran it several times in saucelabs.
|
Haven't seen this in awhile -- closing. |
Per build: https://travis-ci.org/ampproject/amphtml/jobs/393815752
These are failing / flaky on Safari.
Error messages below:
Also from the same test suite, these are flaky / failing on Windows Chrome.
The text was updated successfully, but these errors were encountered: