[web] Engine integration test #16930

nturgut · 2020-03-04T17:08:54Z

Web engine-framework integration test.

Tests text editing behaviour.

collinjackson · 2020-03-04T19:47:22Z

.cirrus.yml

+        - cd $ENGINE_PATH/src/flutter/e2etests/web/text_editing
+        - $FRAMEWORK_PATH/flutter/bin/flutter config --local-engine=host_debug_unopt --no-analytics --enable-web
+        - $FRAMEWORK_PATH/flutter/bin/flutter pub get --local-engine=host_debug_unopt
+        - $FRAMEWORK_PATH/flutter/bin/flutter drive -v --target=test_driver/text_editing_e2e.dart -d web-server --release --browser-name=chrome --local-engine=host_debug_unopt


We're expecting to add more test files in the future, right? It might make sense to split this out into a script, and maybe even shard it if the tests take awhile to run. I guess this could be added incrementally in the future.

Yes I agree. This is configuring web as well. I'll make a script with multiple arguments (so we can add mobile there too)

Would it make sense to add running e2e tests into the felt tool? We do that already for unit-tests and goldens.

Yes I'm planning to that. either felt or a new tool. I'll do it depending on the results of the sheet I sent you. (See my answer to mdebbar's comment)

collinjackson

Thanks this looks really useful. I'll follow a similar pattern for the engine + framework mobile integration tests.

.cirrus.yml

mdebbar · 2020-03-04T19:43:41Z

.cirrus.yml

+        ./clone_flutter.sh
+        cd $FRAMEWORK_PATH/flutter
+        bin/flutter update-packages --local-engine=host_debug_unopt
+      script:


How do you envision adding more integration tests? Are they supposed to be added here as another step in this script: section or as a separate entry?

I think it depends on how these tests will develop. One thing to note is we are able to run these tests in 6 different browsers over 3 different OS. They might be using the same app to test or different apps.
However the setup is different for them (very different in some settings)
I'm planning to setup a meeting for next weeks to identify critical scenarios. If there are over 20 for Chrome Linux I think I will write script which can also be invoked from LUCI recipes.
If they all use the same app (let's say something comprehensive such as gallery) we can add a test runner.
On the other hand if they are all scattered between OSes and devices and apps then I am not planning to invest this tooling effort for each.
This example is particularly important to be as readable as possible cause other team members also asked me how to run integration tests from the engine.

e2etests/web/text_editing/.flutter-plugins-dependencies

e2etests/web/text_editing/build/web/assets/FontManifest.json

yjbanov · 2020-03-05T15:34:05Z

.cirrus.yml

@@ -124,6 +124,27 @@ task:
    - name: web_tests-7_last-linux # last Web shard must end with _last
      << : *WEB_SHARD_TEMPLATE

+    - name: web_integration_test_linux


nit: I imagine in the future we will want to run framework-side integration tests as part of the engine build as well. To make it clear that this shard is running engine tests specifically, I'd put the word "engine" somewhere in the name of the shard.

e2etests/web/text_editing/.flutter-plugins

e2etests/web/text_editing/build/flutter_assets/AssetManifest.json

e2etests/web/text_editing/text_editing.iml

yjbanov · 2020-03-05T15:40:47Z

e2etests/web/text_editing/web/index.html

@@ -0,0 +1,12 @@
+<!DOCTYPE HTML>


I think creating a full app for each test will result in too much boilerplate checked into the repo. I think a better approach is to have a couple of apps each containing a large number of integration tests. For example, we could have e2etests/web/integration for all integration tests that can share the same app configuration. If an e2e test needs specialized app configuration (e.g. PWA vs non-PWA packaging) we can create extra apps under e2etests/web, but I imagine there will be very few specialized cases like that.

I have some concerns on that one. It will be more clear after everybody on the team fills out the sheet I am collecting about the integration tests.
From my experience in the other teams adding to much to this type of integration tests app will make the results flaky and hard to troubleshoot during an error.

We need a very simple app to see if flutter run -d web-server --local-engine load a page in the given os/browser.

For every function we need to test the function only, for example a text editing test
shouldn't fail with other 15 tests if icons stop loading. Ideally one, in reality a handful should fail.

If one wants to test things like navigation/history and everybody shared the same app, the other developers additions to the app can fail the previously written scenarios.

If we need to navigate to a certain location in order to test every time app opens, the test can break not only it's functionality regress but also the steps in between regress. For example a keyboard test will fail more often if you need to scroll down and select a page 'keyboard' sub app, before you come to keyboard tests.

I propose to have at least 9-10 different apps we use. we can discuss more on the sheet.

@yjbanov
We had an offline discussion and here are the agreements we have:

we want to reduce the number of files we have.

felt will on the fly add the drive file. This way we will be able to remove the drive file. (I'll do it in the next PRs)

e2e will have a notation of 'testMain' and 'main' so we can merge the application file (main.dart) and the e2e tests.

we will still keep all the tests under e2etests since in the future we can make roller to run all the integration tests here and roll the PR from engine to flutter accordingly.

I'll rename the package to a more general name so new web engine integration tests won't need to create their own package (their own pubspec and everything). We will only create a new package if we need a specific configuration. So we will have 2-3 of them at most.

In the future PRs I'll make felt to run these tests automatically even if it is run under web_ui with felt test command.

Felt test should give a proper log for the failure ideally with the location of the test so web engine developers won't have to worry about it's location.

nturgut · 2020-03-06T23:34:06Z

Merging the PR. Thanks for the reviews.

* 7c612de Roll fuchsia/sdk/core/linux-amd64 from cXgMr... to cTw2C... (flutter/engine#16970) * 6cfa7fc fix shadows and mask filter blurs (flutter/engine#16963) * bfebadf Roll src/third_party/skia 012f8497802e..93a2a6b8badb (4 commits) (flutter/engine#16974) * 47963a5 Roll src/third_party/skia 93a2a6b8badb..74055566bd14 (2 commits) (flutter/engine#16981) * 98f9941 [fuchsia] fix broken flows when under high load (flutter/engine#16834) * fe051e0 Fix issue viewdidload call while init FlutterViewController (flutter/engine#16672) * 0ad54c2 [web] Fixes IE11 crash due to missing canvas ellipse support and font polyfill failure (flutter/engine#16965) * f6435de Roll fuchsia/sdk/core/mac-amd64 from J6ct_... to 95geB... (flutter/engine#16982) * 43971ca Roll src/third_party/skia 74055566bd14..54de2fa48d85 (3 commits) (flutter/engine#16983) * 45e61a6 Roll fuchsia/sdk/core/linux-amd64 from cTw2C... to K1wwe... (flutter/engine#16984) * 1ab5c36 Roll src/third_party/skia 54de2fa48d85..beaaf4700f50 (3 commits) (flutter/engine#16987) * e2c0454 remove 10s timeouts from tests (flutter/engine#16988) * dfc9c12 Roll src/third_party/skia beaaf4700f50..6e58290ba639 (9 commits) (flutter/engine#16990) * eddda80 fushia licenses fix (flutter/engine#16992) * c15f239 documented fluttertexture.h (flutter/engine#16950) * e1ba7a1 Roll src/third_party/skia 6e58290ba639..24a8e9e170f7 (5 commits) (flutter/engine#16996) * fc5963d [web] Engine integration test (flutter/engine#16930) * d323bac doxygen tooling updates and doxygen for FlutterCodecs.h (flutter/engine#16947) * 03ddc1d Started deleting .DS_Store files so licenses can run on mac os x. (flutter/engine#16998) * 30a8292 Roll src/third_party/skia 24a8e9e170f7..cf573d844da6 (4 commits) (flutter/engine#17004) * d031963 Roll fuchsia/sdk/core/mac-amd64 from 95geB... to hW33F... (flutter/engine#17006) * 41b371d Roll fuchsia/sdk/core/linux-amd64 from K1wwe... to FGMpI... (flutter/engine#17007) * 619acd5 Revert "fix shadows and mask filter blurs (#16963)" (flutter/engine#17008)

* squashing the commits together * directory rename, project rename. addressing reviewer comments * update cirrus file * change tool signature

nturgut added the platform-web Code specifically for the web engine label Mar 4, 2020

googlebot added the cla: yes label Mar 4, 2020

nturgut mentioned this pull request Mar 4, 2020

Engine repo should be able to run tests that depends on the framework. flutter/flutter#51892

Open

nturgut requested review from yjbanov, mdebbar and collinjackson March 4, 2020 19:04

collinjackson reviewed Mar 4, 2020

View reviewed changes

collinjackson approved these changes Mar 4, 2020

View reviewed changes

mdebbar reviewed Mar 4, 2020

View reviewed changes

yjbanov reviewed Mar 5, 2020

View reviewed changes

e2etests/web/text_editing/.flutter-plugins Outdated Show resolved Hide resolved

yjbanov reviewed Mar 5, 2020

View reviewed changes

nturgut force-pushed the integration_tests branch 2 times, most recently from c00259c to fc8ea97 Compare March 6, 2020 18:48

nturgut mentioned this pull request Mar 6, 2020

fushia licenses fix #16992

Merged

Nurhan Turgut added 3 commits March 6, 2020 13:38

squashing the commits together

c62fb7e

directory rename, project rename. addressing reviewer comments

7026e24

update cirrus file

1f3accd

nturgut force-pushed the integration_tests branch from 34f61e7 to 1f3accd Compare March 6, 2020 21:39

change tool signature

53f3de4

nturgut merged commit fc5963d into flutter:master Mar 6, 2020

engine-flutter-autoroll mentioned this pull request Mar 7, 2020

Roll engine 5aff31194809..d323bace5f6f (18 commits) flutter/flutter#52159

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Mar 7, 2020

fc5963d [web] Engine integration test (flutter/engine#16930)

32d2bba

engine-flutter-autoroll mentioned this pull request Mar 7, 2020

Roll engine 5aff31194809..30a82927be82 (20 commits) flutter/flutter#52162

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Mar 7, 2020

fc5963d [web] Engine integration test (flutter/engine#16930)

90c4930

engine-flutter-autoroll mentioned this pull request Mar 7, 2020

Roll engine 5aff31194809..d0319631bfb8 (21 commits) flutter/flutter#52164

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Mar 7, 2020

fc5963d [web] Engine integration test (flutter/engine#16930)

9393a0c

engine-flutter-autoroll mentioned this pull request Mar 7, 2020

Roll engine 5aff31194809..41b371d0b0be (22 commits) flutter/flutter#52165

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Mar 7, 2020

fc5963d [web] Engine integration test (flutter/engine#16930)

f08701b

engine-flutter-autoroll mentioned this pull request Mar 7, 2020

Roll engine 5aff31194809..41b371d0b0be (22 commits) flutter/flutter#52176

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Mar 7, 2020

fc5963d [web] Engine integration test (flutter/engine#16930)

d1b9c73

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Mar 7, 2020

fc5963d [web] Engine integration test (flutter/engine#16930)

351b125

engine-flutter-autoroll mentioned this pull request Mar 7, 2020

Roll engine 5aff31194809..619acd58ca55 (23 commits) flutter/flutter#52177

Merged

blasten pushed a commit to blasten/engine that referenced this pull request Mar 10, 2020

[web] Engine integration test (flutter#16930)

75f8c05

* squashing the commits together * directory rename, project rename. addressing reviewer comments * update cirrus file * change tool signature

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[web] Engine integration test #16930

[web] Engine integration test #16930

nturgut commented Mar 4, 2020 •

edited

Loading

collinjackson Mar 4, 2020

nturgut Mar 4, 2020

yjbanov Mar 5, 2020

nturgut Mar 5, 2020

collinjackson left a comment •

edited

Loading

mdebbar Mar 4, 2020

nturgut Mar 4, 2020

yjbanov Mar 5, 2020

yjbanov Mar 5, 2020

nturgut Mar 5, 2020

nturgut Mar 6, 2020 •

edited

Loading

nturgut commented Mar 6, 2020

[web] Engine integration test #16930

[web] Engine integration test #16930

Conversation

nturgut commented Mar 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

collinjackson left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nturgut Mar 6, 2020 • edited Loading

Choose a reason for hiding this comment

nturgut commented Mar 6, 2020

nturgut commented Mar 4, 2020 •

edited

Loading

collinjackson left a comment •

edited

Loading

nturgut Mar 6, 2020 •

edited

Loading