Skip to content
This repository was archived by the owner on Feb 22, 2023. It is now read-only.

[webview_flutter] Maybe fix flakiness of integration tests #6223

Merged
merged 11 commits into from
Aug 11, 2022

Conversation

bparrishMines
Copy link
Contributor

@bparrishMines bparrishMines commented Aug 10, 2022

Fixes flutter/flutter#86757
Fixes flutter/flutter#90976
Fixes flutter/flutter#94750

From my understanding, the test failures were caused by one or more of the following:

  1. The old implementation.
  2. Ending a test with an expect using the completion matcher. I think it causes a race condition where the test will finish, but Native code would still be running. When running the test locally, one could see tests that have already been considered passing to still be logging.
  3. Failed because reason 2 caused an error while another tests was running.
  4. Not waiting for a url to finish loading before calling WebViewController.currentUrl.

The code below was used in packages/webview_flutter/webview_flutter/example and packages/webview_flutter/webview_flutter_android/example to test flakiness. It runs the integration tests 10 times and records the number of failures of each test.

import 'dart:io';

final Map<String, int> failedTests = <String, int>{};
int allTestsPassCount = 0;

const int runCount = 10;

void main() async {
  for (int i = 0; i < runCount; i++) {
    final Process process = await Process.start(
      'flutter',
      '--no-color test -d emulator-5554 --timeout 240s integration_test/webview_flutter_test.dart'
          .split(' '),
    );

    process.stdout.forEach((List<int> data) {
      stdout.add(data);
      final String output = String.fromCharCodes(data);

      //00:43 +1 -2: test 3 [E]
      if (output.contains('[E]')) {
        final String? testName = RegExp(
          r'(?<=\d+:\d+ [0-9\+\-\s]+: ).+(?= \[E\])',
        ).stringMatch(output);

        if (testName == null) {
          print('FAILED TO PARSE TEST NAME: $output');
        } else {
          failedTests.putIfAbsent(testName, () => 0);
          failedTests[testName] = failedTests[testName]! + 1;
        }
      }

      if (output.contains('All tests passed!')) {
        allTestsPassCount++;
      }
    });
    process.stderr.forEach((List<int> data) {
      stderr.add(data);
    });
    await process.exitCode;

    print('CURRENT TEST FAILURE COUNTS:');
    print(failedTests);
  }

  print('ALL TESTS PASS COUNT: $allTestsPassCount');
  print('Desired count: $runCount');
}

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the relevant style guides and ran the auto-formatter. (Unlike the flutter/flutter repo, the flutter/plugins repo does use dart format.)
  • I signed the CLA.
  • The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
  • I listed at least one issue that this PR fixes in the description above.
  • I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy, or this PR is exempt from version changes.
  • I updated CHANGELOG.md to add a description of the change, following repository CHANGELOG style.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@github-actions github-actions bot added p: webview_flutter Edits files for a webview_flutter plugin platform-android labels Aug 10, 2022
@bparrishMines bparrishMines changed the title Android web tests [webview_flutter_android] Maybe fix flakiness of android plugin tests Aug 10, 2022
@bparrishMines bparrishMines changed the title [webview_flutter_android] Maybe fix flakiness of android plugin tests [webview_flutter] Maybe fix flakiness of android plugin tests Aug 10, 2022
@bparrishMines bparrishMines added override: no versioning needed Override the check requiring version bumps for most changes override: no changelog needed Override the check requiring CHANGELOG updates for most changes labels Aug 10, 2022
@bparrishMines bparrishMines changed the title [webview_flutter] Maybe fix flakiness of android plugin tests [webview_flutter] Maybe fix flakiness of integration tests Aug 10, 2022
Copy link
Contributor

@stuartmorgan-g stuartmorgan-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! LGTM

@bparrishMines bparrishMines added the autosubmit Merge PR when tree becomes green via auto submit App label Aug 11, 2022
@auto-submit auto-submit bot merged commit 860112a into flutter:main Aug 11, 2022
yutaaraki-toydium pushed a commit to yutaaraki-toydium/plugins that referenced this pull request Aug 12, 2022
moisefeelin pushed a commit to feelinproject/plugins that referenced this pull request Aug 26, 2022
mauricioluz pushed a commit to mauricioluz/plugins that referenced this pull request Jan 26, 2023
@bparrishMines bparrishMines deleted the android_web_tests branch January 31, 2023 19:06
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
autosubmit Merge PR when tree becomes green via auto submit App override: no changelog needed Override the check requiring CHANGELOG updates for most changes override: no versioning needed Override the check requiring version bumps for most changes p: webview_flutter Edits files for a webview_flutter plugin platform-android platform-ios
Projects
None yet
2 participants