[Mobile] Appium 2 Migration

.github/workflows/rnmobile-android-runner.yml

@@ -36,6 +36,16 @@ jobs:
             - name: Setup Node.js and install dependencies
               uses: ./.github/setup-node
 
+            - name: Restore tests setup cache
+              uses: actions/cache@704facf57e6136b1bc63b828d79edcd491f0ee84 # v3.3.2
+              with:
+                  path: |
+                      ~/.appium
+                  key: ${{ runner.os }}-tests-setup-${{ hashFiles('package-lock.json') }}
+
+            - name: Prepare tests setup
+              run: npm run native test:e2e:setup
+
             - name: Gradle cache
               uses: gradle/gradle-build-action@b5126f31dbc19dd434c3269bf8c28c315e121da2 # v2.8.1
 

.github/workflows/rnmobile-ios-runner.yml

@@ -18,7 +18,7 @@ jobs:
         if: ${{ github.repository == 'WordPress/gutenberg' || github.event_name == 'pull_request' }}
         strategy:
             matrix:
-                xcode: ['13.3.1']
+                xcode: ['14.2']
                 device: ['iPhone 13']
                 native-test-name: [gutenberg-editor-rendering]
 
@@ -27,9 +27,25 @@ jobs:
               with:
                   show-progress: ${{ runner.debug == '1' && 'true' || 'false' }}
 
+            - name: Switch Xcode version to ${{ matrix.xcode }}
+              run: sudo xcode-select --switch /Applications/Xcode_${{ matrix.xcode }}.app
+
+            - name: Launch simulator
+              run: (open -a Simulator && xcrun simctl boot '${{ matrix.device }}') &
+
             - name: Setup Node.js and install dependencies
               uses: ./.github/setup-node
 
+            - name: Restore tests setup cache
+              uses: actions/cache@704facf57e6136b1bc63b828d79edcd491f0ee84 # v3.3.2
+              with:
+                  path: |
+                      ~/.appium
+                  key: ${{ runner.os }}-tests-setup-${{ hashFiles('package-lock.json') }}
+
+            - name: Prepare tests setup
+              run: npm run native test:e2e:setup
+
             - name: Prepare build cache key
               run: find package-lock.json packages/react-native-editor/ios packages/react-native-aztec/ios packages/react-native-bridge/ios -type f -print0 | sort -z | xargs -0 shasum | tee ios-checksums.txt
 
@@ -54,18 +70,12 @@ jobs:
             - name: Bundle iOS
               run: npm run native test:e2e:bundle:ios
 
-            - name: Switch Xcode version to ${{ matrix.xcode }}
-              run: sudo xcode-select --switch /Applications/Xcode_${{ matrix.xcode }}.app
-
             - name: Build (if needed)
               run: test -e packages/react-native-editor/ios/build/GutenbergDemo/Build/Products/Release-iphonesimulator/GutenbergDemo.app/GutenbergDemo || npm run native test:e2e:build-app:ios
 
             - name: Build Web Driver Agent (if needed)
               run: test -d packages/react-native-editor/ios/build/WDA || npm run native test:e2e:build-wda
 
-            - name: Launch simulator
-              run: open -a Simulator && xcrun simctl boot '${{ matrix.device }}'
-
             - name: Run iOS Device Tests
               run: TEST_RN_PLATFORM=ios npm run native device-tests:local  ${{ matrix.native-test-name }}
 

docs/contributors/code/react-native/README.md

@@ -20,8 +20,8 @@ Also, the mobile client is packaged and released via the [official WordPress app
 
 If you encounter a failed Android/iOS test on your pull request, we recommend the following steps:
 
-1. Re-running the failed GitHub Action job ([guide for how to re-run](https://docs.github.com/en/actions/configuring-and-managing-workflows/managing-a-workflow-run#viewing-your-workflow-history)) - This should fix failed tests the majority of the time. Cases where you need to re-run tests for a pass should go down in the near future as flakiness in tests is actively being worked on. See the following GitHub issue for updated info on known failures: https://github.com/WordPress/gutenberg/issues/23949
-2. You can check if the test is failing locally by following the steps to run the E2E test on your machine from the [mobile getting started guide](/docs/contributors/code/react-native/getting-started-react-native.md#ui-tests), with even more relevant info in the [relevant directory README.md](https://github.com/WordPress/gutenberg/tree/HEAD/packages/react-native-editor/__device-tests__#running-the-tests-locally)
+1. Re-running the failed GitHub Action job ([guide for how to re-run](https://docs.github.com/en/actions/configuring-and-managing-workflows/managing-a-workflow-run#viewing-your-workflow-history)) - This should fix failed tests the majority of the time.
+2. You can check if the test is failing locally by following the steps to run the E2E test on your machine from the [E2E testing documentation](/packages/react-native-editor/__device-tests__/README.md).
 3. In addition to reading the logs from the E2E test, you can download a video recording from the Artifacts section of the GitHub job that may have additional useful information.
 4. Check if any changes in your PR would require corresponding changes to `.native.js` versions of files.
 5. Lastly, if you're stuck on a failing mobile test, feel free to reach out to contributors on Slack in the #mobile or #core-editor chats in the WordPress Core Slack, [free to join](https://make.wordpress.org/chat/).
@@ -32,4 +32,4 @@ Follow the instructions in [Native mobile testing](/docs/contributors/code/react
 
 ## Internationalization (i18n)
 
-Further information about this topic can be found in the [React Native Internationalization Guide](/docs/contributors/code/react-native/internationalization-guide.md).
+Further information about this topic can be found in the [React Native Internationalization Guide](/docs/contributors/code/react-native/internationalization-guide.md).

docs/contributors/code/react-native/getting-started-react-native.md

@@ -141,34 +141,8 @@ Then, open `chrome://inspect` in Chrome to attach the debugger (look into the "R
 
 This project is set up to use [jest](https://facebook.github.io/jest/) for tests. You can configure whatever testing strategy you like, but jest works out of the box. Create test files in directories called `__tests__` or with the `.test.js` extension to have the files loaded by jest. See an example test [here](https://github.com/WordPress/gutenberg/blob/HEAD/packages/react-native-editor/src/test/api-fetch-setup.test.js). The [jest documentation](https://facebook.github.io/jest/docs/en/getting-started.html) is also a wonderful resource, as is the [React Native testing tutorial](https://facebook.github.io/jest/docs/en/tutorial-react-native.html).
 
-## UI Tests
+## End-to-End Tests
 
-This repository uses Appium to run UI tests. The tests live in `__device-tests__` and are written using Appium to run tests against simulators and real devices. To run these you'll need to check off a few things:
+In addition to unit tests, the Gutenberg for Mobile Apps (GMA) project relies upon end-to-end (E2E) tests to automate testing critical flows in an environment similar to that of an end user. We generally prefer unit tests due to their speed and ease of maintenance. However, assertions that require OS-level features (e.g. complex gestures, text selection) or visual regression testing (e.g. dark mode, contrast levels) we use E2E tests.
 
--   When running the tests, you'll need to ensure the Metro bundler (`npm run native start`) is not running.
--   [Appium CLI](https://github.com/appium/appium/blob/1.x/docs/en/about-appium/getting-started.md) installed and available globally. We also recommend using [appium-doctor](https://github.com/appium/appium-doctor) to ensure all of Appium's dependencies are good to go. You don't have to worry about starting the server yourself, the tests handle starting the server on port 4723, just be sure that the port is free or feel free to change the port number in the test file.
--   For iOS a simulator should automatically launch but for Android you'll need to have an emulator _with at least platform version 8.0_ fired up and running.
-
-Then, to run the UI tests on iOS:
-
-```sh
-npm run native test:e2e:ios:local
-```
-
-and for Android:
-
-```sh
-npm run native test:e2e:android:local
-```
-
-To run a single test instead of the entire suite, use `npm run native device-tests:local`. Here's an example that runs only `gutenberg-editor-paragraph.test.js`:
-
-```sh
-npm run native test:e2e:android:local gutenberg-editor-paragraph.test.js
-```
-
-Note: You might experience problems that seem to be related to the tests starting the Appium server, e.g. errors that say `Connection Refused`, `Connection Reset` or `The requested environment is not available`. If so, you can manually start the Appium server via `npm run native appium:start`, and (optionally) comment out related code in the `beforeAll` and `afterAll` block.
-
-For a more detailed outline of the UI tests and how to get started writing one, please visit the [UI Test documentation](/packages/react-native-editor/__device-tests__/README.md) and our [contributing guide](/packages/react-native-editor/__device-tests__/CONTRIBUTING.md).
-
-You might want to use Visual Studio Code as an editor. The project includes the configuration needed to use the above codestyle and linting tools automatically.
+The E2E tests are found in the [`packages/react-native-editor/__device-tests__`](/packages/react-native-editor/__device-tests__) directory. Additional documentation on running and contributing to these tests can be found in the [tests directory](/packages/react-native-editor/__device-tests__#readme).