Chrome parallel-group-params-mismatch error

[jebarjonet]

We use parallelized actions with browser: chrome, it worked well until yesterday. Now sometimes (not always) we have this behavior :

A first container starts with apparently Chrome 98 :

The other containers start a minute later and show us this error :

It says the specs mismatch, I guess it is because of the Chrome version here being Chrome 99 ?

So only 1 container (the first starting) goes ✅ and the others fail like above ❌

The configuration of our Github Action is like this :

- name: Cypress run
              uses: cypress-io/github-action@v2
              with:
                  browser: chrome
                  build: npm run build
                  config-file: cypress/cypress-ci.json
                  group: "Chrome"
                  parallel: true
                  record: true
                  start: [our starting script]
                  wait-on: [our local url]

I don't see a way of specifying a particular browser version with this action, am I missing something ?

[spruce-bruce]

I came to ask this same question. My test jobs failed this morning with the same issue. I'm not sure what weird stars aligning behavior had to happen to make chrome 99 available while spinning up my runners this morning, but I'm sure it's related to how github caches packages from apt or something.

I don't know if we can have a deterministic set of dependencies in gha when setting up parallel boxes... but really I'd rather this be a gha solve, but specifying browser version would be a decent work around for us failing that

[amaps]

We had the same issue this morning with our Cypress suites running on Github Actions.

As a temporary workaround, we have switched to Electron browser to unblock teams.

[jebarjonet]

For information the situation solved itself after a day or two but it is still happening sometimes, like today for a mismatch between Chrome 101 and Chrome 102

[codybrouwers]

Also looking into this issue today. It seems that whenever Chrome releases a new browser version there is a period of time where Github's runners install can run both the old and new versions and Github is in the process of updating chrome this week it looks like.

I was able to modify the script from this comment to ensure that Github is always running the latest stable version of Chrome from Google.

Luckily downloading the new Chrome version is only ~10-20 seconds so it doesn't add too much time to our CI runs if one worker isn't up to date.

Bash Script

  #!/bin/bash
  # Copied and modified from: https://github.com/actions/virtual-environments/issues/5651#issuecomment-1142075171
  # Used to ensure that Cypress tests are always using the latest version of Chrome.
  # This is important because sometimes when Chrome releases a new version, Github Action runners can run the
  # previous version or the new version and when sharding the tests to run in parallel results in this error:
  # https://github.com/cypress-io/github-action/issues/518

  download_with_retries() {
  # Due to restrictions of bash functions, positional arguments are used here.
  # In case if you using latest argument NAME, you should also set value to all previous parameters.
  # Example: download_with_retries $ANDROID_SDK_URL "." "android_sdk.zip"
      local URL="$1"
      local DEST="${2:-.}"
      local NAME="${3:-${URL##*/}}"
      local COMPRESSED="$4"

      if [[ $COMPRESSED == "compressed" ]]; then
          local COMMAND="curl $URL -4 -sL --compressed -o '$DEST/$NAME' -w '%{http_code}'"
      else
          local COMMAND="curl $URL -4 -sL -o '$DEST/$NAME' -w '%{http_code}'"
      fi

      echo "Downloading '$URL' to '${DEST}/${NAME}'..."
      retries=20
      interval=30
      while [ $retries -gt 0 ]; do
          ((retries--))
          # Temporary disable exit on error to retry on non-zero exit code
          set +e
          http_code=$(eval $COMMAND)
          exit_code=$?
          if [ $http_code -eq 200 ] && [ $exit_code -eq 0 ]; then
              echo "Download completed"
              return 0
          else
              echo "Error — Either HTTP response code for '$URL' is wrong - '$http_code' or exit code is not 0 - '$exit_code'. Waiting $interval seconds before the next attempt, $retries attempts left"
              sleep $interval
          fi
          # Enable exit on error back
          set -e
      done

      echo "Could not download $URL"
      return 1
  }

  INSTALLED_CHROME_VERSION=$(google-chrome --product-version)
  INSTALLED_CHROME_VERSION=${INSTALLED_CHROME_VERSION%.*}
  INSTALLED_CHROME_MAJOR_VERSION=$(echo $INSTALLED_CHROME_VERSION | cut -d. -f1)

  # Determine latest release of chromedriver
  # Compatibility of Google Chrome and Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/downloads/version-selection
  LATEST_VERSION_URL="https://chromedriver.storage.googleapis.com/LATEST_RELEASE"
  echo "Fetching latest chromedriver version from: $LATEST_VERSION_URL"
  LATEST_CHROMEDRIVER_VERSION=$(curl "$LATEST_VERSION_URL")
  LATEST_CHROMEDRIVER_MAJOR_VERSION=$(echo $LATEST_CHROMEDRIVER_VERSION | cut -d. -f1)

  echo "Installed Chrome Version: $INSTALLED_CHROME_VERSION"
  echo "Latest ChromeDriver Version: $LATEST_CHROMEDRIVER_VERSION"

  if [ "$INSTALLED_CHROME_MAJOR_VERSION" == "$LATEST_CHROMEDRIVER_MAJOR_VERSION" ]; then
    echo "The latest major version of Chrome is already installed."
    exit 0
  fi

  # Download and install Google Chrome
  CHROME_DEB_URL="https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb"
  CHROME_DEB_NAME="google-chrome-stable_current_amd64.deb"
  download_with_retries $CHROME_DEB_URL "/tmp" "${CHROME_DEB_NAME}"
  sudo apt install "/tmp/${CHROME_DEB_NAME}" -f

  # Download and unpack latest release of chromedriver
  echo "Downloading chromedriver v$LATEST_CHROMEDRIVER_VERSION..."
  wget "https://chromedriver.storage.googleapis.com/$LATEST_CHROMEDRIVER_VERSION/chromedriver_linux64.zip"
  unzip chromedriver_linux64.zip
  rm chromedriver_linux64.zip

  CHROMEDRIVER_DIR="/usr/local/share/chrome_driver"
  CHROMEDRIVER_BIN="$CHROMEDRIVER_DIR/chromedriver"

  sudo rm -rf $CHROMEDRIVER_DIR/*
  sudo rm -rf $CHROMEDRIVER_BIN/*

  sudo mkdir -p $CHROMEDRIVER_DIR
  sudo mv "chromedriver" $CHROMEDRIVER_BIN
  sudo chmod +x $CHROMEDRIVER_BIN
  chromedriver --version

[alejo4373]

@codybrouwers I think you meant in your Bash Script

-              sleep 30
+              sleep $interval

[alexbjorlig]

I really hope Cypress can take this issue seriously. Having stable non-flaky test is 50% good-running CI...

[paulzakin]

I solved this problem by avoiding the browser bundled with GitHub Actions.

Basically, I install the latest version of chromium and just use that.

Here's what that code looks like in our GitHub action.

If the Cypress team doesn't want to update the action itself to handle this mismatch, it might make sense to provide some examples in the documentation for how to avoid this problem!

- uses: browser-actions/setup-chrome@latest
- run: |
    echo "BROWSER_PATH=$(which chrome)" >> $GITHUB_ENV

- name: Run Cypress tests 🚀
  uses: cypress-io/github-action@v4
  with:
    install: false
    record: true
    parallel: true
    browser: "${{ env.BROWSER_PATH }}"
    start: "your start command"
  env:
    CYPRESS_RECORD_KEY: ${{ secrets.recordKey }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }

[chinanderm]

@paulzakin Thanks for the tip! Are you running your tests in a matrix strategy?

[paulzakin]

Nope - what you see is what you get in terms of config :)

It's worked very well thus far though. Haven't had any problems since I've posted this!

[a8trejo]

This workaround has worked like a charm for a while #518 (comment)

But now Test Replay is live and was disappointed to see this Test Replay is not supported in custom chrome

So we either:

• Have unreliable CI testing with chrome browser and Test Replay as there will always be chrome updates and mismatches across github containers every now and then
• Have reliable CI testing without Test Replay

We'll hopefully have someone from cypress finally look into this

[rockhold]

@a8trejo, the custom chrome issue came up internally earlier today as well. We have a change in place on the Cloud side that should allow this through. Would you mind updating this thread after you've tried another run? I believe your previous run will just indicate that Test Replay is not available (as the replay data was not captured). New runs, however, should be captured (though there may be some limits, depending on the browser name we detect).

[a8trejo]

This is the error I see now

The stdout

- Video - 6.96 MB /home/runner/work/.......mp4
  - Screenshot - 109 kB /home/runner/work/......cy.ts/test name (failed).png
  - Test Replay - Nothing to upload - Test Replay is only supported in Chromium browsers

  (Uploaded Cloud Artifacts)

  - Video - Done Uploading 6.96 MB 1/2 /home/runner/work/......mp4
  - Screenshot - Done Uploading 109 kB 2/2 /home/runner/work/......cy.ts/test name (failed).png

As a side note, I also implemented this suggestion for videos, not sure if that's what's breaking it, I can access the retried screenshots and videos just fine
https://docs.cypress.io/guides/guides/screenshots-and-videos#Delete-videos-for-specs-without-failing-or-retried-tests

[rockhold]

@a8trejo, is the above screenshot with the new message for a new run or the original one you posted? My understanding of the change is that we'd need a new run to capture the replay (since it originally didn't upload due to the custom chrome being handled as though it was unsupported).

[a8trejo]

that was a new run from 30 minutes ago @rockhold

[MikeMcC399]

Please note also related issues:

• Ensure consistent Chrome versions during CI-parallelization cypress#25040
• https://github.com/orgs/community/discussions/52956
• Parallel recording with non-Electron browser needs Docker for reliability #865

and the README: parallel documentation:

During staged rollout of a new GitHub-hosted runner version, GitHub may provide a mixture of current and new image versions used by the container matrix. It is recommended to use a Docker image in the parallel job run which avoids any Cypress Cloud errors due to browser major version mismatch from the two different image versions. A Docker image is not necessary if testing against the default built-in Electron browser because this browser version is fixed by the Cypress version in use and it is unaffected by any GitHub runner image rollout.

If there is a need to follow up about the custom Chrome issue, and this can't be resolved soon, then I suggest to open a new issue in https://github.com/cypress-io/cypress/issues as Test Replay is not part of cypress-io/github-action. It is just invoked through the action when parameters are set to record in the Cypress Cloud.

[MikeMcC399]

I have resubmitted an enhancement request to GitHub in a different place. In the community board, which is where enhancement requests for the GitHub runners are supposed to go, there was no response during the last months.

• Different runner images used by containers in same job actions/runner#2812

The GitHub runner team say that they have discussed this issue now. See actions/runner#2812 (comment).

[MikeMcC399]

The GitHub runner team have written in actions/runner#2812 (comment) that there are no plans to make changes in this area in the short- or medium-term which would ensure that containers in the same job would not use mixed runner images. 🙁

The enhancement request https://github.com/orgs/community/discussions/52956 remains on the GitHub Community board for the attention of GitHub Product Management. Given that this request has already been open for five months without any Product Management feedback, I'm not hopeful for any change there.

In the meantime, as far as Cypress recording to Cypress Cloud is concerned, the recommendation continues unchanged in README: parallel to use a Docker image to ensure that fixed, consistent, browser versions are being used (see also #518 (comment) ^^).

[a8trejo]

closing the loop on my previous comment, Test Replay now works properly with the workaround here, so still using that 👍

[MikeMcC399]

@jebarjonet

I suggest to close this issue now as all possible steps have now been addressed by github-action.

The best advice to avoid the issue is to use a Docker image as mentioned above ^^ in #518 (comment).