Setup CI infra to run DevTools tests against multiple React versions

[bvaughn]

PR #19108 caused some Suspense-related DevTools regressions (more info available on #19368) which we did not catch because of the fact that DevTools tests are only run against the version of React in master.

We should follow the precedent of the regression fixtures tests and have CI run DevTools tests against multiple React versions, including v15, all v16 minors, and the current HEAD of master branch.

Setting this up will involve several things:

• Infra to checkout older React packages and run tests against them.
• Some form of gating so that we can account for expected differences in Store snapshots between React versions.
• Some form of gating so that we can avoid running tests against invalid combinations of features and versions (e.g. don't test for Suspense in a version of React that didn't include that component yet).

---

Which React versions should we test?

In addition to testing against the latest React version in source, I propose that we should also test every minor version going back as far as we support (e.g. starting with v15). We could automate this process like so:

const {exec} = require('child_process');
const semver = require('semver');

const versions = {};

exec('npm view react versions --json', (error, stdout, stderr) => {
  if (stdout) {
    const json = JSON.parse(stdout);
    json.forEach(versionString => {
      if (semver.valid(versionString)) {
        if (semver.gte(versionString, '15.0.0')) {
          const {major, minor, patch} = semver.parse(versionString);

          // Filter out RCs and CI-published daily releases.
          if (`${major}.${minor}.${patch}` === versionString) {
            // Store the last patch for each minor.
            // This relies on the view command returning a sorted releases list.
            const key = `${major}.${minor}`;

            versions[key] = versionString;
          }
        }
      }
    });

    // The "versions" object now contains all versions we should test again.
  }
});

[RitikPandey1]

I would like to take this as my first issue , please guide me from where I can start

[bvaughn]

@RitikPandey1 You are welcome to work on this issue if you would be interested!

Unfortunately, it would have to be self-guided work. I don't have any more of an outline for what needs to be done than what is written in the issue.

[RitikPandey1]

Okay, @bvaughn I will try to figure this out .

[MoSattler]

@bvaughn, not sure if this is easily done with checking out older code.
I tried it here, and I keep running into issues where code is either coupled to newer dependencies, or the newer tests are coupled to the newer versions of the code to be tested. See test results here.

Another issue is, that older checkouts have too restrictive devEngines requirements to be installed with modern node, see this.

[MoSattler]

Also tried it by just installing the old dependencies via npm, with similar results. Maybe some of the tests are testing internals, and hence are coupled to internals of the latest code?

[bvaughn]

The build-for-devtools command builds an experimental build (RELEASE_CHANNEL=experimental) of React to be bundled along with the DevTools extension. Generally this build always comes from the master branch, so DevTools is built with the latest version of React.

This task is about using the DevTools to connect to applications written with multiple (usually older) versions of React. Replacing the build-for-devtools command doesn't seem like the right approach here, since that would also be changing which version of React the DevTools is actually running with.

The precedent here would be our "legacy" tests:
https://github.com/facebook/react/tree/master/packages/react-devtools-shared/src/__tests__/legacy

These tests using NPM aliasing to run an older version of React with DevTools (which is still built with the latest experimental build):

react/packages/react-devtools-shared/src/__tests__/legacy/inspectElement-test.js

Lines 76 to 84 in 480626a

	// Redirect all React/ReactDOM requires to the v15 UMD.
	// We use the UMD because Jest doesn't enable us to mock deep imports (e.g. "react/lib/Something").
	jest.mock('react', () => jest.requireActual('react-15/dist/react.js'));
	jest.mock('react-dom', () =>
	jest.requireActual('react-dom-15/dist/react-dom.js'),
	);
	React = require('react');
	ReactDOM = require('react-dom');

[MoSattler]

Thanks for the elaboration, I will check it out!

[ktfth]

@MoSattler can i help you with that task?

[bvaughn]

This task is still up for grabs, if you're interested, @ktfth