[test-runner] Ability to set query parameters

[aomarks]

We'd like to be able to use one invocation of wtr to run the same test files multiple times, with some sort of different configuration each time.

Example use cases:

• Run a test file once with some native feature enabled, and then again with a polyfill force-enabled.
• Run a test file once with some named module resolving to our development build, and then again with it resolving to our minified build.

Currently we just launch two invocations of wtr to do this, with a different set of plugins. This works fine, but it would be nice during development to be able to run a single --watch command that covers all of our test variations, and also it would improve our CI latency (especially when using Sauce), if we could do it with one invocation.

Two possibilities come to mind as a starting point:

• A way to specify URL parameters on the file specification/glob, which are passed through to the URL that gets served. This would let us include some JS in the test files that can check this parameter and e.g. conditionally loads a polyfill before the tests.

  files: [
    "tests/**/*.test.js?polyfill=true",
    "tests/**/*.test.js?polyfill=false",
  ],

• A way to specify sub-suites, with different sets of plugins.

  suites: [
    {
      files: ["tests/**/*test.js"],
      plugins: [resolveLitModulesToDevBuild()],
    },
    {
      files: ["tests/**/*test.js"],
      plugins: [resolveLitModulesToProdBuild()],
    }
  ]

Since a plugin can read URL query parameters, and since a plugin can inject JS, I think you should technically be able to achieve any result with either of these approaches. Though, the query parameter approach seems more natural when you want to toggle some behavior in the test JS, and the plugin approach seems more natural when you want to e.g. rewrite imports -- so maybe both have their place?

Edit: Actually I don't think the URL parameter approach alone can satisfy all use cases, since in order for a plugin to understand the context its executing in, that URL parameter would need to be propagated to the entire module dependency tree. So I think both approaches here, or something like them, may be needed.

[LarsDenBakker]

I'm working on config groups, tracked in: #369

To start off I've only implemented different browser launchers. I need to think a bit more about how we can use different plugins per config group without losing many of the caching benefits.

Using query params in the test file names is pretty interesting. We'd have to strip off the parameters before globbing and then add them back in after constructing the browser import. That does feel a bit weird.

[aomarks]

I'm working on config groups, tracked in: #369

Oh sweet, missed that, thanks! Groups sounds great for the second use case. I'll rename this issue to just concern query params then.

Using query params in the test file names is pretty interesting. We'd have to strip off the parameters before globbing and then add them back in after constructing the browser import. That does feel a bit weird.

Maybe queryParams could instead be its own config parameter? Combined with config groups, that seems like it would work well for our use case:

groups: [
  {
    files: ["tests/**/*test.js"],
    queryParams: {
      polyfills: true,
    },
  },
  {
    files: ["tests/**/*test.js"],
    queryParams: {
      polyfills: false,
    },
]

[LarsDenBakker]

We do have a transformImport hook which can be used to add query params to imports. We don't pass the test file to this hook just yet, but plan to add this as well.

Then it would look something like this:

groups: [
  {
    files: ["tests/**/*test.js"],
    plugins: [
      {
        transformImport({ source, context }) {
          if (isTestFile(context.path)) {
            return `${source}?polyfills=true`
          } 
        }
      }
    ]
  },
  {
    files: ["tests/**/*test.js"],
]

what do you think?

[aomarks]

what do you think?

That could work, and we would read it out of import.meta.url somewhere? Would that survive a SystemJS transform with the legacy plugin too?

My original thought was that these parameters would get passed to the test suite HTML URL, because then when you're in interactive mode you can easily toggle the flags manually by editing the URL, and test code would read it from window.location wherever. But I guess using groups and transformImport, you'd have some URL parameter like wtr-test-group=xxx that would let you switch between the different test group plugin configurations in manual mode?

[LarsDenBakker]

Doing it in the URL in the page would be pretty interesting as well, I agree. But we don't have an easy hook for this right now.

I guess in theory you could write a middleware that returns a http redirect from / (the test HTML page) to /?foo=bar? That might work... I want to avoid creating special hooks unless we have a strong case for it, because it makes it harder to makes change to the test runner's internals later on.

With #634 you can also transform the test file now:

import { isTestFilePath } from '@web/test-runner';

export default {
  plugins: [
    {
      transformImport({ source }) {
        if (isTestFilePath(source)) {
          console.log('transforming', source);
          return `${source}&polyfills=true`;
        }
      },
    },
  ],
};

In your code you could read it from the import.meta.url inside the test file, systemjs implements all the es module features so import.meta.url will work there as well.

Good point about switching between groups in manual mode, need to think about that some more.