ci: add e2e tests to CI #546

michaelfaith · 2025-08-10T20:28:00Z

This change adds a new end-to-end test capability. There are two new jobs in the main CI workflow. One job to build the package (which is also good to have outside of the need to run e2e tests), executed with the latest LTS node. The build artifact is uploaded for the downstream e2e job to use. The e2e job, downloads the artifact, run npm install and then npm run e2e, which executes a custom node script to loop through subdirectories of /e2e, which each contain a project fixture with different setups. Both projects have the same simple plugin source code, and each has this plugin (referenced from the root of the repo (aka the result of the build)) and eslint installed as devDependencies. Each also has a lint package.json script that runs eslint on the simple plugin.

`/e2e/all`

This fixture has a regular js config and runs our all config on the project.

`/e2e/all-typed-config`

This fixture has typescript installed and uses a ts-based config and runs our all config on the project. It also executes tsc as part of the lint command, to ensure everything's typed property.

This change adds a new end-to-end test capability. There are two new jobs in the main CI workflow. One job to build the package (which also good to have outside of the need to run e2e tests), executed with the latest LTS node. The build artifact is uploaded for the downstream `e2e` job to use. The `e2e` job, downloads the artifact, run `npm install` and then `npm run e2e`, which executes a custom node script to loop through subdirectories of `/e2e`, which each contain a project fixture with different setups. Both projects have the same simple plugin source code, and each has this plugin (referenced from the root of the repo (aka the result of the build)) and `eslint` installed as `devDependencies`. Each also has a `lint` package.json script that runs `eslint` on the simple plugin. \## `/e2e/all` This fixture has a regular js config and runs our `all` config on the project. \## `/e2e/all-typed-config` This fixture has `typescript` installed and uses a ts-based config and runs our `all` config on the project. It also executes `tsc` as part of the `lint` command, to ensure everything's typed property.

bmish

Neat.

I also ran this through AI if you'd like to have a look at what it thought: https://chatgpt.com/share/68993709-d248-800c-a59f-c765c9b7b043

e2e/all/.npmrc

eslint.config.ts

.npmpackagejsonlintignore

e2e/runAllTests.js

bmish · 2025-08-11T00:14:18Z

.github/workflows/main.yml

+          - '20.19.0' #minimum supported v20
+          - 20
+        os:
+          - ubuntu-latest


Looks like we may be missing tests on Windows. Should we test that here?

Sure, I can add that. Do you want it added to the unit test matrix too?

Added to the e2e job. If you want me to add it to the test job, I can do that too.

Generally would be good to test on both. Hopefully it doesn't hurt perf too much. Also could be done separately.

.github/workflows/main.yml

michaelfaith · 2025-08-11T18:03:55Z

I was thinking that it might be good to add an intentional violation (like not including meta.schema for instance), with an inline disable, and then including linterOptions.reportUnusedDisableDirectives in the config. the idea being that if our plugin isn't correctly being applied (and thus not reporting any violations), that would alert us. What do you think?

Example output

bmish · 2025-08-12T01:39:22Z

e2e/runAllTests.js

+/**
+ * Run All Tests: This script executes the lint command on all subfolders under `fixtures`, in order
+ * to validate the correctness of our plugin.  Each fixture installs the *built* package.  So this should
+ * only be run after a build has been done.


Out of curiosity, how is the performance impact of this on our builds?

In total, it adds around 40s to a minute to the overall CI runtime. Mainly because the E2E steps have to wait for the build to complete. Just looking at the build readout below, it took 38s for the build job, which ran in parallel to all of the other existing jobs. At that point all the E2E jobs kicked off in parallel, and the longest running of those are the windows ones, which took about a minute each (the ubuntu ones took between 40-50s).

RE: performance, though, there will be a roughly linear impact to the length of those steps, for each e2e test that's added in the future. So, one thing to consider, if many more are added, would be to run some number of them concurrently. I wouldn't necessarily go to that right now, because it adds some amount of complexity. With just two tests, there's not much to gain with doing them concurrently. But if there were 10 or more, it might start to make sense.

michaelfaith · 2025-08-12T11:51:36Z

Btw, I'm thinking it might be a good idea to move the prettier config back to js, until the VSCode extension supports it. prettier/prettier-vscode#3623 IDE-autoformatting isn't working anymore. What do you think?

e2e/.npmrc

bmish

Thanks!

bmish · 2025-08-12T13:00:42Z

Btw, I'm thinking it might be a good idea to move the prettier config back to js, until the VSCode extension supports it. prettier/prettier-vscode#3623 IDE-autoformatting isn't working anymore. What do you think?

Up to you on that. If it's going to be broken for a while, then could make sense to revert. If it's going to be fixed soonish, then could wait.

aladdin-add · 2025-10-19T08:16:39Z

Is there a way to simplify this process? This change introduces 10+ jobs, and I wonder if it's really necessary - although grateful for Microsoft's generosity in offering it for free to open-source projects, I still prefer to minimize the resource consumption.

michaelfaith · 2025-10-19T12:46:07Z

Yeah, it's 12: 6 node versions * two OSs.

Node Matrix

Latest and minimum supported version of node 20
Latest and minimum supported version of node 22
Latest and minimum supported version of node 24
Total: 6

OS Matrix

Windows
Ubuntu

So, I mean, the only way to reduce that is to remove some from those matrices, which would mean you're not testing the full range of versions that the library claims to support.

aladdin-add · 2025-10-19T16:17:25Z

We already have some compatibility tests related to Node versions, such as n/no-unsupported-features/node-builtins.

Moreover, based on my experience maintaining this package, we have hardly received any issues related to Node.js/OS compatibility. So I feel like we're paying a little too many resources on things with a very low probability. 😄

michaelfaith · 2025-10-19T16:24:02Z

Idk, there's been a lot of changes recently to node that directly affect tooling (e.g. require(ESM), type stripping, etc). And so, I think testing the various versions of node is important, since it can avoid blind spots. RE: OSes, I know that we've had issues with Windows compatibility on some of our stuff in eslint-plugin-package-json in the past, and adding tests for windows helped avoid those from coming back. To be honest, it feels like a low cost way of giving you confidence that the support that you're claiming is there, is actually there. 🤷‍♂️

aladdin-add · 2025-10-19T16:36:19Z

right, you sold me!

michaelfaith marked this pull request as ready for review August 10, 2025 20:33

bmish reviewed Aug 11, 2025

View reviewed changes

michaelfaith added 2 commits August 11, 2025 12:35

add cache to node setup and windows to e2e matrix

4dc292f

remove cache option from ci jobs

1a37f3f

michaelfaith added 3 commits August 11, 2025 13:06

add newlint to npmpackagejsonlintignore

0979736

Make script output more actionable, and add tsdoc

7be4d22

add inline disables

3661b44

bmish reviewed Aug 12, 2025

View reviewed changes

michaelfaith added 3 commits August 12, 2025 05:27

Move e2e test dirs into fixtures

32a1987

Move e2e test dirs into fixtures

2cb4950

Format

de3ae93

bmish reviewed Aug 12, 2025

View reviewed changes

e2e/.npmrc Outdated Show resolved Hide resolved

Update e2e/.npmrc

4aafaef

bmish approved these changes Aug 12, 2025

View reviewed changes

bmish merged commit dd9df61 into eslint-community:main Aug 12, 2025
20 checks passed

michaelfaith deleted the test/e2e branch October 19, 2025 12:39

Uh oh!

ci: add e2e tests to CI #546

ci: add e2e tests to CI #546

Uh oh!

Conversation

michaelfaith commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

/e2e/all

/e2e/all-typed-config

Uh oh!

bmish left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bmish Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

michaelfaith Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

michaelfaith Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmish Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michaelfaith commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bmish Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

michaelfaith Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelfaith Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

michaelfaith commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

bmish left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bmish commented Aug 12, 2025

Uh oh!

aladdin-add commented Oct 19, 2025

Uh oh!

michaelfaith commented Oct 19, 2025

Node Matrix

OS Matrix

Uh oh!

aladdin-add commented Oct 19, 2025

Uh oh!

michaelfaith commented Oct 19, 2025

Uh oh!

aladdin-add commented Oct 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

michaelfaith commented Aug 10, 2025 •

edited

Loading

`/e2e/all`

`/e2e/all-typed-config`

michaelfaith Aug 11, 2025 •

edited

Loading

michaelfaith commented Aug 11, 2025 •

edited

Loading

michaelfaith Aug 12, 2025 •

edited

Loading

michaelfaith commented Aug 12, 2025 •

edited

Loading