drop node 12 / matrix against current, and latest 3 LTS node versions

[theinterned]

I noticed in this PR that tests are failing in node 12 but running successfully in node 14, 16, and 18.

Given that node 12 is End-of-Life while 18 is LTS, I think we should just drop testing again 12.

I also changed the matrix to dynamically test against: latest current† (v19), lts/* (v18 is current LTS), lts/-1 (v16), and lts/-2 (v14). See node release schedule. This way these will update as node versions roll along.

There was concern that the dynamic nature would cause confusion when node versions rolled over. To counter-act this, @smockle had the great idea to create an issue when tests start failing on current. So I've set that up:

When tests fail on current, we continue-on-error (meaning the job stays green). Instead we create an issue. Here's an example of that: #378

† Note: I was originally using latest but switched to current. These are aliases, but I switched to current as that's the word used in the node release schedule.

[srt32]

I wonder if we'll get confused / have some unexpected behavior when these things turn over.

Like, the day that a new version gets added, what if CI starts failing? That would be annoying to fix without a hardcoded list. WDYT?

[smockle]

unexpected behavior when these things turn over

Because the matrix includes latest, when lts/* begins pointing to a new version, no failures should occur, because that new version would have already been tested, back when it was latest. So floating version (viz. lts/-2, lts/-1, lts/*) values ought to be safe.

That said, when latest itself begins pointing to a new version, failures might occur. But if we remove latest, then lts/* loses the safety described above.

[smockle]

when latest itself begins pointing to a new version, failures might occur

Here’s one workaround: For latest (and only latest), if any step of the job fails (failure()), open an issue, then exit 0, so the overall workflow run is still green.

So, we’d still be notified of failures, but they wouldn’t block PRs (unless another version is also failing), etc.

[theinterned]

Okay! cool. I got this working!

Here's a sample failed run: https://github.com/github/eslint-plugin-github/actions/runs/3884555154, and the issue it created: #378

To test this I created a failing test 20e83a5 which I will now delete.

[nschonni]

You seem to be going to a lot of effort just to avoid using the version numbers in the matrix. Those changes happen infrequentely and are also a good time to evaluated if major versions of things like ESLint need to change. They usually have a major bump when version support changes

[nschonni]

I'd suggest taking a look at what some of the community supported extensions do like https://github.com/eslint-community/eslint-plugin-promise/blob/main/.github/workflows/CI.yml
The CI matches the supported versions of ESLint, including some older ones that wont be dropped till a semver major release of ESLint. They also have cross version testing of the last major version of ESLint.
Since this is an internal plug-in, you've got more flexibility on choosing what you want to support though.

[theinterned]

You seem to be going to a lot of effort just to avoid using the version numbers in the matrix

Yeah maybe but it was a fun to work on little challenge and I learned some stuff 🤷

Those changes happen infrequentely and are also a good time to evaluated if major versions of things like ESLint need to change. They usually have a major bump when version support changes

Note that current doesn't fail CI — but this kind of signal could alert us that there's work to do to support the next version.

[nschonni]

Note that current doesn't fail CI — but this kind of signal could alert us that there's work to do to support the next version.

Yeah, but that support likely needs to come from ESLint first. Their matrix is also explicit, and is used to test their supported versions https://github.com/eslint/eslint/blob/2952d6ed95811ce0971b6855d66fb7a9767a7b72/package.json#L168-L170

[nschonni]

Anyway, I'm going to unsubscribe from this as I think we're bikeshedding at this point, and it's not worth either of our time

[jfuchs]

Question not blocking.

[jfuchs]

Is this going to create a new issue on every failure?

[theinterned]

Yes it will. Can you think of a way to improve that?

[theinterned]

Maybe this is all too much. Dropping this dynamic approach in favour of #380

It was still a fun little project.