Should there be a Testing WG?

[Trott]

Would it be useful to have a Testing WG? Things a WG might do:

• OMG fix all those flaky tests.
• Create some guidelines around tests. (For example, when should a file containing multiple tests be split into multiple files?)
• Maybe work with the Build WG on the issue of buildbot failures, although I suspect the Build WG is all over that problem. (But just in case: Is there a way to test for buildbot failures? Automate notification to relevant parties so they can be fixed promptly? Is there a way to automate retry when a buildbot fails? Maybe a Test WG can be given limited privileges on the CI infrastructure such that the tedious/mechanistic elements of dealing with buildbot failures can be delegated to them?)
• Maybe see about porting as much of the Python test harness code to Node as is sensible.

Anyone interested?

[Trott]

Oh, and if the Smoke Test WG never got off the ground, maybe there would be interest from a Testing WG to work on citgm.

[jasnell]

Thus far @thealphanerd has been working on evolving citgm and has been making great progress but it would always be great to have additional people. I'm not sure we need yet another WG as much as we just need more people working on it. There have been a ton of test fixes landed here lately and it's great to see.

[cjihrig]

I think putting smoke testing under a more general testing WG would make sense.

[jasnell]

Ref: #2317

[MylesBorins]

Maybe something akin to Tooling?

[jasnell]

I do think a Tooling WG would be very interesting

[MylesBorins]

Seems like a lot of technical challenges to various wg's could fall in here.

Although it may overlap with @nodejs/build a bit

[cjihrig]

I agree. The line between build and tooling would be very blurry.

[jasnell]

build tends to focus specifically on the build toolchain and CI. The kind of tooling I'm thinking would include tools like Rod's iojs-tools, citgm, eslint updates, that sort of thing. Thinking about it tho, that doesn't really put enough focus on the testing aspect. hmm

[jbergstroem]

@jasnell people in build are pretty diverse in how time is spent. Sitting close to all vm's you tend to look at/attempt to fix failed tests, improve tooling (both for the build environment, jenkins and node in general), security, releases and so on. For me, a WG responsible for tests and conformance would make most sense. Stuff like this would suit perfectly in there.

[jasnell]

Right on. OK, I'm on board :) let's do it!
On Nov 17, 2015 2:35 PM, "Johan Bergström" notifications@github.com wrote:

@jasnell https://github.com/jasnell people in build are pretty diverse
in how time is spent. Sitting close to all vm's you tend to look at/attempt
to fix failed tests, improve tooling (both for the build environment,
jenkins and node in general), security, releases and so on. For me, a WG
responsible for tests and conformance would make most sense. Stuff like
this nodejs/build#248 would suit perfectly in
there.

—
Reply to this email directly or view it on GitHub
#3872 (comment).

[joaocgreis]

@Trott regarding your initial comment, we do have a tool to send us an e-mail when buildbots fail, needless to say we had a lot of activity last week. We are tracking the failures in nodejs/build#232 , please call it to our attention if you see any random disconnect. Thanks for the great work solving flaky tests!

[orangemocha]

This sounds like a good idea to me. In the modern days' philosophy of "devs write the tests", I think there is something to be missed from having a dedicated group of people focused on how to improve the quality of the product through testing.

Certainly there is some overlap with the current Build WG responsibilities (especially on the test infra) but that doesn't need to be a problem. And if collaborators want to help improve the test infrastructure, they are welcome to do so regardless of which working groups they belong to 😄

OMG fix all those flaky tests.

I do think this might need to become the responsibility of all collaborators. If a testing WG helps getting traction on flaky tests, that's great. Please join the discussion at nodejs/build#248

Luckily, the buildbot disconnects seem to be fixed.

Adding a few things to the list of things this group might work on:

• Analyze code coverage
• Write stress tests