Allow triggering unstable buildbots with the !buildbot PR comment

[itamaro]

this came up several times recently around testing the NoGIL build on PRs - it's currently not possible to do that using the !buildbot nogil comment (example attempt) because the nogil builders are marked as unstable.
considering the NoGIL build is "experimental", keeping these builders as "unstable" is probably a good idea, but then we need some way to be able to request these builders on PRs.

after chatting with @ambv, I propose that we extend the !buildbot command to support unstable builders.
the mechanics can be discussed in this issue. here are the options I am considering:

1. easy but changes existing behavior: include unstable buildbots by default
2. also easy but kinda hacky: include unstable buildbots if they have a certain tag (e.g. "trigger-on-pr-command")
3. introduce new spelling for explicitly allowing unstable builders (e.g. !buildbot_with_unstable ... or !buildbot[unstable] or !buildbot ... #with_unstable - bikeshedding welcome)

[vstinner]

An alternative is to promote the buildbot to stable.

• https://buildbot.python.org/all/#/builders/1241 : multiple builds succeeded in a row, I would be ok to mark it as STABLE. We can downgrade it to UNSTABLE if it becomes unstable again.
• https://buildbot.python.org/all/#/builders/1225 most builds are success, same.

Do you want to propose a PR to make them STABLE?

[itamaro]

I'm fine with changing these to STABLE, and happy to send a PR to make that change!
I was under the impression that marking a builder as STABLE has broader implications that are not appropriate for an experimental and optional build mode (not sure what my impression was based on, other than my own interpretation of the word "stable" :) )

[ambv]

You're correct, Itamar. Stable buildbots are special for release managers, so at least until the PEP is officially approved, I'd wait with marking nogil BBs as stable.

I believe that the limitation to only allow running stable buildbots with "!buildbot" is unnecessary.

[vstinner]

Oh wait, I was confused about the definition of "stable" here.

For me, unstable means "is known to fail", and stable "is known to be reliable. Don't we have tiers to decide if a builder blocks a released or not? For example, Tier 3 failures do not block a release: https://peps.python.org/pep-0011/#tier-3

[itamaro]

I read the Working with buildbots devguide page again, and it mentions stability in two places:

1. In this section it mentions only stable buildbots will post a message to PRs that break them (makes sense, I wouldn't want to extend this to unstable buildbots)
2. In the section conveniently named "Stability" it mentions stable buildbots are the ones taken into account when making releases

I agree with @vstinner that the semantics of stability vs tiers is confusing. If I combine the devguide with PEP-11, then only TIER_1 and TIER_2 builders can be STABLE, but in practice I see many TIER_3 and NO_TIER builders that are also considered STABLE.

Maybe further clarification is needed. In the meantime, I propose gh-420 to add the unstable builders only to those that can be manually triggered on PRs (@ambv's suggestion above).

[vstinner]

I created PR #422 to fix the Release Status page: only list Tier-1 and Tier-2 builders.

Tier-3 and "No Tier" builders must be omitted there. For example, FreeBSD failures must not be listed there.

[itamaro]

Thanks @vstinner! Should we do gh-420 too? or change the NoGIL builders to stable?