Remove the 'continue-on-error' flag when linkcheck fails

[Mariatta]

Let's see if we still have linkcheck errors.

[Mariatta]

Looks like this URL is returning 500 error at the moment.

(     buildworker: line  178) broken    http://trac.buildbot.net/wiki/RunningBuildbotOnWindows#Service - 500 Server Error: Internal Server Error for url: http://trac.buildbot.net/wiki/RunningBuildbotOnWindows

[Mariatta]

I think buildbot is not using trac.buildbot.net anymore.

I found the webarchive equivalent of this, not sure if there is a more recent url:
https://web.archive.org/web/20201113084744/http://trac.buildbot.net/wiki/RunningBuildbotOnWindows

[zware]

It looks like https://docs.buildbot.net/latest/manual/installation/misc.html#launching-worker-as-windows-service is probably about as close as we'll get to a modern equivalent.

[ezio-melotti]

I just noticed this PR while replying to #727.

I think we should make make html and make check required, and make linkcheck optional, since it might fail sporadically. In order to do this we could:

• Split them in two separate jobs, one required, the other optional;
• Keep them together with continue-on-error on the linkcheck and make it required

The former would be ideal, but I'm not sure how to set up the workflow to only set up one environment and doing one checkout before running two separate jobs in the same environment. If that's not possible, I guess we can do it twice, and we could skip the linkcheck job if the build-and-check job fails.

If we choose the latter, it would be useful to get some notification in case of linkcheck failure. I investigated this in #681 (comment), but I couldn't find a way to report the linkcheck failure.

[hugovk]

I agree, I don't think we can ever have linkcheck as mandatory on normal builds. Links come and go, the network can be flakey.

Instead, it's good to run it as non-failing.

The former would be ideal, but I'm not sure how to set up the workflow to only set up one environment and doing one checkout before running two separate jobs in the same environment. If that's not possible, I guess we can do it twice, and we could skip the linkcheck job if the build-and-check job fails.

It's not possible, each job runs in a fresh environment. Having said that, you can upload/download artifacts and use a cache for re-using things.

How about we keep this like this: html and check required, linkcheck runn but non-failing.

• This we can check the current state for each build, go and check if we fixed sone.

And add a (nightly? weekly?) cron that only does linkcheck is allowed to fail?

• This way we get a report and can easily see if there's links to fix

One concern: if the cron is often failing (which it might, and often), we don't want it to show the repo as failed at the top:

[ezio-melotti]

And add a (nightly? weekly?) cron that only does linkcheck is allowed to fail?

It's somewhat unlikely that recent changes will include invalid links (unless some copy-and-paste mishap happened), so running linkcheck on a fixed schedule sounds like a good idea to me.

One concern: if the cron is often failing (which it might, and often), we don't want it to show the repo as failed at the top:

How does reporting work for this kind of checks that are not directly linked to specific PRs? Where can the results be reported?

[AlexWaygood]

How does reporting work for this kind of checks that are not directly linked to specific PRs? Where can the results be reported?

Over at typeshed, we have a test that runs every night. If it fails, a bot auto-creates an issue for us. It works pretty well!

Here's the .yml file: https://github.com/python/typeshed/blob/master/.github/workflows/daily.yml

And here's an example of an issue that the bot auto-created when the test failed: python/typeshed#7572

[hugovk]

How does reporting work for this kind of checks that are not directly linked to specific PRs? Where can the results be reported?

You'd need to go and check the last run on the cron. Due to frequent flakey failures, I see it more as something to check up on once in a while.

Over at typeshed, we have a test that runs every night. If it fails, a bot auto-creates an issue for us. It works pretty well!

Sounds good! Although I doubt we'd want a new issue created every day/week (whatever the schedule is)? Perhaps update a single long-lived issue?

Or if it was monthly, it might be fine to open a new one, if we (or the bot) closes the old one?

[ezio-melotti]

Removing continue-on-error doesn't seem a viable option, so I'm going to close this PR.
I created #917 to track the proposal of adding a workflow that is run regularly.