Require AppVeyor to pass

[brettcannon]

Not sure if we consider it stable enough, but it would be nice to have as a goal to make https://ci.appveyor.com/project/python/cpython be required just like we do for Travis.

@zware do you think this is ready to be flipped on? And is it fast enough?

[zware]

I think it's stable at this point. The last several builds have all been green except for legitimate failures. I also flipped the option for 'rolling builds', which cancels an in-progress build if another commit comes in on the branch, which should help any speed issues.

I haven't been paying close enough attention to tell whether it's fast enough, but in my limited experience AppVeyor has always finished faster than Travis if Travis was doing a full build.

[zware]

I say that, and then test_site fails again: https://ci.appveyor.com/project/python/cpython/build/3.7.0a0.416

[brettcannon]

That's a persnickety failure that keeps popping up. I think once that test failure is dealt with we can flip on the gating.

[zware]

I thought I had dealt with it with python/cpython#624.

[brettcannon]

Is AppVeyor stable enough to turn this on as a requirement, @zware?

[orsenthil]

I have observed flakiness with appveyor recently. I also found it limiting that, unless you are an administrator, you can not re-run suite. Is there a way to address this problem?

[zware]

@orsenthil You're referring to python/cpython#997? I've not seen that failure before. A fairly simple way to force a re-run is to make some bogus commit, revert it, and push it; then you can either reset the branch back to the last 'real' commit and force-push it, or just remember to remove the bogus commits from the summary when you do the squash-merge (which should be done anyway :) ).

I've just looked through all of the failing builds from the past 9 days, and they all appear to be legitimate (aside from the build for the above-linked PR). I think it's probably good enough to turn on, and reevaluate if there are complaints. I also still have python/cpython#954 open; it may or may not improve the situation in test_site, which is historically the most flakey test on AppVeyor.

[pfmoore]

@zware I believe that a simpler approach to re-trigger CI is to close and reopen the PR.

[brettcannon]

With python/cpython@7a99625 and python/cpython@523776c having just landed, I want to see if we can go a week without failures due to AppVeyor itself (people breaking Windows is another thing entirely). If we can go a week then I will flip on AppVeyor as required.

[brettcannon]

So, is this something we want to flip on now?

[zware]

I think so; I think it's been relatively stable lately, and if it causes problems we can always switch it back off.

[brettcannon]

Done!