[3.4] Backport CI config from master #2475

vstinner · 2017-06-28T14:46:07Z

Add .travis.yml for Travis CI
Add .github/ for AppVeyor and CodeCov.

vstinner · 2017-06-28T14:48:05Z

Ok, here is my first attempt for enable Travis CI and AppVeyor to the 3.4 branch. It should help to test automatically backported security fixes.

I'm not sure about enabling CodeCov. Do we need CodeCov on 3.4?

Once this PR will work and be merged, I will backport it to the 3.3 branch.

vstinner · 2017-06-28T14:54:15Z

The doc job failed with:

Warning, treated as error:
unknown config value 'latex_paper_size' in override, ignoring

=> need to backport the commit e7ffb99.

vstinner · 2017-06-28T14:56:00Z

The Travis CI job failed on:

make: *** No rule to make target `regen-all'.  Stop.

Need to remove this from .travis.yml.

At least, the compilation succeeded ;-)

AppVeyor failed on:

C:\projects\cpython>"C:\projects\cpython\PCbuild\\python"  -Wd -E -bb "C:\projects\cpython\PCbuild\..\lib\test\regrtest.py" -uall -u-cpu -rwW --slowest --timeout 1200 -j0   
...
regrtest.py: error: unrecognized arguments: --slowest

Need to replace the option with --slow.

vstinner · 2017-06-28T15:04:26Z

I removed CodeCov config from .github/ and tried to fix CI failures.

vstinner · 2017-06-28T15:10:17Z

New failure of the Travis CI doc job:

Warning, treated as error:
/home/travis/build/python/cpython/Doc/library/xml.dom.minidom.rst:250:Footnote [#] is not referenced.

vstinner · 2017-06-28T15:10:46Z

Failure of the Travis CI python job:

./python  ./Tools/scripts/run_tests.py -j 1 -u all -W --timeout=3600 -j4 -uall,-cpu,-tzdata
...
regrtest.py: error: argument -u/--use: invalid resource: tzdata

zware

Looks mostly fine, a couple of things pointed out below.

zware · 2017-06-28T15:11:44Z

.travis.yml

+  # Only run on Linux as the check only needs to be run once.
+  - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then ./python Tools/scripts/patchcheck.py --travis $TRAVIS_PULL_REQUEST; fi
+  # `-r -w` implicitly provided through `make buildbottest`.
+  - make buildbottest TESTOPTS="-j4 -uall,-cpu,-tzdata"


s/,-tzdata// (as you probably already saw in the build log :))

Right, I wrote the same fix, again, I will push it shortly.

zware · 2017-06-28T15:13:35Z

.travis.yml

+      after_script:  # Probably should be after_success once test suite updated to run under coverage.py.
+        # Make the `coverage` command available to Codecov w/ a version of Python that can parse all source files.
+        - source ./venv/bin/activate
+        - bash <(curl -s https://codecov.io/bash)


With the codecov config missing, you can just remove this entire build. If you still want GCC coverage, you could change the main Linux build to GCC, since clang is covered by the OSX builder.

Right, I was going to remove it, but I'm looking at failures of the doc job ;-)

vstinner · 2017-06-28T15:41:00Z

I removed the Travis CI doc job. It requires too many changes to fix it. The doc wasn't tested, so it's not like it's a regresion. Moreover, security fixes don't touch Doc/ usually, so it should be fine.

zware · 2017-06-28T16:11:20Z

I removed the Travis CI doc job. It requires too many changes to fix it.

In that case, let's remove the changes in Doc/ as well.

vstinner · 2017-06-28T16:26:00Z

4 tests failed on Windows:

======================================================================
ERROR: test_logincapa_with_client_certfile (test.test_imaplib.RemoteIMAP_SSLTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_imaplib.py", line 473, in test_logincapa_with_client_certfile
    _server = self.imap_class(self.host, self.port, certfile=CERTFILE)
  File "C:\projects\cpython\lib\imaplib.py", line 1222, in __init__
    IMAP4.__init__(self, host, port)
  File "C:\projects\cpython\lib\imaplib.py", line 182, in __init__
    self.open(host, port)
  File "C:\projects\cpython\lib\imaplib.py", line 1235, in open
    IMAP4.open(self, host, port)
  File "C:\projects\cpython\lib\imaplib.py", line 258, in open
    self.sock = self._create_socket()
  File "C:\projects\cpython\lib\imaplib.py", line 1227, in _create_socket
    server_hostname=self.host)
  File "C:\projects\cpython\lib\ssl.py", line 368, in wrap_socket
    _context=self)
  File "C:\projects\cpython\lib\ssl.py", line 586, in __init__
    self.do_handshake()
  File "C:\projects\cpython\lib\ssl.py", line 813, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:600)
======================================================================
ERROR: test_logincapa_with_client_ssl_context (test.test_imaplib.RemoteIMAP_SSLTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_imaplib.py", line 478, in test_logincapa_with_client_ssl_context
    _server = self.imap_class(self.host, self.port, ssl_context=self.create_ssl_context())
  File "C:\projects\cpython\lib\imaplib.py", line 1222, in __init__
    IMAP4.__init__(self, host, port)
  File "C:\projects\cpython\lib\imaplib.py", line 182, in __init__
    self.open(host, port)
  File "C:\projects\cpython\lib\imaplib.py", line 1235, in open
    IMAP4.open(self, host, port)
  File "C:\projects\cpython\lib\imaplib.py", line 258, in open
    self.sock = self._create_socket()
  File "C:\projects\cpython\lib\imaplib.py", line 1227, in _create_socket
    server_hostname=self.host)
  File "C:\projects\cpython\lib\ssl.py", line 368, in wrap_socket
    _context=self)
  File "C:\projects\cpython\lib\ssl.py", line 586, in __init__
    self.do_handshake()
  File "C:\projects\cpython\lib\ssl.py", line 813, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:600)
----------------------------------------------------------------------

======================================================================
FAIL: test_walk_bottom_up (test.test_os.WalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_os.py", line 828, in test_walk_bottom_up
    self.sub2_tree)
AssertionError: Tuples differ: ('@test_856_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3']) != ('@test_856_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
First differing element 1:
['broken_link', 'link']
['link']
- ('@test_856_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3'])
?                                                ----------
+ ('@test_856_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
?                                ++++++++++
======================================================================
FAIL: test_walk_prune (test.test_os.WalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_os.py", line 808, in test_walk_prune
    self.assertEqual(all[1], self.sub2_tree)
AssertionError: Tuples differ: ('@test_856_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3']) != ('@test_856_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
First differing element 1:
['broken_link', 'link']
['link']
- ('@test_856_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3'])
?                                                ----------
+ ('@test_856_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
?                                ++++++++++
======================================================================
FAIL: test_walk_topdown (test.test_os.WalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_os.py", line 791, in test_walk_topdown
    self.assertEqual(all[3 - 2 * flipped], self.sub2_tree)
AssertionError: Tuples differ: ('@test_856_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3']) != ('@test_856_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
First differing element 1:
['broken_link', 'link']
['link']
- ('@test_856_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3'])
?                                                ----------
+ ('@test_856_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
?                                ++++++++++
----------------------------------------------------------------------

vstinner · 2017-06-28T16:26:23Z

On Travis CI, the same 2 test_imaplib tests failed.

vstinner · 2017-06-28T16:33:17Z

@zware: "In that case, let's remove the changes in Doc/ as well."

Oh, it's a mistake. I reverted most Doc changes, but I forgot one. It's now fixed.

I backported another fix for test_imaplib which removes the 2 failing tests.

vstinner · 2017-06-28T16:45:39Z

Oh yeah! Travis CI now pass. AppVeyor will likely fail in test_os again, I didn't identifier the right commit to cherry-pick yet.

@serhiy-storchaka: Do you recall which commit fixed test_os random failures on os.walk on Windows? see #2475 (comment) failures

serhiy-storchaka · 2017-06-28T17:13:07Z

See bpo-23808 and bpo-25911. Seems this issue was not fixed in 3.4.

Maybe removing the True argument from os.symlink('broken', broken_link_path, True) could help.

brettcannon

This review is only for the Travis file.

vstinner · 2017-06-28T20:17:02Z

Ok, the Python test suite now pass on Linux (Travis CI) and Windows (AppVeyor).

Summary of the PR:

Backport Travis CI and AppVeyor configuration
Fix 4 failing tests in test_os and test_smtplib: backport 2 changes which only impact tests, not the stdlib
Hum, that's all :-)

This PR doesn't contain:

CodeCov configuration: I consider that it's useless for 3.4, it's not like we want to enhance the code coverage in this branch
CI docs tests: fixing the doc job (fix warnings, not errors, for Sphinx 1.6.1) requires to backport 3 big changes which creates a lot of conflicts, and so there is a high risk of introducing bugs in the doc ... whereas the purpose of a CI is to prevent mistakes :-)

@brettcannon: I really prefer to fix all CI issues in a single PR, and not have to handle 2 or 3 PR to get a working CI.

@zware: Would you mind to review this PR? You worked on these CI recently, so you should be able to check it :-)

@larryhastings: Would you mind to review this PR? It should help to be more confident when merging security fixes in 3.4, since the 3.4 buildbots were removed.

Thanks @serhiy-storchaka, I backported you bpo-23808 fix and it seems like it was just enough ;-)

zware

LGTM, though I have added a suggestion.

zware · 2017-06-28T20:26:01Z

.travis.yml

+        exit
+      fi
+      ./configure --with-pydebug
+      make -j4


We should perhaps keep the make clinic check. That might require also backporting .gitattributes (which might not be a bad idea anyway).

We should perhaps keep the make clinic check.

I don't think that it's worth it to add too many tests. I expect less than 10 commits in 3.4 next months, and each commit is carefully reviewed, and is very likely simply a backport.

That might require also backporting .gitattributes (which might not be a bad idea anyway).

This is for Windows end of line? I tried to forget this one. Tests pass on Windows on AppVeyor. Do we really need gitattributes?

Anyway, I suggest to create a separated change once this one is merged, if you consider that we should backport gitattributes.

brettcannon · 2017-07-05T01:11:37Z

@Haypo I have no problem with rolling up the fixes to go with the .travis.yml file. All I meant was that I only looked at the .travis.yml file and not the fixes themselves.

larryhastings · 2017-07-11T12:18:32Z

Lib/test/test_imaplib.py

@@ -468,16 +468,6 @@ def test_logincapa(self):
            _server = self.imap_class(self.host, self.port)
            self.check_logincapa(_server)

-    def test_logincapa_with_client_certfile(self):


Why were these tests removed?

See the commit 2bef165e1a142b83bc3f14e1017cd2aad50c9af8 of this PR which contains the full context, it's just a backport; http://bugs.python.org/issue30231

Okay. Why is this bugfix part of the "Backport CI config from master" PR? As far as I can tell, it has nothing to do with backporting the config for AppVeyor, CodeConv, and Travis.

Unless this bugfix is necessary for running AppVeyor, CodeConv, or Travis, I would like this change removed from the PR.

@larryhastings: Please see my summary of this PR, #2475 (comment)

Technically, I don't think that you can merge a PR if the CI fails. So it's not posisble to add the CI config in one PR and then fix failing tests. I could do the opposite, but it's not possible to make a PR depends on another one. Moreover, I used this issue to see which tests on the CI and test my fixes.

I think this reply below was actually meant to be in-line here. I am replying to it here.

2bef165 says "Remove the two tests which are already skipped." If they're skipped, then they're not failing, and they don't need to be removed.

Are they actually failing?

Oh, I see. I used "git cherry-pick -x" which keeps the commit message unchanged. In Python 3.6 and master, first I skipped the test using skipIf(True, "...") to take decide what to do with these tests. But later, we decided that it's not possible to fix the test, so I removed them in all branches.

The IMAP server now always reject our TLS certificate, so the test always fail. The question is more why the server accepted your TLS certificate in the past? :-)

larryhastings · 2017-07-11T12:18:48Z

Lib/test/test_os.py

@@ -770,7 +770,11 @@ def setUp(self):
        if support.can_symlink():
            os.symlink(os.path.abspath(t2_path), self.link_path)
            os.symlink('broken', broken_link_path, True)
-            self.sub2_tree = (sub2_path, ["link"], ["broken_link", "tmp3"])
+            if os.path.isdir(broken_link_path):


Why was this code modified?

See bpo-23808.

Okay. Why is this bugfix part of the "Backport CI config from master" PR? As far as I can tell, it has nothing to do with backporting the config for AppVeyor, CodeConv, and Travis.

Unless this bugfix is necessary for running AppVeyor, CodeConv, or Travis, I would like this change removed from the PR.

The CI fails without this fix. If the CI fails, you cannot merge this PR.

Moreover, a prefer a green CI to more easily spot failures, than to have to compare with the previous CI build to check if there are new failures ;-)

Okay, I'll accept this into 3.4, Even Though It's In Security Fixes Only Mode etc etc etc.

larryhastings · 2017-07-11T12:45:46Z

FWIW, there's some automation when cutting releases that removes dotfiles we don't want to leak into Python release tarballs. New dotfiles like .travis.yaml (and ".mention-bot", and ".github") need to be explicitly pruned. I'm gonna follow up and make sure this happens for 3.6.2 final and 3.6.4rc1. In the future, we release managers would appreciate a heads-up warning us when people add new dotfiles to the root of cpython, so we can make sure we don't ship them.

vstinner · 2017-07-11T20:00:51Z

FWIW, there's some automation when cutting releases that removes dotfiles we don't want to leak into Python release tarballs.

I guess that you use some scripts, are they public?

brettcannon · 2017-07-11T21:36:38Z

@Haypo my guess is https://github.com/python/release-tools, but there's no explicit dot file list that I see there so I may be wrong

larryhastings · 2017-07-12T12:47:14Z

Brett's link to release-tools is the right place. The logic to strip the dotfiles is hidden inside. But please don't touch it right now. @ned-deily has done a lot of work on it to update it to Github, but he hasn't checked in those changes yet. I appreciate you being willing to fix release-tools when you add new dotfiles but please wait until release.py doesn't call "hg" anymore ;-)

vstinner · 2017-07-12T12:57:18Z

@larryhastings: Please see my summary of this PR, #2475 (comment)

Technically, I don't think that you can merge a PR if the CI fails. So it's not posisble to add the CI config in one PR and then fix failing tests. I could do the opposite, but it's not possible to make a PR depends on another one. Moreover, I used this issue to see which tests on the CI and test my fixes.

vstinner · 2017-07-12T13:14:52Z

I just rebased this PR on top of the 3.4 branch, since many security PRs just have been merged. So it would help to check if everything is fine on 3.4 :-)

vstinner · 2017-07-12T13:37:32Z

I just rebased this PR on top of the 3.4 branch, since many security PRs just have been merged. So it would help to check if everything is fine on 3.4 :-)

Oh, everything is not fine, test_subprocess fails: http://bugs.python.org/issue30730#msg298218

(Regression of PR #2362, commit fe82c46).

serhiy-storchaka · 2017-07-12T14:38:22Z

Oh, I forgot that null character/byte errors were of type TypeError before 3.5. The simplest fix is changing corresponding ValueError in self.assertRaises() to the tuple (ValueError, TypeError). You can add 5e22721 to this PR.

* Add .travis.yml for Travis CI * Add .github/ for AppVeyor and CodeCov.

The regen-all Makefile rule doesn't exist in Python 3.4, only since Python 3.5 and newer (and 2.7).

tzdata resource doesn't exist in Python 3.4.

Fixing Sphinx warnings requires to backport huge intrusive changes like: - commit d97b7dc - commit 5c67933

The public cyrus.andrew.cmu.edu IMAP server (port 993) doesn't accept TLS connection using our self-signed x509 certificate. Remove the two tests which are already skipped. (cherry picked from commit 7895a05)

On Windows a symlink can has the FILE_ATTRIBUTE_DIRECTORY flag. (cherry picked from commit 388b90f)

Fix test_invalid_cmd() and test_invalid_env(), TypeError is raised on Python 3.4. (cherry picked from commit 5e22721)

vstinner · 2017-07-13T11:26:14Z

Oh, I forgot that null character/byte errors were of type TypeError before 3.5. The simplest fix is changing corresponding ValueError in self.assertRaises() to the tuple (ValueError, TypeError). You can add 5e22721 to this PR.

It took me one minute to understand that this commit comes from PR #2363 which is not merged yet :-)

I changed the commit message and added it at the end of this PR.

I also rebased this PR on top of 3.4.

vstinner · 2017-07-13T11:50:42Z

Yeah, tests pass again \o/

@larryhastings: Ok, it's your turn ;-)

vstinner · 2017-07-21T09:50:14Z

Ping @larryhastings: does it now look good to you?

vstinner · 2017-07-21T09:50:41Z

(The PR was already approved by 3 core devs, maybe 4 if you include myself.)

larryhastings · 2017-07-22T19:22:49Z

I look forward to 3.4 entering the age of automation!

the-knights-who-say-ni added the CLA signed label Jun 28, 2017

vstinner requested review from brettcannon and zware June 28, 2017 14:46

vstinner added the trivial label Jun 28, 2017

zware reviewed Jun 28, 2017

View reviewed changes

brettcannon approved these changes Jun 28, 2017

View reviewed changes

zware approved these changes Jun 28, 2017

View reviewed changes

vstinner mentioned this pull request Jun 28, 2017

[3.3] Backport CI config from master #2481

Merged

larryhastings reviewed Jul 11, 2017

View reviewed changes

vstinner mentioned this pull request Jul 11, 2017

[3.4] bpo-30500: urllib: Simplify splithost by calling into urlparse. (#1849) #2291

Merged

vstinner and others added 10 commits July 13, 2017 13:15

Backport CI config from master

6922ce8

* Add .travis.yml for Travis CI * Add .github/ for AppVeyor and CodeCov.

Travis CI: remove "make regen-all" check

b298221

The regen-all Makefile rule doesn't exist in Python 3.4, only since Python 3.5 and newer (and 2.7).

appveyor: replace --slowest with --slow

10f9ce0

Travis CI: remove the GCC coverage job

265f464

Travis CI: remove tzdata resource from regrtest

681bb41

tzdata resource doesn't exist in Python 3.4.

Travis CI: remove the doc job

a7ed6fa

Fixing Sphinx warnings requires to backport huge intrusive changes like: - commit d97b7dc - commit 5c67933

appveyor: set version to 3.4.6+

457fb5d

bpo-30231: Remove skipped test_imaplib tests (#1419) (#2193)

14c13a5

The public cyrus.andrew.cmu.edu IMAP server (port 993) doesn't accept TLS connection using our self-signed x509 certificate. Remove the two tests which are already skipped. (cherry picked from commit 7895a05)

Issues #23808, #25911: Trying to fix walk tests on Windows.

2143f31

On Windows a symlink can has the FILE_ATTRIBUTE_DIRECTORY flag. (cherry picked from commit 388b90f)

bpo-30730: Fix test_os tests.

577f926

Fix test_invalid_cmd() and test_invalid_env(), TypeError is raised on Python 3.4. (cherry picked from commit 5e22721)

serhiy-storchaka approved these changes Jul 13, 2017

View reviewed changes

vstinner mentioned this pull request Jul 13, 2017

[3.4] bpo-26617: Ensure gc tracking is off when invoking weakref callbacks. #2695

Merged

brettcannon added skip issue and removed trivial labels Jul 14, 2017

Merge branch '3.4' into travis34

93d03c2

larryhastings merged commit b154917 into python:3.4 Jul 22, 2017

vstinner deleted the travis34 branch August 10, 2017 23:38

vstinner mentioned this pull request Sep 27, 2017

[3.4] bpo-31170: Fix inclusion of expat in Windows build projects #3785

Merged

Uh oh!

[3.4] Backport CI config from master #2475

[3.4] Backport CI config from master #2475

Uh oh!

Conversation

vstinner commented Jun 28, 2017

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

zware left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

zware commented Jun 28, 2017

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

serhiy-storchaka commented Jun 28, 2017

Uh oh!

brettcannon left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner commented Jun 28, 2017

Uh oh!

zware left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brettcannon commented Jul 5, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

larryhastings commented Jul 11, 2017

Uh oh!