Cannot provision new VM

[flibbertigibbet]

Attempting to provision a fresh VM fails on attempting to install docker-compose.

TASK [azavea.docker : Configure Docker] **************************************** changed: [default] => {"changed": true, "checksum": "89dee2cdbad5d5190b31432a4fa095ea428a2bfe", "dest": "/etc/default/docker", "gid": 0, "group": "root", "md5sum": "da9a056689d42b30025051b81962ecc7", "mode": "0644", "owner": "root", "size": 651, "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1548786324.06-135183084308066/source", "state": "file", "uid": 0}

TASK [pfb.docker : Install docker-compose] *************************************
fatal: [default]: FAILED! => {"changed": false, "cmd": "/usr/local/bin/pip2 install docker-compose==1.9.0", "failed": true, "msg": "stdout: Collecting docker-compose==1.9.0\n Downloading https://files.pythonhosted.org/packages/37/bc/82444964a31604aedf78b302cdfc55bc56807aa8a4fafd9b1db85ec3ebe8/docker-compose-1.9.0.tar.gz (156kB)\nCollecting cached-property<2,>=1.2.0 (from docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/3b/86/85c1be2e8db9e13ef9a350aecd6dea292bd612fa288c2f40d035bb750ded/cached_property-1.5.1-py2.py3-none-any.whl\nCollecting docopt<0.7,>=0.6.1 (from docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nRequirement already satisfied: PyYAML<4,>=3.10 in /usr/lib/python2.7/dist-packages (from docker-compose==1.9.0) (3.10)\nCollecting requests!=2.11.0,<2.12,>=2.6.1 (from docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/ea/03/92d3278bf8287c5caa07dbd9ea139027d5a3592b0f4d14abf072f890fab2/requests-2.11.1-py2.py3-none-any.whl (514kB)\nCollecting texttable<0.9,>=0.8.1 (from docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/74/49/23d944a6f7d4dd20846b2df2914fe108c96a253567c140c416edbf270f80/texttable-0.8.8.tar.gz\nCollecting websocket-client<1.0,>=0.32.0 (from docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/26/2d/f749a5c82f6192d77ed061a38e02001afcba55fe8477336d26a950ab17ce/websocket_client-0.54.0-py2.py3-none-any.whl (200kB)\nCollecting docker-py<2.0,>=1.10.6 (from docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/23/c7/1fd6d4d620809fe2f323869d719e2dd0086c939b67021303a9ec40f5a05b/docker_py-1.10.6-py2.py3-none-any.whl (50kB)\nCollecting dockerpty<0.5,>=0.4.1 (from docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/8d/ee/e9ecce4c32204a6738e0a5d5883d3413794d7498fe8b06f44becc028d3ba/dockerpty-0.4.1.tar.gz\nRequirement already satisfied: six<2,>=1.3.0 in /usr/lib/python2.7/dist-packages (from docker-compose==1.9.0) (1.5.2)\nCollecting jsonschema<3,>=2.5.1 (from docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl\nRequirement already satisfied: enum34<2,>=1.0.4 in /usr/local/lib/python2.7/dist-packages (from docker-compose==1.9.0) (1.1.6)\nCollecting docker-pycreds>=0.2.1 (from docker-py<2.0,>=1.10.6->docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/f5/e8/f6bd1eee09314e7e6dee49cbe2c5e22314ccdb38db16c9fc72d2fa80d054/docker_pycreds-0.4.0-py2.py3-none-any.whl\nCollecting backports.ssl-match-hostname>=3.5; python_version < "3.5" (from docker-py<2.0,>=1.10.6->docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/ff/2b/8265224812912bc5b7a607c44bf7b027554e1b9775e9ee0de8032e3de4b2/backports.ssl_match_hostname-3.7.0.1.tar.gz\nRequirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from docker-py<2.0,>=1.10.6->docker-compose==1.9.0) (1.0.22)\nCollecting functools32; python_version == "2.7" (from jsonschema<3,>=2.5.1->docker-compose==1.9.0)\n Downloading https://files.pythonhosted.org/packages/c5/60/6ac26ad05857c601308d8fb9e87fa36d0ebf889423f47c3502ef034365db/functools32-3.2.3-2.tar.gz\nBuilding wheels for collected packages: docker-compose, docopt, texttable, dockerpty, backports.ssl-match-hostname, functools32\n Building wheel for docker-compose (setup.py): started\n Building wheel for docker-compose (setup.py): finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/0c/8e/a2/388a468c784be62dfe7d276cc5fa0d8d2440084237814954b2\n Building wheel for docopt (setup.py): started\n Building wheel for docopt (setup.py): finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\n Building wheel for texttable (setup.py): started\n Building wheel for texttable (setup.py): finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/e6/50/66/cffbd36db351677a36e03cc8f071e519cbf6a907ef9bad117d\n Building wheel for dockerpty (setup.py): started\n Building wheel for dockerpty (setup.py): finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/e5/1e/86/bd0a97a0907c6c654af654d5875d1d4383dd1f575f77cee4aa\n Building wheel for backports.ssl-match-hostname (setup.py): started\n Building wheel for backports.ssl-match-hostname (setup.py): finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/f1/ad/d7/c184cf1ef67b07f98565dc52f91ed4f9a759b7b46674b2fd76\n Building wheel for functools32 (setup.py): started\n Building wheel for functools32 (setup.py): finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/b5/18/32/77a1030457155606ba5e3ec3a8a57132b1a04b1c4f765177b2\nSuccessfully built docker-compose docopt texttable dockerpty backports.ssl-match-hostname functools32\nInstalling collected packages: cached-property, docopt, requests, texttable, websocket-client, docker-pycreds, backports.ssl-match-hostname, docker-py, dockerpty, functools32, jsonschema, docker-compose\n Found existing installation: requests 2.2.1\n\n:stderr: /usr/local/lib/python2.7/dist-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.\n utils.DeprecatedIn23,\nDEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.\n/usr/local/lib/python2.7/dist-packages/pip/vendor/urllib3/util/ssl.py:354: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n SNIMissingWarning\n/usr/local/lib/python2.7/dist-packages/pip/vendor/urllib3/util/ssl.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n InsecurePlatformWarning\n/usr/local/lib/python2.7/dist-packages/pip/vendor/urllib3/util/ssl.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n InsecurePlatformWarning\nCannot uninstall 'requests'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\n"}

RUNNING HANDLER [azavea.docker : Restart Docker] *******************************
to retry, use: --limit @/vagrant/deployment/ansible/pfb.retry

PLAY RECAP *********************************************************************
default : ok=19 changed=15 unreachable=0 failed=1

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

[flibbertigibbet]

The python-requests apt package (not the pypi version) is installed on the VM host, which appears to be related.

[jeancochrane]

The traceback looks similar to the problems we saw in project-icp/bee-pollinator-app#441. Might be an issue with pip 19. Does applying a fix similar to project-icp/bee-pollinator-app#447 fix the build error?

[flibbertigibbet]

No, that and azavea/cac-tripplanner#1098 were about AMI building. This is a vagrant provisioning issue.

[flibbertigibbet]

After pinning the pip role's version in group_vars, docker-compose install fails with

InsecurePlatformWarning\nCannot uninstall 'requests'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\nYou are using pip version 18.1, however version 19.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.

The apt version of requests seems to have a couple of dependent packages:

vagrant@pfb-network-connectivity:/vagrant$ sudo apt-get purge python-requests
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  cloud-init* python-requests* ssh-import-id*

[KlaasH]

I logged into the partially-provisioned machine and tried the failing command (sudo pip install docker-compose==1.9.0) with different versions of pip. It gives the

Cannot uninstall 'requests'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

error for everything after 9.0.3 (that is, 10.x, 18.x, and 19.x. pip versions are crazy).

The azavea.pip role is coming in via azavea.aws-cli and azavea.docker, with no version specified in the former and version 1.0.0 in the latter. Version 1.0.0 of azavea.pip specifies pip_version: "9.0.*", but the condition that checks the installed version and decides whether to reinstall or move on uses search(pip_version) in the filter. I think the release of pip 19 caused this VM to jump all the way from 9.0.3 to 19.0.1 because that matches search("9.0.*"), so it thinks the installed version is OK.

[jeancochrane]

Great digging @KlaasH, I think you're on to something. Pinning pip_version in group_vars to a specific patch release of 9.0.x that doesn't overlap at all with 19.x, e.g. 9.0.3, allows me to get past this error during provisioning.

The quick fix to this issue is to just pin pip_version to 9.0.3. I think the slower but more sustainable fix is to update ansible-pip so that the version filter is more robust to double-digit major versions.