Pipenv fails to install latest version of pyarrow on linux

[HeavenVolkoff]

Pipenv fails to install and sync latest version of pyarrow on linux. As far as I could tell this is due to a difference between their versioning of the latest release:

• linux, windows: 0.9.0 (https://pypi.python.org/pypi/pyarrow)
• macOS: 0.9.0.post1 (https://pypi.python.org/pypi/pyarrow/0.9.0.post1)

When I attempt to install pyarrow on a linux, pipenv returns the following:

And on a macOS, it works:

Howerver if I specify the version to: pyarrow==0.9.0
On linux, it works:

And on macOS, it does not:

I know that this is a edge case, and according to here, it should be fixed when the pyarrow team releases a working macOS wheel of the 0.9.0 version. However pip can correctly identify and install each OS specific version. So, I, personally, think this should work on pipenv too.

$ python -m pipenv.help output

Pipenv version: '11.9.0'

Pipenv location: '/usr/local/lib/python3.6/dist-packages/pipenv'

Python location: '/home/vasv/.local/bin/python'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 3.6.3: /home/vasv/.local/bin/python

• 2.7.14: /usr/bin/python

• 2.7.14: /usr/bin/python2

• 3.6.3: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.13.0-38-generic',
 'platform_system': 'Linux',
 'platform_version': '#43-Ubuntu SMP Wed Mar 14 15:20:44 UTC 2018',
 'python_full_version': '3.6.3',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• LS_COLORS
• LESSCLOSE
• LANG
• GDM_LANG
• HISTIGNORE
• DISPLAY
• COLORTERM
• NO_AT_BRIDGE
• XDG_VTNR
• SSH_AUTH_SOCK
• MANDATORY_PATH
• S_COLORS
• XDG_SESSION_ID
• XDG_GREETER_DATA_DIR
• USER
• DESKTOP_SESSION
• DEFAULTS_PATH
• QT_QPA_PLATFORMTHEME
• PWD
• HOME
• SSH_AGENT_PID
• QT_ACCESSIBILITY
• XDG_SESSION_TYPE
• XDG_DATA_DIRS
• XDG_SESSION_DESKTOP
• TERM
• SHELL
• XDG_SEAT_PATH
• SHLVL
• XDG_SEAT
• LANGUAGE
• WINDOWID
• GDMSESSION
• TERMINFO
• LOGNAME
• DBUS_SESSION_BUS_ADDRESS
• XDG_RUNTIME_DIR
• XAUTHORITY
• XDG_SESSION_PATH
• XDG_CONFIG_DIRS
• PATH
• PS1
• PKG_CONFIG_PATH
• PYTHON
• GCC_COLORS
• LESSOPEN
• _
• OLDPWD
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONUNBUFFERED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/vasv/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /home/vasv/Workspace/antenna_protocol

---

Contents of Pipfile ('/home/vasv/Workspace/antenna_protocol/Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pyarrow = ">=0.9.0,<0.9.1"
logzero = "*"
async-timeout = "*"
aioreactive = "*"
uvloop = {version = "*", sys_platform = "!= 'win32'"}

[dev-packages]
pipenv = "*"

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/home/vasv/Workspace/antenna_protocol/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "840ae8c8bc18ece383055eb1462561e4490412e84042a0bc25d5feb13b09f693"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "aioreactive": {
            "hashes": [
                "sha256:201de56d7bc12854dda45a7e289dce49c99f7d3a50296feb48a401992f37b678",
                "sha256:6c31397f2be441fdcf951f74298ddb7bf64670b8e3b34cd0f537b02016eb40a9"
            ],
            "index": "pypi",
            "version": "==0.5.0"
        },
        "async-timeout": {
            "hashes": [
                "sha256:00cff4d2dce744607335cba84e9929c3165632da2d27970dbc55802a0c7873d0",
                "sha256:9093db5b8ddbe4b8f6885d1a6e0ad84ae3155464cbf6877c387605244c285f3c"
            ],
            "index": "pypi",
            "version": "==2.0.1"
        },
        "futures": {
            "hashes": [
                "sha256:9ec02aa7d674acb8618afb127e27fde7fc68994c0437ad759fa094a574adb265",
                "sha256:ec0a6cb848cc212002b9828c3e34c675e0c9ff6741dc445cab6fdd4e1085d1f1"
            ],
            "version": "==3.2.0"
        },
        "logzero": {
            "hashes": [
                "sha256:34fa1e2e436dfa9f37e5ff8750e932bafe0c5abbb42e1f669e4cf5ce1f179142",
                "sha256:818072e4fcb53a3f6fb4114a92f920e1135fe6f47bffd9dc2b6c4d10eedacf27"
            ],
            "index": "pypi",
            "version": "==1.5.0"
        },
        "numpy": {
            "hashes": [
                "sha256:0739146eaf4985962f07c62f7133aca89f3a600faac891ce6c7f3a1e2afe5272",
                "sha256:07e21f14490324cc1160db101e9b6c1233c33985af4cb1d301dd02650fea1d7f",
                "sha256:0f6a5ed0cd7ab1da11f5c07a8ecada73fc55a70ef7bb6311a4109891341d7277",
                "sha256:0fd65cbbfdbf76bbf80c445d923b3accefea0fe2c2082049e0ce947c81fe1d3f",
                "sha256:20cac3123d791e4bf8482a580d98d6b5969ba348b9d5364df791ba3a666b660d",
                "sha256:528ce59ded2008f9e8543e0146acb3a98a9890da00adf8904b1e18c82099418b",
                "sha256:56e392b7c738bd70e6f46cf48c8194d3d1dd4c5a59fae4b30c58bb6ef86e5233",
                "sha256:675e0f23967ce71067d12b6944add505d5f0a251f819cfb44bdf8ee7072c090d",
                "sha256:6be6b0ca705321c178c9858e5ad5611af664bbdfae1df1541f938a840a103888",
                "sha256:719d914f564f35cce4dc103808f8297c807c9f0297ac183ed81ae8b5650e698e",
                "sha256:768e777cc1ffdbf97c507f65975c8686ebafe0f3dc8925d02ac117acc4669ce9",
                "sha256:7f76d406c6b998d6410198dcb82688dcdaec7d846aa87e263ccf52efdcfeba30",
                "sha256:8c18ee4dddd5c6a811930c0a7c7947bf16387da3b394725f6063f1366311187d",
                "sha256:99051e03b445117b26028623f1a487112ddf61a09a27e2d25e6bc07d37d94f25",
                "sha256:a1413d06abfa942ca0553bf3bccaff5fdb36d55b84f2248e36228db871147dab",
                "sha256:a7157c9ac6bddd2908c35ef099e4b643bc0e0ebb4d653deb54891d29258dd329",
                "sha256:a958bf9d4834c72dee4f91a0476e7837b8a2966dc6fcfc42c421405f98d0da51",
                "sha256:bb370120de6d26004358611441e07acda26840e41dfedc259d7f8cc613f96495",
                "sha256:d0928076d9bd8a98de44e79b1abe50c1456e7abbb40af7ef58092086f1a6c729",
                "sha256:d858423f5ed444d494b15c4cc90a206e1b8c31354c781ac7584da0d21c09c1c3",
                "sha256:e6120d63b50e2248219f53302af7ec6fa2a42ed1f37e9cda2c76dbaca65036a7",
                "sha256:f2b1378b63bdb581d5d7af2ec0373c8d40d651941d283a2afd7fc71184b3f570",
                "sha256:facc6f925c3099ac01a1f03758100772560a0b020fb9d70f210404be08006bcb"
            ],
            "version": "==1.14.2"
        },
        "pyarrow": {
            "hashes": [
                "sha256:5d06a1df6c1c43a122cd05077a3d47ae8a53dabc0f3620aba78c7d434075b2c8",
                "sha256:92c7441d3a57870e34a47bd30d55dfd4c5d49d29893e3221955608e4a10fa57d",
                "sha256:df1b0b67ab4029628264219c5f236f3c828c030e14b455cfd2b2a9d41cf3609b"
            ],
            "index": "pypi",
            "version": "==0.9.0.post1"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        },
        "uvloop": {
            "hashes": [
                "sha256:01cf7199728867f406ba5af78cc47c80acd663ccc52cae105e737a997f1b2bca",
                "sha256:68574150720a380509a3409bf2941be0199cfdacff144a97502fb29c250ba927",
                "sha256:7ee14835a75c72227d3f8a3f370519a3106a6f02e5453f275f16437ebdb92953",
                "sha256:7fba5f390db607b2f026bc598df6b2a2a9e062bffe82910b5ffe2b88560135e5",
                "sha256:89c3bfaad77625490c42a6b99af1879234767ab0c31dd193486a909506e5e549",
                "sha256:b057ef2b0d0162c1ef257f43a95f59bfec37ee9a75cc5412d6b7f9ac6d1d69cb",
                "sha256:e7c871ba3edd5fcf2afb756de88a9a65245070161e24f75abe79c0a241bb8c76"
            ],
            "index": "pypi",
            "markers": "sys_platform != 'win32'",
            "version": "==0.9.1"
        }
    },
    "develop": {
        "certifi": {
            "hashes": [
                "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296",
                "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d"
            ],
            "version": "==2018.1.18"
        },
        "pipenv": {
            "hashes": [
                "sha256:325b26672172f11cf46ee5fa31cc4c751df24f920617ca21eeb68fce70a394e2",
                "sha256:dab6b2e5347757cda7973e6c6065c30c020c5ff8b0687306b83e962aea4adc6c"
            ],
            "index": "pypi",
            "version": "==11.10.0"
        },
        "virtualenv": {
            "hashes": [
                "sha256:1d7e241b431e7afce47e77f8843a276f652699d1fa4f93b9d8ce0076fd7b0b54",
                "sha256:e8e05d4714a1c51a2f5921e62f547fcb0f713ebbe959e0a7f585cc8bef71d11f"
            ],
            "version": "==15.2.0"
        },
        "virtualenv-clone": {
            "hashes": [
                "sha256:4507071d81013fd03ea9930ec26bc8648b997927a11fa80e8ee81198b57e0ac7",
                "sha256:b5cfe535d14dc68dfc1d1bb4ac1209ea28235b91156e2bba8e250d291c3fb4f8"
            ],
            "version": "==0.3.0"
        }
    }
}

$ python -m pipenv.help (Linux):

$ python -m pipenv.help output

Pipenv version: '11.9.0'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.6'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/local/bin/python3.6m

• 3.6: /usr/local/bin/python3.6

• 3.6.5: /usr/local/opt/python/libexec/bin/python

• 2.7.10: /usr/bin/python

• 3.6.5: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.5.0: Mon Mar  5 22:24:32 PST '
                     '2018; root:xnu-4570.51.1~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

• LANG
• XPC_FLAGS
• HISTIGNORE
• DISPLAY
• TERM_SESSION_ID
• EDITOR
• JAVA_HOME
• SSH_AUTH_SOCK
• USER
• LSCOLORS
• PWD
• HOME
• LC_CTYPE
• TERM_PROGRAM
• TERM_PROGRAM_VERSION
• Apple_PubSub_Socket_Render
• TMPDIR
• XPC_SERVICE_NAME
• SHELL
• TERM
• SHLVL
• CPPFLAGS
• LOGNAME
• LDFLAGS
• PATH
• PS1
• PYTHON
• HOMEBREW_GITHUB_API_TOKEN
• _
• OLDPWD
• __CF_USER_TEXT_ENCODING
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONUNBUFFERED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/bin:/usr/local/sbin:/usr/local/opt/python/libexec/bin:/usr/local/opt/go/libexec/bin:/usr/bin:/bin:/usr/sbin:/sbin
• SHELL: /usr/local/bin/bash
• EDITOR: code
• LANG: en_US.UTF-8
• PWD: /Users/hvolkoff/Workspace/Python/antenna_protocol

---

Contents of Pipfile ('/Users/hvolkoff/Workspace/Python/antenna_protocol/Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pyarrow = ">=0.9.0,<0.9.1"
logzero = "*"
async-timeout = "*"
aioreactive = "*"
uvloop = {version = "*", sys_platform = "!= 'win32'"}

[dev-packages]
pipenv = "*"

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/Users/hvolkoff/Workspace/Python/antenna_protocol/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "840ae8c8bc18ece383055eb1462561e4490412e84042a0bc25d5feb13b09f693"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "aioreactive": {
            "hashes": [
                "sha256:201de56d7bc12854dda45a7e289dce49c99f7d3a50296feb48a401992f37b678",
                "sha256:6c31397f2be441fdcf951f74298ddb7bf64670b8e3b34cd0f537b02016eb40a9"
            ],
            "index": "pypi",
            "version": "==0.5.0"
        },
        "async-timeout": {
            "hashes": [
                "sha256:00cff4d2dce744607335cba84e9929c3165632da2d27970dbc55802a0c7873d0",
                "sha256:9093db5b8ddbe4b8f6885d1a6e0ad84ae3155464cbf6877c387605244c285f3c"
            ],
            "index": "pypi",
            "version": "==2.0.1"
        },
        "logzero": {
            "hashes": [
                "sha256:34fa1e2e436dfa9f37e5ff8750e932bafe0c5abbb42e1f669e4cf5ce1f179142",
                "sha256:818072e4fcb53a3f6fb4114a92f920e1135fe6f47bffd9dc2b6c4d10eedacf27"
            ],
            "index": "pypi",
            "version": "==1.5.0"
        },
        "numpy": {
            "hashes": [
                "sha256:0739146eaf4985962f07c62f7133aca89f3a600faac891ce6c7f3a1e2afe5272",
                "sha256:07e21f14490324cc1160db101e9b6c1233c33985af4cb1d301dd02650fea1d7f",
                "sha256:0f6a5ed0cd7ab1da11f5c07a8ecada73fc55a70ef7bb6311a4109891341d7277",
                "sha256:0fd65cbbfdbf76bbf80c445d923b3accefea0fe2c2082049e0ce947c81fe1d3f",
                "sha256:20cac3123d791e4bf8482a580d98d6b5969ba348b9d5364df791ba3a666b660d",
                "sha256:528ce59ded2008f9e8543e0146acb3a98a9890da00adf8904b1e18c82099418b",
                "sha256:56e392b7c738bd70e6f46cf48c8194d3d1dd4c5a59fae4b30c58bb6ef86e5233",
                "sha256:675e0f23967ce71067d12b6944add505d5f0a251f819cfb44bdf8ee7072c090d",
                "sha256:6be6b0ca705321c178c9858e5ad5611af664bbdfae1df1541f938a840a103888",
                "sha256:719d914f564f35cce4dc103808f8297c807c9f0297ac183ed81ae8b5650e698e",
                "sha256:768e777cc1ffdbf97c507f65975c8686ebafe0f3dc8925d02ac117acc4669ce9",
                "sha256:7f76d406c6b998d6410198dcb82688dcdaec7d846aa87e263ccf52efdcfeba30",
                "sha256:8c18ee4dddd5c6a811930c0a7c7947bf16387da3b394725f6063f1366311187d",
                "sha256:99051e03b445117b26028623f1a487112ddf61a09a27e2d25e6bc07d37d94f25",
                "sha256:a1413d06abfa942ca0553bf3bccaff5fdb36d55b84f2248e36228db871147dab",
                "sha256:a7157c9ac6bddd2908c35ef099e4b643bc0e0ebb4d653deb54891d29258dd329",
                "sha256:a958bf9d4834c72dee4f91a0476e7837b8a2966dc6fcfc42c421405f98d0da51",
                "sha256:bb370120de6d26004358611441e07acda26840e41dfedc259d7f8cc613f96495",
                "sha256:d0928076d9bd8a98de44e79b1abe50c1456e7abbb40af7ef58092086f1a6c729",
                "sha256:d858423f5ed444d494b15c4cc90a206e1b8c31354c781ac7584da0d21c09c1c3",
                "sha256:e6120d63b50e2248219f53302af7ec6fa2a42ed1f37e9cda2c76dbaca65036a7",
                "sha256:f2b1378b63bdb581d5d7af2ec0373c8d40d651941d283a2afd7fc71184b3f570",
                "sha256:facc6f925c3099ac01a1f03758100772560a0b020fb9d70f210404be08006bcb"
            ],
            "version": "==1.14.2"
        },
        "pyarrow": {
            "hashes": [
                "sha256:5d06a1df6c1c43a122cd05077a3d47ae8a53dabc0f3620aba78c7d434075b2c8",
                "sha256:92c7441d3a57870e34a47bd30d55dfd4c5d49d29893e3221955608e4a10fa57d",
                "sha256:df1b0b67ab4029628264219c5f236f3c828c030e14b455cfd2b2a9d41cf3609b"
            ],
            "index": "pypi",
            "version": "==0.9.0.post1"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        },
        "uvloop": {
            "hashes": [
                "sha256:01cf7199728867f406ba5af78cc47c80acd663ccc52cae105e737a997f1b2bca",
                "sha256:68574150720a380509a3409bf2941be0199cfdacff144a97502fb29c250ba927",
                "sha256:7ee14835a75c72227d3f8a3f370519a3106a6f02e5453f275f16437ebdb92953",
                "sha256:7fba5f390db607b2f026bc598df6b2a2a9e062bffe82910b5ffe2b88560135e5",
                "sha256:89c3bfaad77625490c42a6b99af1879234767ab0c31dd193486a909506e5e549",
                "sha256:b057ef2b0d0162c1ef257f43a95f59bfec37ee9a75cc5412d6b7f9ac6d1d69cb",
                "sha256:e7c871ba3edd5fcf2afb756de88a9a65245070161e24f75abe79c0a241bb8c76"
            ],
            "index": "pypi",
            "markers": "sys_platform != 'win32'",
            "version": "==0.9.1"
        }
    },
    "develop": {
        "certifi": {
            "hashes": [
                "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296",
                "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d"
            ],
            "version": "==2018.1.18"
        },
        "pipenv": {
            "hashes": [
                "sha256:325b26672172f11cf46ee5fa31cc4c751df24f920617ca21eeb68fce70a394e2",
                "sha256:dab6b2e5347757cda7973e6c6065c30c020c5ff8b0687306b83e962aea4adc6c"
            ],
            "index": "pypi",
            "version": "==11.10.0"
        },
        "virtualenv": {
            "hashes": [
                "sha256:1d7e241b431e7afce47e77f8843a276f652699d1fa4f93b9d8ce0076fd7b0b54",
                "sha256:e8e05d4714a1c51a2f5921e62f547fcb0f713ebbe959e0a7f585cc8bef71d11f"
            ],
            "version": "==15.2.0"
        },
        "virtualenv-clone": {
            "hashes": [
                "sha256:4507071d81013fd03ea9930ec26bc8648b997927a11fa80e8ee81198b57e0ac7",
                "sha256:b5cfe535d14dc68dfc1d1bb4ac1209ea28235b91156e2bba8e250d291c3fb4f8"
            ],
            "version": "==0.3.0"
        }
    }
}

$ python -m pipenv.help (macOS):

---

Expected result

Pipenv should correctly install each OS specific version.

Actual result

Pipenv fails to install latest version of pyarrow under Linux

Steps to replicate

Install pyarrow under Linux with pipenv

[techalchemy]

I get your frustration but you must understand that pip and pipenv are fundamentally different. Pipenv can also install the correct version, the issue is dependency resolution which is non-deterministic. This means that what ends up in your lockfile is to some extent dependent on what the author of the package does in their setup file. If they simply don’t cut a release for one specific platform I’m afraid there isn’t much we can do.

[HeavenVolkoff]

Hey, thanks for the quick answer.
About the issue. I do understand that pipenv and pip are very different things. However, it was my understanding that pipenv would install the latest available release of a package, if that was inside the required version range. In this case, on linux that would be the 0.9.0 and on macOS the 0.9.0.post1, considering that I installed without any version constraint.

[uranusjr]

@HeavenVolkoff It’s a pretty difficult situation. Generally Pipenv tries to use the same version if possible, so your build is stable across platforms, which is the point of a lock file. You wouldn’t be able to guarantee deterministic builds if it always installs the latest version, and this is generally what you want if you use Pipenv.

The best solution to this would be for pyarrow to release 0.9.0post1 for all platforms. This is the best practice for platforms with well-established version locking practices, since it is not as reasonable to require users to specify different versions on cross-platform builds. Python, unfortunately, does not have a well-established culture in this aspect, but it is still a nice thing to do anyway.

With that said, maybe Pipfile should be able to specify a package with different versions based on environment markers, as @frostming mentioned—see #1967 (comment). This would provide a workaround to this problem.

[HeavenVolkoff]

I understand the problem, and agree that the best would be for pyarrow to standardise it's release version across platforms. But the possibility for multiple entries with different markers in Pipfile would be a good work-around IMHO. Anyway, thanks for the help, I will put a comment in the other issue in favor of this feature.

[techalchemy]

It’s not about having multiple entries or not. If you run resolution on platforms A or B, you get one result, but if you run resolution on platform C, you get a different result. Since these platforms are different and resolve differently, and pipenv uses a resolver to resolve dependencies (that is the nature of the tool after all), platforms A and B don’t even know that anything is different about platform C. You see, this knowledge would be impossible to have unless you were platform C. That’s the nature of how pythons packaging ecosystem operates unfortunately.

This would be no problem if in fact the difference were indicated in markers but markers are not the same as version numbers or post releases. Pip doesn’t even touch the wheels for incompatible platforms, so we will literally never know about a version discrepancy like this. So I know you might think you’re asking for something simple (judging by your request that we simply add another entry), but you aren’t. Please appreciate that we don’t enjoy having issues like this, if it were easy to fix we would have done it already.

[HeavenVolkoff]

Hello, I think my issue might have been misunderstood. If you initialize a clean pipenv environment, under Linux, and attempt to install pyarrow, it will fail. Should this be the correct behavior for pipenv? Even considering pyarrow versions discrepancies under different platforms.

[techalchemy]

@HeavenVolkoff it shouldn't fail in a clean environment, but what you posted was installing from an existing lockfile -- can you give us the logs from a clean install if that is failing?

[HeavenVolkoff]

Sure, here it is...
install.txt
pipenv.txt

[techalchemy]

Well based on those logs pyarrow requires futures which you can never install on python 3 so I don’t see how this will ever work

[HeavenVolkoff]

So, that's the weird thing.
if I install pyarrow using only pip, inside the pipenv venv, it install correctly.
Here's the log: install.pip.txt
It also installs correctly with pipenv install --skip-lock pyarrow.
Here's the log: install.skip.txt

[techalchemy]

Oh. Upgrade pipenv. You were hitting a resolver bug that should be fixed.

[HeavenVolkoff]

Hi, I am on the latest pipenv release, and it still fails:

[kadrach]

Also seeing this issue, latest pipenv.

[uranusjr]

Can’t we just tell pyarrow to please make the post1 version available for every platform? Did anyone actually try to raise the issue to them?

[kadrach]

@uranusjr I don't understand the cause of the issue well enough. I can pip install pyarrow just fine, but pipenv cannot. What is platform in this context?

pipenv install --skip-lock pyarrow also works.

[uranusjr]

pyarrow 0.9.0 contains a packaging problem (iirc), and the wheels they built for macOS couldn’t be installed. To remedy this (you cannot re-upload a file with the same name on PyPI, in case you’re wondering), they released 0.9.0.post1, which only contains macOS wheels. This means that when you pip install pyarrow, you get 0.9.0.post1 on macOS, and 0.9.0 everywhere else.

When Pipenv does locking, it needs to (well, tries to) make the lock file cross-platform. With a wildcard installation (pyarrow = "*"), it needs to find a latest version for all platforms—but there isn’t one! pyarrow 0.9.0.post1 only works on macOS, and 0.9.0 does not work on macOS (they deleted the broken macOS wheels). That’s why locking fails.

pip works because it only needs to care about the platform you’re running on. There is a valid latest version no matter where it is running, and that’s good enough for pip.

Knowing the root cause, you can actually work around this problem by pipenv install "pyarrow!=0.9.0,!=0.9.0.post1". As long as Pipenv does not need to consider those versions, it can resolve pyarrow correctly (to 0.8.0, until a newer version is released). But please, please acknowledge this issue to the pyarrow team. They need to know this is a problem so not to do this again in the future.

[kadrach]

Thank you for the detailed explanation @uranusjr. There is an existing, closed issue apache/arrow#1950 in the arrow project.

[uranusjr]

@kadrach Thanks for the effort!