Document and fix Python installation problems on Big Sur

[armenzg]

This issue is to track all known issues to doing development of Sentry on Big Sur (Mac's latest OS update). I will update the description of this issue as issues are documented and the associated workaround. If possible, I will look to file upstream issues where appropriate.

Known issues

• pyenv cannot install Python 3.6.10 w/o custom command. See comment
  • LANDED
  • Here's the command: make setup-pyenv
    • Equivalent to:

# pyenv 1.2.22 and greater (no need for LDFLAGS
pyenv install --skip-existing --patch 3.6.10 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
# Before pyenv 1.2.22
`LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib ${LDFLAGS}" pyenv install --skip-existing --patch 3.6.10 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)`

• brew install librdkafka seems to be necessary. See comment
  • LANDED

If you don't use the Python version installed via pyenv as we recommend you might encounter this issue:

• Failing to install psycopg2-binary
  • Workaround SYSTEM_VERSION_COMPAT=1 pip install psycopg2-binary

Upstream issues

• (pypa/packaing) [MERGED] add support for macos 11.0, arm64, universal2 #319
  • Released as part of 20.5 or 20.7
• [Waiting for release to remove hack] Add LDFLAGS for zlib on macOS >= 1100 #1711
• psycopg2 issue

[armenzg]

I've installed the Python 3.9 release candidate and it fails in the building of the wheels (as I recalled).

I will determine which packages are having trouble.

Here's the output of one of the failures (this one seems to think we're on mac 10.9; am I reading this right?):

Building wheels for collected packages: confluent-kafka, exam, googleapis-common-protos, grpc-google-iam-v1, hiredis, lxml, maxminddb, mmh3, msgpack, openapi-core, parse, parsimonious, petname, phabricator, Pillow, pyrsistent, python-u2flib-server, PyYAML, rb, setproctitle, simplejson, strict-rfc3339, toronado, uwsgi, wrapt, xmlsec
  Building wheel for confluent-kafka (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/armenzg/code/sentry/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/setup.py'"'"'; __file__='"'"'/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-wheel-qskfoqi8
       cwd: /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/
  Complete output (54 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.9-x86_64-3.9
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka
  copying confluent_kafka/error.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka
  copying confluent_kafka/serializing_producer.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka
  copying confluent_kafka/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka
  copying confluent_kafka/deserializing_consumer.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/avro.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/error.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/serialization
  copying confluent_kafka/serialization/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/serialization
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/admin
  copying confluent_kafka/admin/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/admin
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro
  copying confluent_kafka/avro/error.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro
  copying confluent_kafka/avro/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro
  copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro
  copying confluent_kafka/avro/load.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/kafkatest
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro/serializer
  copying confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro/serializer
  copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro/serializer
  running build_ext
  building 'confluent_kafka.cimpl' extension
  creating build/temp.macosx-10.9-x86_64-3.9
  creating build/temp.macosx-10.9-x86_64-3.9/private
  creating build/temp.macosx-10.9-x86_64-3.9/private/var
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka/src
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -I/Users/armenzg/code/sentry/.venv/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka/src/Admin.c -o build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka/src/Admin.o
  In file included from /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka/src/Admin.c:17:
  /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka/src/confluent_kafka.h:22:10: fatal error: 'librdkafka/rdkafka.h' file not found
  #include <librdkafka/rdkafka.h>
           ^~~~~~~~~~~~~~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for confluent-kafka

[armenzg]

I've been told that another engineer on Big Sur all they had to do is:

brew install librdkafka
brew install zlib
export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"

I will verify this.

[wedamija]

Here's the output of one of the failures (this one seems to think we're on mac 10.9; am I reading this right?):

@taylangocmen Had similar output to that, I don't remember if we figured out why.

[armenzg]

With the OS' Python 3:

❯ /usr/bin/python3 --version
Python 3.8.2
❯ /usr/bin/python3 -m pip install virtualenv
❯ /usr/bin/python3 -m pip install --user --upgrade pip
❯ /usr/bin/python3 -m virtualenv .venv
❯ source .venv/bin/activate
❯ make install-py-dev

It fails to build a wheel for uwsgi

clang: error: no such file or directory: '/Users/armenzg/code/sentry/.venv/lib/python3.8/config-3.8/libpython3.8.a'
  *** error linking uWSGI ***
  ----------------------------------------
  ERROR: Failed building wheel for uwsgi

Having installed the Python 3.9 RC1 installation gets on the way (this shows that having multiple ways of installing Python can get on the way and each developer can end up on a different unique situation):

sentry on  big_sur_wheels [!] via 🐍 v3.8.2 (.venv)
❯ python-config --ldflags
-L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -ldl -framework CoreFoundation

sentry on  big_sur_wheels [!] via 🐍 v2.7.16
❯ python3-config --ldflags
-L/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin -ldl -framework CoreFoundation

I had to unistall it from /Applications and sudo mv /Library/Frameworks/Python.framework Library_Frameworks_Python.framework.

I decided to use brew to get me Python and go from there:

❯ which python3
/usr/local/bin/python3
❯ python3 --version
Python 3.9.0
❯ python3 -m virtualenv .venv
❯ source .venv/bin/activate
❯ pip install uwsgi
# This worked! Let's track the cflags and ldflags
❯ python3-config --cflags
-I/usr/local/Cellar/python@3.9/3.9.0_5/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/usr/local/Cellar/python@3.9/3.9.0_5/Frameworks/Python.framework/Versions/3.9/include/python3.9 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers

❯ python3-config --ldflags
-L/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin -ldl -framework CoreFoundation

Using the brewed Python version I got a lot further:

clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/armenzg/code/sentry/.venv/include -I/usr/local/Cellar/python@3.9/3.9.0_5/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/confluent_kafka/src/Admin.c -o build/temp.macosx-11-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/confluent_kafka/src/Admin.o
    In file included from /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/confluent_kafka/src/Admin.c:17:
    /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/confluent_kafka/src/confluent_kafka.h:22:10: fatal error: 'librdkafka/rdkafka.h' file not found
    #include <librdkafka/rdkafka.h>
             ^~~~~~~~~~~~~~~~~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/armenzg/code/sentry/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/setup.py'"'"'; __file__='"'"'/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-record-3_x6_1nh/install-record.txt --single-version-externally-managed --compile --install-headers /Users/armenzg/code/sentry/.venv/include/site/python3.9/confluent-kafka Check the logs for full command output.

To resolve this running brew install librdkafka seemed sufficient, however, I got this message:

DEPRECATION: confluent-kafka was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.

Conclusion:

• Use Python from brew (so far I've tried with Python 3.9)
• brew install librdkafka

I need to test with Python 3.6 and see what the situation is for pyenv.

[armenzg]

Before I started, the following comment I removed all packages installed via brew to start from a known state. I then installed pyenv through brew.

Pyenv

Install recommended build dependencies:

brew install openssl readline sqlite3 xz zlib

Build Python 3.6.10 with pyenv

armenzg@Armens-MacBook-Pro sentry % pyenv install 3.6.10
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.6.10.tar.xz...
-> https://www.python.org/ftp/python/3.6.10/Python-3.6.10.tar.xz
Installing Python-3.6.10...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.0.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/python-build.20201208134740.78464
Results logged to /var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/python-build.20201208134740.78464.log

Last 10 log lines:
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
./Modules/posixmodule.c:10432:5: warning: code will never be executed [-Wunreachable-code]
    Py_FatalError("abort() called from Python code didn't abort!");
    ^~~~~~~~~~~~~
1 warning generated.
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include   -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes   -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl@1.1/include -I/Users/armenzg/.pyenv/versions/3.6.10/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl@1.1/include -I/Users/armenzg/.pyenv/versions/3.6.10/include   -DPy_BUILD_CORE  -c ./Modules/pwdmodule.c -o Modules/pwdmodule.o
1 warning and 1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....

This can be resolved by using CLAGS, LDFLAGS with --patch (step 1 from this comment is not necessary). I also had to brew install zlib since I have not run brew bundle :

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" pyenv install --patch 3.6.10 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
# There's also this warning:
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?

This set up seems to work.

Brew installation

# We can't get to 3.6.10 anymore
brew install python@3.7
/usr/local/Cellar/python@3.7/3.7.9_2/bin/python3 -m pip install virtualenv
/usr/local/Cellar/python@3.7/3.7.9_2/bin/python3 -m virtualenv .venv
source .venv/bin/activate
make install-py-dev

This set up seems to work.

The platform for each Python installed

pyenv, System & brew installed Python's platform:

% /usr/bin/python3 -c "import platform; print(platform.platform())"
macOS-11.0.1-x86_64-i386-64bit
% /usr/local/bin/python3 -c "import platform; print(platform.platform())"
macOS-11.0.1-x86_64-i386-64bit
% ~/.pyenv/shims/python3 -c "import platform; print(platform.platform())"
Darwin-20.1.0-x86_64-i386-64bit

[joshuarli]

I highly recommend focusing on getting pyenv to work, with a high tolerance for workarounds. Brew python's burned us in the past, especially when they removed 2.7, and you can't get patch version granularity. I'd also avoid system python, you absolutely are at the mercy of their version too and in general I don't mess around with system python because it's easy to break in silent + unexpected ways.

With pyenv we have full control over versions and can easily upgrade people with the tooling revolving around .python-version.

[evanpurkhiser]

Agreed with josh. pyenv is the way to go. If there are problems with it on Big Sur I would work to resolve those, even if it means contributing to pyenv.

[armenzg]

Remove all brew dependencies:

• brew remove --force $(brew list --formula) --ignore-dependencies
• brew remove --cask --force $(brew list --cask) --ignore-dependencies

Remove venv and pyenv:

• rm -rf ~/.pyenv && rm -rf .venv
  Install dependencies:
• brew bundle

Installing pyenv with make setup-pyenv (with local patch) seems to work, however, there's no symlink at ~/.pyenv/bin.

For the curious:

% echo $PATH
/Users/armenzg/.pyenv/bin:/Users/armenzg/.pyenv/shims:/Library/Frameworks/Python.framework/Versions/3.9/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/armenzg/.volta/bin:/Users/armenzg/.pyenv/bin:/Users/armenzg/.pyenv/shims

Meanwhile, while I figure out the above:

~/.pyenv/versions/3.6.10/bin/python3 -m pip install --upgrade pip
~/.pyenv/versions/3.6.10/bin/python3 -m pip install virtualenv
~/.pyenv/versions/3.6.10/bin/python3 -m virtualenv .venv
source .venv/bin/activate
make install-py-dev
make bootstrap
sentry devserver --workers

librdkafka did not seem necessary.

[armenzg]

Note: The PR is a hack until a new pyenv is released and we upgrade the Python version we use.