Unable to build for Python 3.11 on macOS

[milosivanovic]

Thanks a lot for making this package. Given that Homebrew might be removing GNU readline for good (as discussed here #61), python-gnureadline seemed exactly like what I needed to hopefully get readline back into the brew-provided Python 3.11. I know that the README says it's only been tested up to Python 3.10, but I'm not sure if that's the reason for the compilation failure I'm getting.

From...

If you want to use this module as a drop-in replacement for readline in the standard Python shell, it has to be installed with the less polite easy_install script found in setuptools. Please take note that easy_install has been deprecated for a while and is about to be dropped from setuptools. Proceed at your own risk!

...I inferred that I should clone this repo and run python3.11 setup.py install since I want readline working in the Python REPL. Did I understand that correctly?

After running that command, the build process seemed to progress nicely until it reached this part:

============ Building the readline extension module ============

running install
running bdist_egg
running egg_info
creating gnureadline.egg-info
writing gnureadline.egg-info/PKG-INFO
writing dependency_links to gnureadline.egg-info/dependency_links.txt
writing top-level names to gnureadline.egg-info/top_level.txt
writing manifest file 'gnureadline.egg-info/SOURCES.txt'
reading manifest file 'gnureadline.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'gnureadline.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-12-arm64/egg
running install_lib
running build_py
creating build
creating build/lib.macosx-12-arm64-cpython-311
copying readline.py -> build/lib.macosx-12-arm64-cpython-311
warning: build_py: byte-compiling is disabled, skipping.

running build_ext
building 'gnureadline' extension
creating build/temp.macosx-12-arm64-cpython-311
creating build/temp.macosx-12-arm64-cpython-311/Modules
creating build/temp.macosx-12-arm64-cpython-311/Modules/3.x
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -DHAVE_RL_APPEND_HISTORY -DHAVE_RL_CALLBACK -DHAVE_RL_CATCH_SIGNAL -DHAVE_RL_COMPLETION_APPEND_CHARACTER -DHAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK -DHAVE_RL_COMPLETION_MATCHES -DHAVE_RL_COMPLETION_SUPPRESS_APPEND -DHAVE_RL_PRE_INPUT_HOOK -DHAVE_RL_RESIZE_TERMINAL -I. -IModules/3.x -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c Modules/3.x/readline.c -o build/temp.macosx-12-arm64-cpython-311/Modules/3.x/readline.o
Modules/3.x/readline.c:347:19: error: implicit declaration of function 'append_history' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    errno = err = append_history(
                  ^
Modules/3.x/readline.c:1160:5: error: use of undeclared identifier 'rl_completion_suppress_append'
    rl_completion_suppress_append = 0;
    ^
Modules/3.x/readline.c:1317:5: error: use of undeclared identifier 'rl_catch_signals'
    rl_catch_signals = 0;
    ^
Modules/3.x/readline.c:1340:17: error: implicit declaration of function 'rl_resize_terminal' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                rl_resize_terminal();
                ^
Modules/3.x/readline.c:1340:17: note: did you mean 'rl_reset_terminal'?
/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/editline/readline.h:181:8: note: 'rl_reset_terminal' declared here
void             rl_reset_terminal(const char *);
                 ^
4 errors generated.
error: command '/usr/bin/clang' failed with exit code 1

It looks like the first error is that append_history is missing, but I'm not sure where that's supposed to be defined. Could you let me know if this is just a 3.11 incompatibility that has yet to be fixed, or if I'm building it wrong?

[ludwigschwardt]

Thanks for reporting this! I’m also on Homebrew so you have got my attention 🙂 I’ll have some time in the next few days to try a 3.11 version of the readline module.

[milosivanovic]

@ludwigschwardt kindly following up - will it be possible to use this in Python 3.11 and later?

[ludwigschwardt]

Whoops Milos, I lost track of this - sincere apologies!

It's not a Python 3.11 issue per se... I already have 3.11 wheels on PyPI since October 2022 (#59). (Just checking: have you tried pip install gnureadline?)

I'm currently on Homebrew but only for non-Python libraries. I use pyenv to provide Python 3.11. The package also compiles fine on that - but then pyenv has readline...

I think the main compilation issue is that you don't have any readline headers around anymore. Try adding

CFLAGS=-Irl/readline-lib python3.11 setup.py install

to use the local readline headers inside the package.

In the meantime I'll fish out my old laptop that still has brew python and see if there is a better solution.

[milosivanovic]

@ludwigschwardt thanks a lot for responding. My use case is that I just want to have it in the shell so I can use CTRL+R and other readline features on macOS (as would be normal on Linux installs) since brew Python 3.11 removed readline support going forward, meaning Python 3.10 was the last version to be compiled against readline.

I tried pip3 install gnureadline but it didn't seem to have any effect on the shell. In the README I see the following:

If you want to use this module as a drop-in replacement for readline in the standard Python shell, it has to be installed with the less polite easy_install script found in [setuptools](https://pypi.python.org/pypi/setuptools). Please take note that easy_install has been deprecated for a while and is about to be dropped from setuptools. Proceed at your own risk!

I believe that's what I tried to do last time, which failed with the error in my original comment.

I just tried running CFLAGS=-Irl/readline-lib python3.11 setup.py install in the python-gnureadline directory but I got the same set of errors as before:

============ Building the readline extension module ============

running install
running bdist_egg
running egg_info
writing gnureadline.egg-info/PKG-INFO
writing dependency_links to gnureadline.egg-info/dependency_links.txt
writing top-level names to gnureadline.egg-info/top_level.txt
reading manifest file 'gnureadline.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'gnureadline.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-12-arm64/egg
running install_lib
running build_py
creating build
creating build/lib.macosx-12-arm64-cpython-311
copying readline.py -> build/lib.macosx-12-arm64-cpython-311
warning: build_py: byte-compiling is disabled, skipping.

running build_ext
building 'gnureadline' extension
creating build/temp.macosx-12-arm64-cpython-311
creating build/temp.macosx-12-arm64-cpython-311/Modules
creating build/temp.macosx-12-arm64-cpython-311/Modules/3.x
clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -Irl/readline-lib -DHAVE_RL_APPEND_HISTORY -DHAVE_RL_CALLBACK -DHAVE_RL_CATCH_SIGNAL -DHAVE_RL_COMPLETION_APPEND_CHARACTER -DHAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK -DHAVE_RL_COMPLETION_MATCHES -DHAVE_RL_COMPLETION_SUPPRESS_APPEND -DHAVE_RL_PRE_INPUT_HOOK -DHAVE_RL_RESIZE_TERMINAL -I. -IModules/3.x -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c Modules/3.x/readline.c -o build/temp.macosx-12-arm64-cpython-311/Modules/3.x/readline.o
Modules/3.x/readline.c:347:19: error: implicit declaration of function 'append_history' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    errno = err = append_history(
                  ^
Modules/3.x/readline.c:1160:5: error: use of undeclared identifier 'rl_completion_suppress_append'
    rl_completion_suppress_append = 0;
    ^
Modules/3.x/readline.c:1317:5: error: use of undeclared identifier 'rl_catch_signals'
    rl_catch_signals = 0;
    ^
Modules/3.x/readline.c:1340:17: error: implicit declaration of function 'rl_resize_terminal' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                rl_resize_terminal();
                ^
Modules/3.x/readline.c:1340:17: note: did you mean 'rl_reset_terminal'?
/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/editline/readline.h:181:8: note: 'rl_reset_terminal' declared here
void             rl_reset_terminal(const char *);
                 ^
4 errors generated.
error: command '/usr/bin/clang' failed with exit code 1

Appreciate any further help.

[ludwigschwardt]

I tried pip3 install gnureadline but it didn't seem to have any effect on the shell.

Yes, it plays "nicely" with your Python installation and requires you to call it explicitly (unlike readline with its behind-the-scenes magic). This use case was originally developed for IPython, but they have since moved on to prompt_toolkit. I only ever use IPython to get readline features like Ctrl+R, which is why I've missed this issue so far :-)

We used to have a direct readline replacement package (still on PyPI at version 6.2.4) that could be installed via

easy_install readline

to do exactly what you require. This is what I meant by "easy_install" installation. It's different from python setup.py install. You shouldn't even be doing that these days 😅

The problem is that easy_install is not so easy anymore (ahem). It's still available but with dire warnings and discouragement. The command-line script is not installed anymore. You could try your luck with ez_setup to get the easy_install script - also deprecated. I couldn't resurrect it yet on my side.

On your build issue: the include directory flag should have been -I., but I see your clang line already has that... 🤔 Do you have a readline symlink in the same directory as setup.py?

The bigger challenge is to get the gnureadline package to replace the original readline module. Dirty tricks include easy_install, the easy-install.pth file, eggs (as opposed to wheels) and sys.modules['readline'] fiddling.

[ludwigschwardt]

For myself, I use pyenv and IPython for tab completion, for what it's worth.

[ludwigschwardt]

I have a new scheme to try out... 🙂

1. Install gnureadline by pip install gnureadline.
2. Create a config file like ~/.pythonrc:

import sys
try:
    import gnureadline as readline
except ImportError:
    import readline
sys.modules['readline'] = readline
del readline
del sys

3. Add the following to your ~/.profile:

export PYTHONSTARTUP=~/.pythonrc

These commands will run at the start of every interactive Python session, which is when you need tab completion etc.

It also works in virtualenvs, if you remember to install gnureadline again...

Let me know if the resulting keystrokes are what you expect.

[ludwigschwardt]

Mmm, I see now that PYTHONSTARTUP is too late to the party. It happens right at the end of the startup sequence. The second-last step is to import rlcompleter, which pulls in the original readline. Presumably cmd also happens somewhere around there (see #61). See for yourself if you run python3 -v.

There are two earlier opportunities to tweak things: sitecustomize followed by usercustomize. The former is created and used by Homebrew, which makes it harder to fiddle with.

I had good success with usercustomize. Try the following:

USER_SITE=$(python3 -m site --user-site)  # just a temp variable - no need to set in profile
mkdir -p $USER_SITE
cp ~/.pythonrc $USER_SITE/usercustomize.py

This also works in virtualenvs 🙂

[milosivanovic]

@ludwigschwardt Wow, thank you so much! That works perfectly!

Screen.Recording.2023-05-24.at.9.21.19.AM.mov

[keeely]

Whilst I appreciate you looking into this, I'll stick with the sys.modules['readline'] fiddling as it has worked so far and my users don't have to do anything (really important for me).

[milosivanovic]

For anyone else wanting this solution, here's a single command to copy-paste to set up the gnureadline injection:

mkdir -p $(python3 -m site --user-site) && cat << 'EOF' > $(python3 -m site --user-site)/usercustomize.py
import sys
try:
    import gnureadline as readline
except ImportError:
    import readline
sys.modules['readline'] = readline
del readline
del sys
EOF

After that, just install gnureadline if it isn't already: pip3 install gnureadline

Thanks again @ludwigschwardt!

[ludwigschwardt]

I'm very glad it worked for you, Milos! I know PYTHONSTARTUP but never used usercustomize before. I wish I knew about it sooner 😁

The del readline and del sys steps are now superfluous. Previously I was worried about polluting the namespace on startup but it looks like usercustomize cleans up afterwards so we can leave that out.

@keeely, glad you could sort out your users too. Your solution makes sense given the constraints. Do you import cmd in your own app or do your users interact with the usual Python REPL? I'm wondering if the meta_path trick can override rlcompleter too to fix the standard Python shell.

[ludwigschwardt]

Here is an example of a Python script that updates usercustomize.py in a reasonably safe way. I'm considering including such a script with gnureadline but I'm unsure if packages are "allowed" to modify user site-packages like this 🤔 😅