Readline issues on new install of FreeBSD 13.1

[dcontiveros]

Too many issues will kill our team's development velocity, drastically.
Make sure you have checked all steps below.

Prerequisite

• Make sure your problem is not listed in the common build problems.
• Make sure no duplicated issue has already been reported in the pyenv issues. You should look for closed issues, too.
• Make sure you are not asking us to help solving your specific issue.
  • GitHub issues is opened mainly for development purposes. If you want to ask someone to help solving your problem, go to some community site like Gitter, StackOverflow, etc.
• Make sure your problem is not derived from packaging (e.g. Homebrew).
  • Please refer to the package documentation for the installation issues, etc.
• Make sure your problem is not derived from plugins.
  • This repository is maintaining pyenv and the default python-build plugin only. Please refrain from reporting issues of other plugins here.

Description

• Platform information (e.g. Ubuntu Linux 16.04): FreeBSD 13.1-RELEASE-p3
• OS architecture (e.g. amd64): amd64
• pyenv version: pyenv 2.3.9
• Python version: Python 3.9.16 (installed via pkg)
• C Compiler information (e.g. gcc 7.3): FreeBSD clang version 13.0.0
• Please attach the debug trace of the failing command as a gist:
  • Run env PYENV_DEBUG=1 <faulty command> 2>&1 | tee trace.log and attach trace.log. E.g. if you have a problem with installing Python, run env PYENV_DEBUG=1 pyenv install -v <version> 2>&1 | tee trace.log (note the -v option to pyenv install).

---

Recently provisioned a remote VPS on FreeBSD 13.1 and wanted to give pyenv a whirl. Seems I'm hit with readline issues. I attempted to install readline libs, but what I believe to be happening is that pyenv make cannot resolve readline. Here is what I performed:

1. installed pyenv via bash script
2. Ran:

pyenv install 3.11.1

Errors reported:

pyenv: /home/blogger/.pyenv/versions/3.11.1 already exists
continue with installation? (y/N) y
Downloading Python-3.11.1.tar.xz...
-> https://www.python.org/ftp/python/3.11.1/Python-3.11.1.tar.xz
Installing Python-3.11.1...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.11.1 to /home/blogger/.pyenv/versions/3.11.1

Packages installed:

# pkg info | grep -i read
readline-8.2.0                 Library for editing command lines as they are typed
tclreadline-2.3.8              GNU readline for TCL

What I believe is happening is that GNU readline requirement is not being read properly as the output shows:

checking how to link readline libs... none

Debug log attached.

trace.log

[native-api]

Could you please also attach config.log from the build directory?

Do you have Readline development headers and lib files installed? Through what means? (I dunno if BSD have package managers. I know they have "ports" which are kinda that and may need additional compiler flags to link against.)

[dcontiveros]

The config.log file does not appear to be created:

[~/.pyenv] $ find . | grep config.log (returns no results)

I made sure to install readline libs using pkg command as stated in my first update. Just to be sure, I logged on and verified:

# pkg info readline
readline-8.2.0
Name           : readline
Version        : 8.2.0
Installed on   : Fri Jan  6 12:31:28 2023 EST
Origin         : devel/readline
Architecture   : FreeBSD:13:amd64
Prefix         : /usr/local
Categories     : devel
Licenses       : GPLv3
Maintainer     : sunpoet@FreeBSD.org
WWW            : https://tiswww.case.edu/php/chet/readline/rltop.html
Comment        : Library for editing command lines as they are typed
Options        :
        BRACKETEDPASTE : on
        DOCS           : on
Shared Libs provided:
        libreadline.so.8
        libhistory.so.8
Annotations    :
        FreeBSD_version: 1301000
        cpe            : cpe:2.3:a:gnu:readline:8.2:::::freebsd13:x64
        repo_type      : binary
        repository     : FreeBSD
Flat size      : 1.78MiB
Description    :
The GNU Readline library provides a set of functions for use by applications
that allow users to edit command lines as they are typed in. Both Emacs and vi
editing modes are available. The Readline library includes additional functions
to maintain a list of previously-entered command lines, to recall and perhaps
reedit those lines, and perform csh-like history expansion on previous commands.

WWW: https://tiswww.case.edu/php/chet/readline/rltop.html

# find /usr/local/ | grep readline
/usr/local/share/licenses/readline-8.2.0
/usr/local/share/licenses/readline-8.2.0/catalog.mk
/usr/local/share/licenses/readline-8.2.0/LICENSE
/usr/local/share/licenses/readline-8.2.0/GPLv3
/usr/local/share/licenses/tclreadline-2.3.8
/usr/local/share/licenses/tclreadline-2.3.8/catalog.mk
/usr/local/share/licenses/tclreadline-2.3.8/LICENSE
/usr/local/share/licenses/tclreadline-2.3.8/BSD3CLAUSE
/usr/local/share/doc/readline
/usr/local/share/doc/readline/CHANGES
/usr/local/share/doc/readline/INSTALL
/usr/local/share/doc/readline/README
/usr/local/share/readline
/usr/local/share/readline/excallback.c
/usr/local/share/readline/fileman.c
/usr/local/share/readline/histexamp.c
/usr/local/share/readline/manexamp.c
/usr/local/share/readline/rl-fgets.c
/usr/local/share/readline/rl.c
/usr/local/share/readline/rlbasic.c
/usr/local/share/readline/rlcat.c
/usr/local/share/readline/rlevent.c
/usr/local/share/readline/rlptytest.c
/usr/local/share/readline/rltest.c
/usr/local/share/readline/rlversion.c
/usr/local/share/info/readline.info
/usr/local/share/vim/vim90/indent/readline.vim
/usr/local/share/vim/vim90/ftplugin/readline.vim
/usr/local/share/vim/vim90/syntax/readline.vim
/usr/local/include/readline
/usr/local/include/readline/chardefs.h
/usr/local/include/readline/history.h
/usr/local/include/readline/keymaps.h
/usr/local/include/readline/readline.h
/usr/local/include/readline/rlconf.h
/usr/local/include/readline/rlstdc.h
/usr/local/include/readline/rltypedefs.h
/usr/local/include/readline/tilde.h
/usr/local/include/editline/readline.h
/usr/local/include/tclreadline.h
/usr/local/lib/libreadline.a
/usr/local/lib/libreadline.so
/usr/local/lib/libreadline.so.8
/usr/local/lib/libreadline.so.8.2
/usr/local/lib/python3.9/lib-dynload/readline.cpython-39.so
/usr/local/lib/python3.9/lib2to3/fixes/__pycache__/fix_xreadlines.cpython-39.opt-1.pyc
/usr/local/lib/python3.9/lib2to3/fixes/__pycache__/fix_xreadlines.cpython-39.opt-2.pyc
/usr/local/lib/python3.9/lib2to3/fixes/__pycache__/fix_xreadlines.cpython-39.pyc
/usr/local/lib/python3.9/lib2to3/fixes/fix_xreadlines.py
/usr/local/lib/python3.9/test/__pycache__/test_readline.cpython-39.opt-1.pyc
/usr/local/lib/python3.9/test/__pycache__/test_readline.cpython-39.opt-2.pyc
/usr/local/lib/python3.9/test/__pycache__/test_readline.cpython-39.pyc
/usr/local/lib/python3.9/test/test_readline.py
/usr/local/lib/libtclreadline.so
/usr/local/lib/libtclreadline-2.3.8.so
/usr/local/lib/tclreadline2.3.8
/usr/local/lib/tclreadline2.3.8/pkgIndex.tcl
/usr/local/lib/tclreadline2.3.8/tclreadlineCompleter.tcl
/usr/local/lib/tclreadline2.3.8/tclreadlineInit.tcl
/usr/local/lib/tclreadline2.3.8/tclreadlineSetup.tcl
/usr/local/libdata/pkgconfig/readline.pc
/usr/local/man/man3/readline.3.gz
/usr/local/man/man3/libssh2_knownhost_readline.3.gz
/usr/local/man/mann/tclreadline.n.gz

[native-api]

config.log is in the build directory under /tmp. In the log, it's /tmp/python-build.20230106142056.35755.

[dcontiveros]

Log attached.
python-build.20230107112237.54254.log

[native-api]

I said config.log, not the build log.
It should be under the build directory, e.g. /tmp/python-build.20230106142056.35755/config.log.

[dcontiveros]

I removed all files from /tmp and this is the only log that is being generated.

[native-api]

Oh, that's right. Your build succeeds, so the build path is being removed.

Please run pyenv install with -k to keep it.

[dcontiveros]

It appears config.log was generated in another location with the -k switch:

[~] $ find . | grep config.log
./.pyenv/sources/3.11.1/Python-3.11.1/config.log

Please see attached. I am seeing linker errors, so am assuming would need to append something to either LDFLAGS or CFLAGS. Let me know.

config.log

[native-api]

Okay, it looks like unlike Linux, in BSD, the compiler is configured to not look in /usr/local/ by default:

configure:20925: checking how to link readline libs
configure:20952: cc -pthread -o conftest  -I/home/blogger/.pyenv/versions/3.11.1/include  -L/home/blogger/.pyenv/versions/3.11.1/lib -Wl,-rpath=/home/blogger/.pyenv/versions/3.11.1/lib  conftest.c -lreadline -ldl  -lutil >&5
ld: error: unable to find library -lreadline
cc: error: linker command failed with exit code 1 (use -v to see invocation)
configure:20952: $? = 1
configure: failed program was:
<...>
| char readline ();
| int
| main ()
| {
| return readline ();
|   ;
|   return 0;
| }

[native-api]

Could you test the codebase from https://github.com/native-api/pyenv/tree/use_freebsd_ports ?

[dcontiveros]

Using your code tree the latest python available is 3.9.7. Is there anyway I can test 3.11.1?

[native-api]

Use the use_freebsd_ports branch

[dcontiveros]

Same error. Looks like the same output:

I rebuilt to regenerate relevant logs. Please see attached.

[~/.pyenv] $ git branch
  master
* use_freebsd_ports

$ pyenv install 3.11.1 -k
Installing Python-3.11.1...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.11.1 to /home/blogger/.pyenv/versions/3.11.1

config.log

[native-api]

Okay, I was able to set up a FreeBSD VM and reproduce the issue.

It turns out, you don't have pkg-config installed.

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

Which, counterintuitively, is in the pkgconf Ports package.

All ports modules have .pc files that allow to locate them using pkg-config (which Python's Configure uses as well).

---

We can still add compiler flags for the case when readline package is installed and pkg-config is missing.
I've checked that either way works.