[BUG][2.4.0rc1] fresh install - Error: pg_config executable not found

[raphaelauv]

Apache Airflow version

main (development)

What happened

FROM python:3.10-slim-buster
ENV PIP_NO_CACHE_DIR=TRUE
ENV PIP_DISABLE_PIP_VERSION_CHECK=TRUE
RUN python3 -m pip install apache-airflow[postgres]==2.4.0rc1 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.4.0rc1/constraints-3.10.txt"

fail with

#0 27.60   Downloading psycopg2-2.9.3.tar.gz (380 kB)
#0 27.72      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 380.6/380.6 kB 3.4 MB/s eta 0:00:00
#0 27.80   Preparing metadata (setup.py): started
#0 27.99   Preparing metadata (setup.py): finished with status 'error'
#0 28.00   error: subprocess-exited-with-error
#0 28.00   
#0 28.00   × python setup.py egg_info did not run successfully.
#0 28.00   │ exit code: 1
#0 28.00   ╰─> [25 lines of output]
#0 28.00       /usr/local/lib/python3.10/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
#0 28.00         warnings.warn(msg, warning_class)
#0 28.00       running egg_info
#0 28.00       creating /tmp/pip-pip-egg-info-x_w7xpgd/psycopg2.egg-info
#0 28.00       writing /tmp/pip-pip-egg-info-x_w7xpgd/psycopg2.egg-info/PKG-INFO
#0 28.00       writing dependency_links to /tmp/pip-pip-egg-info-x_w7xpgd/psycopg2.egg-info/dependency_links.txt
#0 28.00       writing top-level names to /tmp/pip-pip-egg-info-x_w7xpgd/psycopg2.egg-info/top_level.txt
#0 28.00       writing manifest file '/tmp/pip-pip-egg-info-x_w7xpgd/psycopg2.egg-info/SOURCES.txt'
#0 28.00       
#0 28.00       Error: pg_config executable not found.
#0 28.00       
#0 28.00       pg_config is required to build psycopg2 from source.  Please add the directory
#0 28.00       containing pg_config to the $PATH or specify the full executable path with the
#0 28.00       option:
#0 28.00       
#0 28.00           python setup.py build_ext --pg-config /path/to/pg_config build ...
#0 28.00       
#0 28.00       or with the pg_config option in 'setup.cfg'.
#0 28.00       
#0 28.00       If you prefer to avoid building psycopg2 from source, please install the PyPI
#0 28.00       'psycopg2-binary' package instead.
#0 28.00       
#0 28.00       For further information please check the 'doc/src/install.rst' file (also at
#0 28.00       <https://www.psycopg.org/docs/install.html>).
#0 28.00       
#0 28.00       [end of output]
#0 28.00   
#0 28.00   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 28.00 error: metadata-generation-failed
#0 28.00 
#0 28.00 × Encountered error while generating package metadata.
#0 28.00 ╰─> See above for output.
#0 28.00 
#0 28.00 note: This is an issue with the package mentioned above, not pip.
#0 28.00 hint: See above for details.
------
failed to solve: executor failed running [/bin/sh -c python3 -m pip install apache-airflow[postgres]==2.4.0rc1 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.4.0rc1/constraints-3.10.txt"]: exit code: 1

it was not the case with 2.4.0b1

What you think should happen instead

No response

How to reproduce

No response

Operating System

ubuntu 22.04.1

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[uranusjr]

This is due to your environment not containing required Postgres libraries. Not sure why 2.4.0b1 was successful (maybe different constraints?) but this is not an Airflow issue IMO.

On Linux you can fix this with https://stackoverflow.com/questions/11618898/pg-config-executable-not-found

[potiuk]

I also suggest to use the official Airflow image to start from. The "slim" version is pretty small and you can add any providers you want, it also contains all the necesary libraries to connect to postgres. And you do not have to installl airflow or worry about constraints - because it is already done. https://airflow.apache.org/docs/docker-stack/index.html

[Taragolis]

@raphaelauv I've got the same issue when prepared new image based on slim version with newest version of apache-airflow-providers-postgres==5.2.1.

There is some changes in requirements for 5.2.1 psycopg2-binary replaced by psycopg2. In my case it fixed by add additional debian packages libpq-dev and build-essential before install providers.

[potiuk]

This is correct. The postgres provider dependenceis are now following the recommendation of Postgres when it comes to postgres dependency (see discussion here: #25710) - this is not considered as a breaking change, because postgres provider is optional and you can install those dependencies additionally as needeed (reasoning explained in the issue).

I also recommmend going the "custom image route" instead of using slim image - you will make the image much smaller as you do not need build-essentials - https://airflow.apache.org/docs/docker-stack/build.html#customizing-the-image