Skip to content

Conversation

@brki
Copy link
Contributor

@brki brki commented Aug 14, 2022

The non-binary version of psycopg2 is now used, and the version requirement changed to psycopg2>=2.8.0.
The non-binary version is used because the binary version of psycopg2 is not recommended for production
use. The minimum version of psycopg2 was changed because psycopg2 < 2.8.0 includes a binary distribution
which will be installed by default.
If not already present, you will need to add the postgresql development libraries. On debian-based systems,
this is provided by the package libpq-dev.

Closes: #25712


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg
Copy link

boring-cyborg bot commented Aug 14, 2022

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@brki brki force-pushed the non-binary-psycopg2 branch 2 times, most recently from 362d42f to d3ebcc7 Compare August 14, 2022 14:53
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this file is generated automatically so you shouldn't edit it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RosterIn is right. see note just abiive:

Please, only add notes to the Changelog just below the "Changelog" header when there are some breaking changes
and you want to add an explanation to the users on how they are supposed to deal with them.
The changelog is updated and maintained semi-automatically by release manager.

Also I think the change is not breaking. Dependency change is not a breaking change, bumping min version is not breaking either, and the behaviour is not going to change either

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @RosterIn , I've removed the changes to index.rst.

Copy link
Contributor Author

@brki brki Aug 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@potiuk I considered it a breaking change because it will break for some people, because it introduces new operating system dependencies, that were not necessarily present before. For example on Debian-based distros, libpq-dev is necessary. This is the case for the slim docker images, for example airflow:slim-2.3.3-python3.10 .

Copy link
Member

@potiuk potiuk Aug 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not what we consider as breaking. Dependency changes are not introducing. Our SemVer rules state (see readme)

Airflow: SemVer rules apply to core airflow only (excludes any changes to providers). Changing limits for versions of Airflow dependencies is not a breaking change on its own.

And while this is not changing the version but other 'variant' of the same dependency, this only affect how you install airflow. And it is not breaking the 'use' of Airflow which is the important factor for breaking/non breaking decision.

The main thing here that changes that are only affecting the way how you install Airfloe are not really 'breaking' any user code or ways how you use airflow. If for any reason you cannot install Airflow new version, you just cannot install it and you don't break anything. Once you installed it, it works without changes. So it does not break anything.

If you follow the logic of 'dependencies are breaking changes' then it basically would mean than any release where any dependency change is breaking - because it might cause troubles when installing Airflow or providers.

@brki
Copy link
Contributor Author

brki commented Aug 20, 2022 via email

@brki brki force-pushed the non-binary-psycopg2 branch from 60bafca to 868a8b0 Compare August 21, 2022 08:13
@potiuk
Copy link
Member

potiuk commented Aug 23, 2022

Cool!

@potiuk potiuk merged commit 14d56a5 into apache:main Aug 23, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Aug 23, 2022

Awesome work, congrats on your first merged pull request!

potiuk added a commit to potiuk/airflow that referenced this pull request Dec 12, 2022
The psycopg dependency is slightly more optimized than than the
binary one, but the problems it creates when it is used as
dependency are numerous - it always expects to compile the
psycopg dependencies, so you cannot install it when there are no
build-essentials installed and on some systems additional libraries
are needed.

We introduced it in apache#25710 for optimisation reasons.

However, the optimisations it brings are not really justified with the
installation complexities it introduces, that's why we revert it.
potiuk added a commit that referenced this pull request Dec 12, 2022
The psycopg dependency is slightly more optimized than than the
binary one, but the problems it creates when it is used as
dependency are numerous - it always expects to compile the
psycopg dependencies, so you cannot install it when there are no
build-essentials installed and on some systems additional libraries
are needed.

We introduced it in #25710 for optimisation reasons.

However, the optimisations it brings are not really justified with the
installation complexities it introduces, that's why we revert it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

postgres provider: use non-binary psycopg2 (recommended for production use)

4 participants