Make pandas an optional core dependency

[kaxil]

We only use pandas in DbApiHook.get_pandas_df. Not all users use it, plus
while pandas now supports many pre-compiled packages it still can take forever where
it needs to be compiled.

So for first-time users this can be a turn off. If pandas is already installed this
will work fine, but if not users have an option to run pip install apache-airflow[pandas]

Related #12500

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Huge yes.

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

Just realized -we also need to add it to set of extras in production Dockerfile (to keep it behave the way it did).

airflow/Dockerfile

Line 37 in 9875757

ARG AIRFLOW_EXTRAS="async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,google_auth,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv"

[potiuk]

And the Dockerfile will continue to work for old airflow versions as well. AFAIK when you specifiy extra that is missing, it will be silently ignored, so it will work for pre 2.2 Airflow versions as well.

[ashb]

I think this closes #12500 - are there any other uses of numpy in the code base?

[potiuk]

I think this closes #12500 - are there any other uses of numpy in the code base?

Not yet. It is used at least in AirflowJsonEncoder 😱

[uranusjr]

Suggested change

	# Pandas stopped releasing 3.6 binaries for 1.2.* series.
	'pandas>=0.17.1, <1.2;python_version<"3.7"',
	'pandas>=0.17.1, <2.0;python_version>="3.7"',
	'pandas>=0.17.1, <2.0',

Pandas 1.2+ provides the Requires-Python metadata, so 1.2+ versions will be automatically excluded when pip install on Python 3.6:

$ python3.6 -m venv x
$ . x/bin/activate
(x) $ pip install pandas
Collecting pandas
  Using cached https://.../pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl
Collecting python-dateutil>=2.7.3 (from pandas)
...

So the python_version<"3.7" line is not needed. (Although it should not have a negative impact either.)

Note: Python 3.6.0 ships with pip 9.1 by default, which supports Requires-Python, so it’s highly unlikely someone on 3.6 running on pip version not supporting this metadata.

[kaxil]

Good point, updated

[kaxil]

398ebfe should make numpy optional as well

[edwardwang888]

@kaxil Should this import also be put inside a try block, if numpy is not a dependency required in the devel extra?

airflow/tests/utils/test_json.py

Line 24 in 11e3453

import numpy as np

[kaxil]

@kaxil Should this import also be put inside a try block, if numpy is not a dependency required in the devel extra?

airflow/tests/utils/test_json.py

Line 24 in 11e3453

import numpy as np

We install that in Dockerfile so we have it while running tests

[potiuk]

We install that in Dockerfile so we have it while running tests

Agree. The "test" environment has all the deps in - including pandas and numpy. The "test" dependency ([devel_all] extra) defines all things needed to run tests.

I think while numpy is not needed as "airflow" dependency, it's perfectly fine to have it as "test airflow" dependency.

[edwardwang888]

We install that in Dockerfile so we have it while running tests

Agree. The "test" environment has all the deps in - including pandas and numpy. The "test" dependency ([devel_all] extra) defines all things needed to run tests.

I think while numpy is not needed as "airflow" dependency, it's perfectly fine to have it as "test airflow" dependency.

@kaxil @potiuk Thanks for the feedback. The reason I ask is because of the discussion here: #18343 (comment). For the test environment there is no issue as you've mentioned, so I guess the concern there is about the local development environment? Of course, if you run a test locally and numpy is missing, you can just install it.

[potiuk]

I guess the concern there is about the local development environment? Of course, if you run a test locally and numpy is missing, you can just install it.

We have PLENTY of tests that will fail when some extra is not installed. That's why we have https://github.com/apache/airflow/blob/main/BREEZE.rst with all deps installed in the docker image (which is 1-1 replica of CI test environment), and https://github.com/apache/airflow/blob/main/LOCAL_VIRTUALENV.rst#creating-a-local-virtualenv where we explain that you need to add "devel" extra if you want to install local virtualenv for development.

So I think all is good here

[edwardwang888]

We have PLENTY of tests that will fail when some extra is not installed. That's why we have https://github.com/apache/airflow/blob/main/BREEZE.rst with all deps installed in the docker image (which is 1-1 replica of CI test environment), and https://github.com/apache/airflow/blob/main/LOCAL_VIRTUALENV.rst#creating-a-local-virtualenv where we explain that you need to add "devel" extra if you want to install local virtualenv for development.

So I think all is good here

@potiuk Thanks. One clarification - from #18343 (comment), I got the impression that devel extra contains the libraries needed for core testing (in local virtualenv). Does test_json.py count as core testing? I think that's where I'm confused about why numpy only needs to be installed in devel_all, but not devel.

[potiuk]

Does test_json.py count as core testing?

Depends what you mean by "core". There is no mentioning of "core" in the extras doc:

From https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html#bundle-extras

extra	install command	enables
all	pip install 'apache-airflow[all]'	All Airflow user facing features (no devel and doc requirements)
all_dbs	pip install 'apache-airflow[all_dbs]'	All databases integrations
devel	pip install 'apache-airflow[devel]'	Minimum dev tools requirements (without providers)
devel_hadoop	pip install 'apache-airflow[devel_hadoop]'	Same as devel + dependencies for developing the Hadoop stack
devel_all	pip install 'apache-airflow[devel_all]'	Everything needed for development (devel_hadoop + providers)
devel_ci	pip install 'apache-airflow[devel_ci]'	All dependencies required for CI build.

It's clear that devel is for all tests except providers. the test.json is not in providers. Do you think it's ambiguous? maybe you would like to improve the docs if things are not clear enough ? We have more than 1700 contibutors and clarifications from users like you who have doubts, are most welcome. Do you think this can be improved in the docs? How?

[edwardwang888]

Depends what you mean by "core". There is no mentioning of "core" in the extras doc:

Ah okay, that is a good point.

It's clear that devel is for all tests except providers. the test.json is not in providers. Do you think it's ambiguous? maybe you would like to improve the docs if things are not clear enough ? We have more than 1700 contibutors and clarifications from users like you who have doubts, are most welcome. Do you think this can be improved in the docs? How?

Sounds good. If I come up with something concrete, I'll submit a PR.

Thanks for clarifying my questions!