Make pandas an optional dependency for amazon provider

[manugarri]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

No response

Apache Airflow version

latest

Operating System

any

Deployment

Other

Deployment details

No response

What happened

First of all, apologies if this is not the right section to post a GH issue. I looked for provider specific feature requests but couldnt find such section.

We use the aws provider at my company to interact from airflow with AWS services. We are using poetry for building the testing environment to test our dags.

However the build times are quite long, and the reason is building pandas, which is a dependency of the amazon provider.

By checking the provider's code, it seems pandas is used in a small minority of functions inside the provider:

./aws/transfers/hive_to_dynamodb.py:93:        data = hive.get_pandas_df(self.sql, schema=self.schema)

and

./aws/transfers/sql_to_s3.py:159:        data_df = sql_hook.get_pandas_df(sql=self.query, parameters=self.parameters)

Forcing every AWS Airflow user that do not use hive or want to turn sql into an s3 file to install pandas is a bit cumbersome.

What you think should happen instead

given how heavy the package is and how little is used in the amazon provider, pandas should be an optional dependency.

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Taragolis]

@manugarri Do you use Python 3.7 or Alpine Linux?

According to https://pypi.org/project/pandas/#files

• No pre-build wheels for Python 3.7 (EOL is 27 Jun 2023)
• No pre-build wheels for musllinux

[manugarri]

@Taragolis we use AWS MWAA, so we are stuck with python 3.7 unfortunately :/

[Taragolis]

MWAA right now support Airflow 2.2.2 however even if someone add this changes, which to be honest not a straightforward and required introduce breaking chages, it wouldn't be applicable to Airflow<2.3: https://github.com/apache/airflow/blob/main/README.md#release-process-for-providers

However I see at Slack that AWS might add support of new version Airflow into MWAA soon

[eladkal]

This is probably leftovers from where pandas was a core dependency.

@o-nikolas @ferruzzi @vincbeck probbaly something worth looking into.

[potiuk]

Yeah. Pandas is quite a "drag" as a dependency- regardles if it is a binary wheel or not so making it optional for Amazon provider would be a good idea.

[potiuk]

I can even make a draft PR for that - we have a few prior-art cases (plyvel in google provider was pretty similar and we made it optional, we even have the AirflowOptionalProviderFeatureException foreseen for the case where people would use an exssting code in the provider and the needed dependency is not installed. It's actually pretty easy to do.

[vincbeck]

I dont have much context on this one so if creating a PR for it is easy for you @potiuk, be my guest :) Happy to review it if needed!

[Taragolis]

Is it might be better make in optional for all providers where it listed as core dependency?

• amazon
• apache.hive
• exasol
• google
• presto
• salesforce
• trino