Cannot create connection without defining host using CLI

[aru-trackunit]

Apache Airflow version

2.5.0

What happened

In order to send logs to s3 bucket after finishing the task, I added a connection to airflow using cli.

airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com

Then I got a logging warning saying:
[2023-01-06T13:28:39.585+0000] {logging_mixin.py:137} WARNING - :8 DeprecationWarning: Host s3 specified in the connection is not used. Please, set it on extra['endpoint_url'] instead

Instead I was trying to remove the host from the conn-uri I provided but every attempt to create a connection failed (list of my attempts below):

airflow connections add connection_id_1 --conn-uri aws://?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com
airflow connections add connection_id_1 --conn-uri aws:///?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com

What you think should happen instead

I believe there are 2 options:

1. Allow to create connection without defining host
   or
2. Remove the warning log

How to reproduce

Create an S3 connection using CLI:

airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com

Operating System

Linux - official airflow image from docker hub apache/airflow:slim-2.5.0

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes | 5.0.0 | Kubernetes
apache-airflow-providers-common-sql | 1.3.1 | Common SQL Provider
apache-airflow-providers-databricks | 4.0.0 | Databricks
apache-airflow-providers-ftp | 3.2.0 | File Transfer Protocol (FTP)
apache-airflow-providers-hashicorp | 3.2.0 | Hashicorp including Hashicorp Vault
apache-airflow-providers-http | 4.1.0 | Hypertext Transfer Protocol (HTTP)
apache-airflow-providers-imap | 3.1.0 | Internet Message Access Protocol (IMAP)
apache-airflow-providers-postgres | 5.3.1 | PostgreSQL
apache-airflow-providers-sqlite | 3.3.1 | SQLite

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

This log message is printed every second minute so it is pretty annoying.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Taragolis]

How to reproduce
Create an S3 connection using CLI:

airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com

This is incorrect connection URL, please use snippet from documentation for generate correct one, you also could check AWS Connection reference for available values.

[aru-trackunit]

Thank you for reply @Taragolis . I followed the documentation you sent and generated the conn_uri with get_uri method

conn=Connection(conn_id="test-s3",conn_type="aws", extra={"endpoint_url":"https://s3.eu-west-1.amazonaws.com", "region_name"="eu-west-1"})

The outcome of conn.get_uri() method was
’aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1'

After following your documentation here https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/logging/s3-task-handler.html
it took me to the connection I wrote above and when I try to execute
airflow connections add connection_id_1 --conn-uri aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1 then it’s not accepted by airflow cli method.

However when I add s3 (or whatever string to the uri that corresponds host) then it becomes parseable, however during runtime I get warnings.

The code below works but gives warnings as described above.

airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com

However I am able to add this connection described above via UI

The behaviour seems inconsistent between two ways of inserting connections.

Maybe I missed something in your reply, but can you elaborate more?

[Taragolis]

@aru-trackunit Ahhh... I see. I've just check and able to reproduce your problem.

URI aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1 is valid for Connection.

Python 3.9.10 (main, Feb 25 2022, 16:54:01) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 8.7.0
Python 3.9.10 (main, Feb 25 2022, 16:54:01) 
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
In[2]: from airflow.models.connection import Connection
In[3]: conn = Connection(uri="aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1")
In[4]: conn.host
Out[4]: ''
In[5]: conn.extra_dejson
Out[5]: 
{'endpoint_url': 'https://s3.eu-west-1.amazonaws.com',
 'region_name': 'eu-west-1'}

But unfortunetly it is not valid for cli. So there is actually exists inconsistency between cli airflow connections add and airflow.models.connection.Connection. Connection URI created by Connection could be invalid for cli in case if all fields host, login, password, port are empty.

airflow/airflow/cli/commands/connection_command.py

Lines 133 to 138 in 71306b3

	def _valid_uri(uri: str) -> bool:
	"""Check if a URI is valid, by checking if both scheme and netloc are available."""
	uri_parts = urlsplit(uri)
	return uri_parts.scheme != "" and uri_parts.netloc != ""

There is no problem with Amazon Provider as well as Deprecation Warning, however problem exists in core.

As workaround you could add @ instead of empty host for now

❯ airflow connections add connection_id_1 --conn-uri "aws://@/?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1"

Successfully added `conn_id`=connection_id_1 : aws://@/?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1

❯ airflow connections get connection_id_1
id | conn_id         | conn_type | description | host | schema | login | password | port | is_encrypted | is_extra_encrypted | extra_dejson             | get_uri                   
===+=================+===========+=============+======+========+=======+==========+======+==============+====================+==========================+===========================
60 | connection_id_1 | aws       | None        |      |        |       | None     | None | False        | False              | {'endpoint_url':         | aws:///?endpoint_url=https
   |                 |           |             |      |        |       |          |      |              |                    | 'https://s3.eu-west-1.am | %3A%2F%2Fs3.eu-west-1.amaz
   |                 |           |             |      |        |       |          |      |              |                    | azonaws.com',            | onaws.com&region_name=eu-w
   |                 |           |             |      |        |       |          |      |              |                    | 'region_name':           | est-1                     
   |                 |           |             |      |        |       |          |      |              |                    | 'eu-west-1'}             |                           

Reopen Issue

[aru-trackunit]

Thank you! Workaround really works, even if you assume that this ticket has very little impact and low priority it's worth mentioning the workaround in the docs or even in the code :)

[potiuk]

Thank you! Workaround really works, even if you assume that this ticket has very little impact and low priority it's worth mentioning the workaround in the docs or even in the code :)

Feel free to add a PR @aru-trackunit . This is an easy way to become a contributor - now that you know how it works, you are likely the best person in the world to know where to add it (and where people like you would look for it). Changing in code - is easy, documentation even easier - just click "Suggest a change on this page" and PR will be opened where you will be able to add the docs - without leaving GitHub UI - so adding a DOC PR is as easy as adding a comment here.

This is how Airflow contributions work - similarly to > 2300 contributors - most of which like you who added a doc they missed or change they felt were missing you can become and Airflow contributor.

Can we count on this please?

[aru-trackunit]

If this is preferred way of solving this issue then sure, let me do it

[euq8]

@aru-trackunit for apache-airflow[amazon]==2.6.0 you can add airflow connections for aws s3 using CLI like -

airflow connections add "s3_connection" \
    --conn-type aws \
    --conn-login "${AWS_ACCESS_KEY_ID}" \
    --conn-password "${AWS_SECRET_ACCESS_KEY}" \
    --conn-extra "{\"region_name\": \"${AWS_REGION}\", \"bucket_name\": \"${AWS_S3_BUCKET_NAME}\"}"