Add documentation about cli add connection and AWS connection URI

[aru-trackunit]

relates to: #28766

I believe that this is a bug in the get_conn_uri method but I am not sure whether this is the best approach towards it.
Happy to make only documentation changes so everyone else doesn't face the same problem

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

@aru-trackunit the actual problem in cli command, not in Connection object.

We should fix this function:

airflow/airflow/cli/commands/connection_command.py

Lines 134 to 137 in 45dd0c4

	def _valid_uri(uri: str) -> bool:
	"""Check if a URI is valid, by checking if both scheme and netloc are available."""
	uri_parts = urlsplit(uri)
	return uri_parts.scheme != "" and uri_parts.netloc != ""

[aru-trackunit]

@Taragolis ok, I will revert my changes and update only docs.

As a side note: I am not a python developer and I find contributing pretty difficult. Especially setting up the environment and running tests.
When setting up a local environment I went into issues like missing pygraphviz, then after executing tests using breeze it was missing some packages. Then zshrc script couldn't find compdef. Stackoverflow again to the rescue. Finally when I got them to work then some tests failed even when I haven't done any code changes.

When running breeze testing tests it took my local aws account and tried to execute something which obviously failed

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetParameter operation: User: arn:aws:sts::----:assumed-role/---- is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1: :parameter/example_step_functions because no identity-based policy allows the ssm:GetParameter action

Steep starting learning curve, just to make a few line changes. Probably I know to little about the ecosystem to propose solution, but the problems I have faced discourage me from contributing.

Don't get me wrong, I am trying to not complain just show my experience.

[Taragolis]

@Taragolis ok, I will revert my changes and update only docs.

Contribution to documentation it still important part, personally I think that documentation part more important rather than code.

As a side note: I am not a python developer and I find contributing pretty difficult. Especially setting up the environment and running tests.

No problem if you are not feel comfortable someone could pick up part for solve actual issue. I have a plan to work on it in the next couple days if someone not pick it up.

The documentation part is still important. Just one suggestion it would be nice also add info about issue with adding connection by URI in cli into the Amazon Web Services Connection page.

We mentioned here that user could use aws:// in case of default boto3 credentials strategy (env variables, EC2/ECS metadata and etc). And this connection URI is valid in all places except cli.

❯ airflow connections add sample-aws-conn --conn-uri "aws://"
The URI provided to --conn-uri is invalid: aws://

I could say it is not a common way to add connection by cli as URI because nobody mentioned it before and users could add it by parts or since airflow 2.3 as json (doc) however it nice that we know that we have this bug.

When running breeze testing tests it took my local aws account and tried to execute something which obviously failed

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetParameter operation: User: arn:aws:sts::----:assumed-role/---- is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1: :parameter/example_step_functions because no identity-based policy allows the ssm:GetParameter action

Interesting, I thought we solve issue when amazon-provider tests obtain actual credentials. Could you provide a bit more info here?

1. How you store your credentials? Environment variables, profile and etc.
2. Where you try to run this test? Laptop / PC or in EC2?
3. Is failed tests stored into test/providers/amazon ?

[aru-trackunit]

Sure!

1. I am storing them in .aws/config file as profiles with default profile enabled.
2. Mac M1
3. Yes:

ERROR tests/system/providers/amazon/aws/example_athena.py - botocore.exceptio...
ERROR tests/system/providers/amazon/aws/example_batch.py - botocore.exception...
ERROR tests/system/providers/amazon/aws/example_cloudformation.py - botocore....
ERROR tests/system/providers/amazon/aws/example_datasync.py - botocore.except...
ERROR tests/system/providers/amazon/aws/example_dms.py - botocore.exceptions....

I am open to give you more input if needed

[Taragolis]

Sure!

1. I am storing them in .aws/config file as profiles with default profile enabled.
2. Mac M1
3. Yes:

ERROR tests/system/providers/amazon/aws/example_athena.py - botocore.exceptio...
ERROR tests/system/providers/amazon/aws/example_batch.py - botocore.exception...
ERROR tests/system/providers/amazon/aws/example_cloudformation.py - botocore....
ERROR tests/system/providers/amazon/aws/example_datasync.py - botocore.except...
ERROR tests/system/providers/amazon/aws/example_dms.py - botocore.exceptions....

I am open to give you more input if needed

@ferruzzi @vincbeck @o-nikolas guys, I'm not familiar with system tests is it expected that system tests tried to obtain credentials?

[vincbeck]

Yes. System tests which need external values (example here) try to fetch these values in SSM parameter store unless they are defined as environment variable. See logic here. It seems like we are already handling the case when no credentials is provided (here). We might want to update this logic to handle the case when the credentials do not have enough permissions to read parameters from SSM

[aru-trackunit]

@Taragolis I've seen you added another PR. Is the docs that I wrote still relevant or this PR should be closed?

[uranusjr]

If I’m reading the linked issue correctly, this is fixed in main, right? In that case we should only add this to older versions. Or is there something I misunderstand?

[Taragolis]

@aru-trackunit Yeah this PR still relevant, because #28922 in main and potentially would include in 2.5.2 (or later), however latest version of Amazon Provider has requirement AIrflow 2.3+.

So would be nice to have this info for users who will use old version of Airflow

[boring-cyborg]

Awesome work, congrats on your first merged pull request!