Skip to content

Conversation

@topherinternational
Copy link
Contributor

This change tightens up the Mongo connection UX:

  • The MongoHook constructor validates the connection object to ensure it is of the correct conn_type (mongo), raising an exception otherwise
  • If the user have set the scheme/conn_type to mongodb+srv, the constructor exception provides an informative message telling the user to use the mongo conn_type and set srv=true in the extras
  • The UI has been expanded with special boolean widgets for the supported srv, ssl and allow_insecure extra parameters

This is a replacement for #41371 which aimed to improve the Mongo connection parameters in a different and less-functional way.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@topherinternational
Copy link
Contributor Author

@jscheffl @uranusjr anything left to do before getting this merged?

Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM... @uranusjr any objections?

Copy link
Member

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit, otherwise looks good to me.

@topherinternational
Copy link
Contributor Author

I simplified the failing sensor test which was added by someone else in a different PR. Now it mocks the MongoHook and only tests the interactions between the sensor code and the hook instance inside the sensor, the details of connections etc are not necessary to confirm the sensor is working as designed. (IMO if we do want to test more realistically, we should use mongomock to mock the Mongo endpoint, and have real non-mocked MongoHook and Connection objects, but that's beyond the scope of this PR).

@zeev-finaloop
Copy link

zeev-finaloop commented Sep 24, 2024

Hi, I believe that this PR is now breaking the functionality of connections that are imported from env vars for example. When parsing the connection from a uri, in case of mongo, the scheme == conn_type and it will now fail in validations as schema is either mongodb or mongodb+srv. I guess we need to add a small PR and add additional case in _normalize_conn_type method. I am not in the developers community here, so just raising the issue this PR caused.
This is where a fix is needed:

conn_type = uri_parts.scheme

@jscheffl
Copy link
Contributor

Hi, I believe that this PR is now breaking the functionality of connections that are imported from env vars for example. When parsing the connection from a uri, in case of mongo, the scheme == conn_type and it will now fail in validations as schema is either mongodb or mongodb+srv. I guess we need to add a small PR and add additional case in _normalize_conn_type method. I am not in the developers community here, so just raising the issue this PR caused. This is where a fix is needed:

conn_type = uri_parts.scheme

Can you elaborate some details, provide an example and post a separate issue entry as provider bug report? Maybe linking to this PR.

joaopamaral pushed a commit to joaopamaral/airflow that referenced this pull request Oct 21, 2024
* hook changes

* test changes

* UI

* test and alert

* use AirflowConfigException

* fix new test

* Update airflow/providers/mongo/hooks/mongo.py

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

* tweak sensor test

---------

Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants