Allow check_response to be defined in extra_options of HTTP connection

[dabla]

This PR allows you to define the check_response option in the extra options fields of the HTTP connection so that you can disable the check globally for that connection instead of always having to pass it to each HttpOperator using that same connection id.

I also took the opportunity to refactor the get_conn method of the HttpHook so you can easily override the default host and default headers in the specialized hooks so those subclasses (e.g. LivyHook or DingdingHook) don't need to override the get_conn method anymore, otherwise as the signature has changed (e.g. added extra_options parameter) those has to be changed also and the exisiting solution wasn't DRY also, now it is.

In the past if you want to disable the response_check in the HttpHook through the HtpOperator, you had to define the HttpOperator like this:

validate_custom_fields_task = HttpOperator(
    task_id="get_data",
    method="GET",
    http_conn_id="http_conn_id",
    endpoint="/endpoint",
    headers={},
    response_check=response_check,
    extra_options={"check_response": False},  # allows you to use the custom response_check instead of the HttpHook buildin check
    dag=dag,
)

Now thanks to this PR, you can define the check_response globally in the extra_options of the connection, so it will be applied to each HttpOperator using that connection.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[dabla]

@potiuk or @eladkal Could any of you have a look and check if this MR can be merged please? It's a fix for issue #45237

[potiuk]

You will need to bump HTTP provider version to new minor and make the other providers depend on it, otherwise someone can install old version of HTTP provider and new version of (say) livy and it wil not work.

[dabla]

You will need to bump HTTP provider version to new minor and make the other providers depend on it, otherwise someone can install old version of HTTP provider and new version of (say) livy and it wil not work.

Ok thx @potiuk bumped the version of the http, livy and dingding provider and made them dependent on version 5.0.1 of http provider.

[dabla]

You will need to bump HTTP provider version to new minor and make the other providers depend on it, otherwise someone can install old version of HTTP provider and new version of (say) livy and it wil not work.

I've bumped the http provider (in provider.yaml) but is it normal I have to modify the version manually in module as well?

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "5.0.0"
+__version__ = "5.0.1"

Maybe it's because I can't use the pre-commits on my Windows machine.

[pierrejeambrun]

I think something went wrong with the branch. Maybe a bad rebase or something, can you check and clean the branch please ?

[eladkal]

We better have the field as boolean rather than define it as json in extra.
You can see Snowflake connection (insecure_mode)for reference

airflow/providers/snowflake/src/airflow/providers/snowflake/hooks/snowflake.py

Line 202 in 7e9b9ed

insecure_mode = _try_to_boolean(self._get_field(extra_dict, "insecure_mode"))

[dabla]

We better have the field as boolean rather than define it as json in extra. You can see Snowflake connection (insecure_mode)for reference

airflow/providers/snowflake/src/airflow/providers/snowflake/hooks/snowflake.py

Line 202 in 7e9b9ed

insecure_mode = _try_to_boolean(self._get_field(extra_dict, "insecure_mode"))

Alright good catch @eladkal will adapt that thx 😃

[dabla]

We better have the field as boolean rather than define it as json in extra. You can see Snowflake connection (insecure_mode)for reference

airflow/providers/snowflake/src/airflow/providers/snowflake/hooks/snowflake.py

Line 202 in 7e9b9ed

insecure_mode = _try_to_boolean(self._get_field(extra_dict, "insecure_mode"))

I've just checked the snowflake Hook, couldn't we do this change in another PR as this is quite a big change? Also I think we should think of some common code that could be reused across hooks, so I would prefer not to do this change in this PR but do it in another one.