[v3-2-test] Fix conf.has_option not respects default provider metadata (#64209)

[jason810496]

• Fix provider config loading and fallback logic

Simplifies and centralizes provider configuration loading by removing redundant methods and properties, ensuring consistent state management. Updates fallback logic to use a sentinel value instead of None, improving accuracy in config lookups. Improves test coverage for provider metadata fallbacks and clarifies lazy initialization behavior. Streamlines provider manager interfaces to reduce duplication and potential import cycles.

• Cleanup load_providers_configuration and respect with new provider conf mechanism

• Fix configuration_description cached_property error to restore .write compatibility

• Try fix unit tests error

• Fix provider config merging and section separation

Ensures provider configuration sections do not overlap with core sections and gives priority to provider metadata when merging configuration options. Refactors tests to verify correct toggling of provider configuration and proper separation of provider and core config sections. Updates fallback lookup order and cleans up unused properties for clarity and maintainability.

• Fix check-default-configuration check and harden the error case

Root cause: providers_discovery.py used structlog.getLogger() directly. Before structlog.configure() is called (which happens later in settings.py:726), structlog's default PrintLogger writes to stdout with no level filtering. So debug logs during early provider
discovery pollute the stdout of airflow config list --default, corrupting the generated config file.

Fix: Switched to logging.getLogger() (stdlib). stdlib logging defaults to WARNING level and writes to stderr, so debug logs are suppressed and stdout stays clean. This is also the correct pattern for shared library code — structlog configuration is the application's
responsibility.

• Use conf.invalidate_cache() instead of dict.pop

• Harden the provider conf tests

• Address Copilot's review comments

• Use invalidate_cache instead of dict.pop

• Restore dict.pop back instead of invalidate_cache

• Fix lookup_sequence not respect _use_providers_configuration

• Address Wei's review comment for run_check_default_configuration

• Add dedicated _invalidate_provider_flag_caches method

• Create fresh conf instance for tests case that might mutate its internal state for thread-safe concern

• Deprecate load_providers_configuration, restore_core_default_configuration instead of directly remove

• Address Amogh's comment: Deprecate already_initialized_provider_configs instead of directly removing it

Add deprecation warning to already_initialized_provider_configs property in both ProvidersManager and ProvidersManagerTaskRuntime, guiding users to use provider_configs instead.

• Address Amogh's comment: Use VALUE_NOT_FOUND_SENTINEL for provider cfg fallback consistency

Change fallback=None to fallback=VALUE_NOT_FOUND_SENTINEL in get_from_provider_cfg_config_fallback_defaults and update the corresponding sentinel check for consistency with the metadata variant.

• Address Amogh's comment: Fix typo in _use_providers_configuration comment

---

(cherry picked from commit 4c63306)

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.