CloudSecretManagerBackend fails with ADC when explicit project_id provided (regression in 19.4.0)

[kiranpreet-kaur17]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

apache-airflow-providers-google==19.4.0

Apache Airflow version

3.x affected

Operating System

All

Deployment

Other

Deployment details

N/A

What happened

After upgrading to apache-airflow-providers-google==19.4.0, CloudSecretManagerBackend fails to initialize when using Application Default Credentials (ADC) without a default project configured in gcloud, even when an explicit project_id is provided in BACKEND_KWARGS.

This issue was encountered in a local development environment.

Error:

airflow.exceptions.AirflowException: Project ID could not be determined from default credentials. Please provide `key_secret_project_id` parameter.

Configuration:

AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS='{"project_id":"my-project-id"}'

# Using ADC without default project (common when working with multiple projects)
gcloud auth application-default login
gcloud config unset project

This configuration worked fine in version 19.3.0 and earlier.

What you think should happen instead

CloudSecretManagerBackend should use the explicit project_id provided in BACKEND_KWARGS, regardless of whether ADC returns a default project. The validation should only fail if no project_id is available from ANY source.

How to reproduce

1. Install apache-airflow-providers-google==19.4.0
2. Configure secrets backend:

   export AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
   export AIRFLOW__SECRETS__BACKEND_KWARGS='{"project_id":"my-project-id"}'

3. Set up ADC without a default project:

   gcloud auth application-default login
   gcloud config unset project

4. Start Airflow
5. Observe the error during secrets backend initialization

Anything else

CloudSecretManagerBackend should use the explicit project_id provided in BACKEND_KWARGS, regardless of whether ADC returns a default project. The validation should only fail if no project_id is available from ANY source.

Root Cause:

Commit 0a3ee82ba3 (PR #60146) added validation in _CredentialProvider._get_credentials_using_adc() that raises an exception if google.auth.default() returns None for project_id.

The problem is that this validation happens before CloudSecretManagerBackend.__init__ can use the explicit project_id parameter:

# In CloudSecretManagerBackend.__init__ (lines 120-137)
self.credentials, self.project_id = get_credentials_and_project_id(...)
# ↑ This raises exception in 19.4.0 before reaching the code below ↓

# This code is never reached:
if project_id:
    self.project_id = project_id  # Would have used explicit project_id!

Why the error message is misleading:

The error suggests using key_secret_project_id, but this parameter is for a different purpose (specifying which project contains the credentials secret, not which project contains the Airflow secrets). It doesn't solve the problem for users providing project_id in BACKEND_KWARGS.

Comparison:

Version 19.3.0 (working):

def _get_credentials_using_adc(self) -> tuple[Credentials, str]:
    credentials, project_id = google.auth.default(scopes=self.scopes)
    return credentials, project_id  # No validation

Version 19.4.0 (broken):

def _get_credentials_using_adc(self) -> tuple[Credentials, str]:
    scopes = list(self.scopes) if self.scopes else None
    credentials, project_id = google.auth.default(scopes=scopes)
    if not project_id:  # Fails before caller can use explicit project_id
        raise AirflowException(...)
    return credentials, project_id

Proposed Fix:

Move the validation to the caller level, after checking all possible sources of project_id:

# In CloudSecretManagerBackend.__init__
self.credentials, adc_project_id = get_credentials_and_project_id(...)

# Use explicit project_id if provided, otherwise use ADC project_id
if project_id:
    self.project_id = project_id
else:
    self.project_id = adc_project_id

# NOW validate after checking all sources
if not self.project_id:
    raise AirflowException(
        "Project ID could not be determined. "
        "Please provide 'project_id' in backend configuration or ensure "
        "your credentials include a default project."
    )

Workaround:

Set the GOOGLE_CLOUD_PROJECT environment variable:

export GOOGLE_CLOUD_PROJECT=my-project-id

This makes google.auth.default() return a project_id, satisfying the validation.

Impact:

This is a regression that breaks a valid and common use case:

• Using ADC for authentication (Google's recommended approach)
• Working with multiple GCP projects without a default
• Explicitly configuring project_id in secrets backend settings

Affects both Airflow 2.x and 3.x users who upgrade to google provider 19.4.0+.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[shahar1]

Thank you for reporting, feel free to raise a PR!
I'll be happy to review.