Can't use logical_date in GCSToBigQueryOperator

[obarisk]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

16.1.0

Apache Airflow version

3.0.3

Operating System

debian

Deployment

Other Docker-based deployment

Deployment details

No response

What happened

for dag that trigger by Asset without schedule,
logical_date is not available in dag_run, context

in GCSToBigQueryOperator, we use context['logical_date'] to generate job_id
that'll be an error for dags trigger by asset.

What you think should happen instead

no error

How to reproduce

trigger the dag directly is okay. => with logical_date.
trigger the dag by trigger the asset. => error.

from airflow.sdk import DAG, Asset
from airflow.providers.google.cloud.transfers.gcs_to_bigquery import (
  GCSToBigQueryOperator
)

source_asset = Asset("/source_data")

with DAG(
  dag_id="no_logical_date",
  schedule=[source_asset],
  tags=["debug"]
):
  gcs_to_gbq = GCSToBigQueryOperator(
    task_id="gcs_to_gbq",
    bucket="example",
    source_objects=["some_data.csv"],
    destination_project_dataset_table="demo.example_table",
    gcp_conn_id="gcp-default",
  )

  gcs_to_gbq

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[uranusjr]

This is expected, logical date does not make sense for an asset-triggered task. We could add back the key (set the value to None), but that would not help with the issue (since every such task would just have None in their job ID). The operator should use another value in the context instead.

[obarisk]

@uranusjr sorry, then isn't this a bug in GCSToBigQueryOperator?
Because GCSToBigQueryOperator should not use logical_date for the bigquery's job name?

As logical_data is not an always available attribute, there should be any of the Operators put an assumption that logical_data existed.

---

But if the following is true. Then it's okay GCSToBigQueryOperator use logical_date to generate the name of bigquery job_id.

GCSToBigQueryOperator shouldn't be used for Asset trigger task

Otherwise, I'll say this is a bug. And we have to fix it. I probably know how to fix, but i'm not confident for writing the correct unit test for this.

[uranusjr]

This is a bug in GCSToBigQueryOperator, but the bug is not it can’t use logical_date. It should not use logical_date. Feel free to send a PR for it to use something else.