Scheduler crashes deserializing templated string in `*_date` task field not in `template_fields`

[abhishekbhakat]

Apache Airflow version

2.11.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

• A DAG defines a custom operator that accepts start_date/end_date strings and passes them into an args list. The task instance is created with a Jinja-templated string for end_date (and/or start_date).
• The operator does not list these fields in template_fields.
• Airflow serializes the DAG and stores the templated string (e.g., "{{ (data_interval_end - macros.timedelta(days=4)).strftime('%Y-%m-%d') }}") under the end_date key.
• When the scheduler later creates a DAG run and needs to deserialize, it treats any key ending with _date as a datetime and force-coerces it via _deserialize_datetime. Because the value is a string template, deserialization raises a TypeError and the scheduler crashes.

What you think should happen instead?

• The scheduler should never crash due to a single misconfigured/bad DAG. It should log and skip the DAG.
• Deserialization should not attempt to coerce templated strings or non-timestamp values for fields matching *_date.
• Proposed remediations (happy to submit a PR):
  1. Scheduler guard: wrap dag = self.dagbag.get_dag(dag_model.dag_id, session=session) in try/except in both _create_dag_runs and _create_dag_runs_dataset_triggered, similar to the existing guard around dag.create_dagrun(...).
  2. Deserialization guard: in airflow/serialization/serialized_objects.py, change the _date handling to be safe for non-numeric values, e.g.:
     • Only coerce when not isinstance(v, str), or
     • Wrap _deserialize_datetime(v) in try/except and leave v as-is on failure.
       Both can be complementary. The scheduler guard provides resilience. The deserialization guard fixes the root cause across components (scheduler/webserver/CLI).

How to reproduce

1. Define operators:

from airflow.models.baseoperator import BaseOperator
from airflow.utils.decorators import apply_defaults

class FooOperator(BaseOperator):

    # Trigger of the bug: template_fields = ("args",) is a tuple but ("args") is a string
    template_fields = ("args")

    @apply_defaults
    def __init__(self, args=None, **kwargs):
        super().__init__(**kwargs)
        self.args = args

    def execute(self, context):
        self.log.info("Executing FooOperator with args: %s", self.args)

class BarOperator(FooOperator):
    def __init__(self, start_date: str, end_date: str, **kwargs):
        super().__init__(
            args=["--start_date", start_date, "--end_date", end_date],
            **kwargs,
        )

2. Define DAG:

from airflow.models import DAG
from airflow.operators.empty import EmptyOperator

START_DATE = "{{ (data_interval_start - macros.timedelta(days=11)).strftime('%Y-%m-%d') }}"
END_DATE = "{{ (data_interval_end - macros.timedelta(days=4)).strftime('%Y-%m-%d') }}"

with DAG(
    dag_id="foo_bar_dag",
    schedule_interval="0 7 * * *",
    catchup=False,
    max_active_runs=1,
) as dag:
    start = EmptyOperator(task_id="start")
    end = EmptyOperator(task_id="end")

    foo_task = BarOperator(
        task_id="foo_task",
        start_date=START_DATE,
        end_date=END_DATE,
    )

    start >> foo_task >> end

3. Behavior:

• On initial parse, the serialized DAG stores end_date as the templated string.
• During DAG run creation, the scheduler deserializes and attempts to coerce end_date via _deserialize_datetime, which expects a numeric timestamp, causing TypeError and crashing the scheduler.

Operating System

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian

Versions of Apache Airflow Providers

Not applicable

Deployment

Docker-Compose

Deployment details

No response

Anything else?

• Relation to Issue DAG fails serialization if template_field contains execution_timeout #29819 / PR Add a check for not templateable fields #29821 (forbidden_fields):

  • PR Add a check for not templateable fields #29821 introduced a forbidden_fields check to prevent listing BaseOperator constructor args (e.g., execution_timeout, start_date, end_date) in template_fields, preventing one class of crash.
  • This issue is the inverse: a templated string in a *_date field that is not templated. The deserializer coerces any *_date value, which fails on strings. forbidden_fields does not intercept this case and also forbids adding start_date/end_date to template_fields to skip coercion.

• Proposal:

  • Scheduler: add try/except around get_dag(...) in _create_dag_runs and _create_dag_runs_dataset_triggered.
  • Deserialization: guard _date coercion (type-check or try/except) in both operator and DAG deserialization paths.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ashb]

@abhishekbhakat Can we see the stack trace please? Part of the issue is start_date and End_date have special meaning to Airflow and cant be templated.

But as you show in your example where args is all that is templated I'm surprised by the change needed or the error.

Is this example 100% representative?

[abhishekbhakat]

@abhishekbhakat Can we see the stack trace please? Part of the issue is start_date and End_date have special meaning to Airflow and cant be templated.

But as you show in your example where args is all that is templated I'm surprised by the change needed or the error.

Is this example 100% representative?

#29819 and #54775 has all the stack trace.

We don't have proper guardrails to stop users from templating start_date and end_date. Many users have tried the method of extending the operator class and making it a templatable field. This indicates that there is an ask from users.
We can either:

• force fail the dag if we find user trying to make it dynamic
• or add a dry rendering at the time of deserialization to allow templating.

I went with the 2nd option, as that's what users were asking.

[abhishekbhakat]

I've tried to create test dags for every way I found users trying. That will also give you some idea.