Using DatabricksSubmitRunOperator inside @task — is pool applied correctly

[Rishabh1627rawat]

Hi everyone,

I’m using Airflow 3.x with the @task decorator (TaskFlow API), and I’m trying to better understand how Airflow handles execution and pools when an operator is called inside a Python task.

Right now, I’m using this pattern:

@task(pool="databricks_superset_med", retries=3)
def run_databricks(run_payload, **context):
    op = DatabricksSubmitRunOperator(
        task_id="data_transformation",
        databricks_conn_id="databricks",
        json=run_payload,
        wait_for_termination=True,
    )
    return op.execute(context=context)

In this setup:

• The @task has a pool assigned.
• Inside that function, I create a DatabricksSubmitRunOperator.
• I manually call op.execute(context=context).

This successfully triggers my Databricks notebook, and because wait_for_termination=True, the task waits until the notebook run finishes.

However, I want to better understand what is happening internally.

Specifically:

• Is the pool applied only to the outer Python @task?
• From the scheduler’s perspective, is the DatabricksSubmitRunOperator treated as a separate task?
• Or is it completely invisible because it is executed manually inside the Python task?
• Why does the pool on the inner operator not take effect?
• Would it be better practice to define DatabricksSubmitRunOperator directly in the DAG instead of wrapping it inside a @task?

I understand that manually calling .execute() may bypass some of Airflow’s orchestration mechanisms. I’m especially curious how this pattern affects:

• Pool slot acquisition
• Task lifecycle tracking
• Scheduler awareness
• UI visibility

This is not a functional issue — everything runs successfully. I just want to understand the internal behavior better and ensure I’m following best practices without unintentionally bypassing Airflow’s concurrency controls.

I would appreciate any clarification on how the scheduler treats this pattern internally.

[potiuk]

You are not supposed to run operator in @task decorated methods - this is a bad pattern than should not be used. You can call hooks inside the tasks - but calling operators in @task decorated methods is just a calling "execute" method of the operator, without applying any semantics of operator. I belive in modern airflow versions you get a warning when you do that.

[Rishabh1627rawat]

Hi,

I noticed that in this implementation, the DatabricksSubmitRunOperator is created and its execute() method is called inside a @task-decorated function.

My senior implemented it this way and mentioned that the pool configuration is working correctly. However, I’m trying to better understand how this works internally.

Since the pool is defined on the @task, does that mean the pool slot is applied only to the outer TaskFlow task, and the Databricks operator is simply being executed as regular Python code rather than as a separately scheduled Airflow task?

I just want to clarify whether this is the intended and recommended pattern, or if defining the operator directly as a DAG task would be more appropriate.

Thanks in advance for the clarification!