Skip to content

Conversation

@shubhamraj-git
Copy link
Contributor

related: #45966

Debugging Airflow workflows often involves dealing with incorrect or missing XCom values, which can cause tasks to fail or behave unexpectedly.
Traditionally, fixing such issues requires rerunning tasks, modifying Dag logic, or even manually editing the Airflow metadata database—none of which are ideal in a production setting. With API-based XCom updates, users can quickly correct erroneous values without restarting entire workflows.
This is especially useful in long-running jobs, where reprocessing can be costly, or in dependency-heavy pipelines, where a single incorrect value can block multiple downstream tasks. By allowing targeted updates, this feature speeds up debugging, reduces downtime, and ensures smoother workflow execution.


Steps to play around the feature.

Add the following Dag.

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
import time

def push_xcom_value(ti):
    value_to_push = "Hello from XCom!"
    ti.xcom_push(key='my_key', value=value_to_push)
    time.sleep(60)  # Wait for 1 minute, meanwhile you can update the XCom via API

def pull_xcom_value(ti):
    pulled_value = ti.xcom_pull(task_ids='push_task', key='my_key')
    print(f"Pulled Value: {pulled_value}") # Updated value should be present

default_args = {
    'owner': 'airflow',
    'catchup': False,
    'start_date': datetime(2024, 2, 5),
}

dag = DAG(
    'xcom_push_pull_dag',
    default_args=default_args,
    description='A simple DAG to demonstrate XCom push and pull',
    catchup=False,
)

push_task = PythonOperator(
    task_id='push_task',
    python_callable=push_xcom_value,
    dag=dag,
)

pull_task = PythonOperator(
    task_id='pull_task',
    python_callable=pull_xcom_value,
    dag=dag,
)

push_task >> pull_task

As soon as push_task has pushed to xcom, just update the value via API.
http://localhost:29091/docs#/XCom/update_xcom_entry

image

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added the area:UI Related to UI/UX. For Frontend Developers. label Feb 5, 2025
@eladkal eladkal added this to the Airflow 3.0.0 milestone Feb 5, 2025
@eladkal eladkal added the type:new-feature Changelog: New Features label Feb 5, 2025
@eladkal eladkal merged commit c813c32 into apache:main Feb 6, 2025
45 checks passed
insomnes pushed a commit to insomnes/airflow that referenced this pull request Feb 6, 2025
* Add XCom update API

* Add tests for XCom update API

* Fix failures on execution
insomnes pushed a commit to insomnes/airflow that referenced this pull request Feb 6, 2025
* Add XCom update API

* Add tests for XCom update API

* Fix failures on execution
niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
* Add XCom update API

* Add tests for XCom update API

* Fix failures on execution
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
* Add XCom update API

* Add tests for XCom update API

* Fix failures on execution
@shubhamraj-git shubhamraj-git deleted the issue-45966-update branch September 7, 2025 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:UI Related to UI/UX. For Frontend Developers. type:new-feature Changelog: New Features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants