Dataset Next Trigger Modal Not Populating Latest Update

[tseruga]

Apache Airflow version

2.4.1

What happened

When using dataset scheduling, it isn't obvious which datasets a downstream dataset consumer is awaiting in order for the DAG to be scheduled.

I would assume that this is supposed to be solved by the Latest Update column in the modal that opens when selecting x of y datasets updated, but it appears that the data isn't being populated.

Although one of the datasets has been produced, there is no data in the Latest Update column of the modal.

In the above example, both datasets have been produced > 1 time.

What you think should happen instead

The Latest Update column should be populated with the latest update timestamp for each dataset required to schedule a downstream, dataset consuming DAG.

Ideally there would be some form of highlighting on the "missing" datasets for quick visual feedback when DAGs have a large number of datasets required for scheduling.

How to reproduce

1. Create a DAG (or 2 individual DAGs) that produces 2 datasets
2. Produce both datasets
3. Then produce only one dataset
4. Check the modal by clicking from the home screen on the x of y datasets updated button.

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[tseruga]

Just to expand upon the state that is causing this issue a bit more...

The DatasetEvent model looks correct in the database:

Unsure why this query isn't populating the lastUpdate field in API response:

airflow/airflow/www/views.py

Lines 3447 to 3451 in 8898db9

	for info in session.query(
	DatasetModel.id,
	DatasetModel.uri,
	func.max(DatasetEvent.timestamp).label("lastUpdate"),
	)

[bbovenzi]

@tseruga could you share the created_at time for the most recent dag run? I wonder if somehow dag run was created after the next update to test_dataset

airflow/airflow/www/views.py

Lines 3461 to 3468 in 8898db9

	.join(
	DatasetEvent,
	and_(
	DatasetEvent.dataset_id == DatasetModel.id,
	DatasetEvent.timestamp > DatasetDagRunQueue.created_at,
	),
	isouter=True,
	)

[michaelmicheal]

Feel free to assign this to me. I agree this should be more robust and clear in the UI