fix(datasets): persist dataset extra dict to dataset event

[error418]

---

Persists Dataset extras to the according DatasetEvent to be used in Tasks invoked by Data aware scheduling. Please see linked issue for more information

closes: #35297

[uranusjr]

Do we need to do the merge at all? From what I can tell, the extra parameter on register_dataset_change is currently not used anywhere, and I suspect the original intention was to use that field to forward extra from the dataset directly to dataset event instead. Maybe we should just remove extra=None from register_dataset_change and just do the forwarding.

cc @blag for some history on this

[error418]

@uranusjr I also thought the implementation was intended this way. I removed the merge with the latest commit to this PR for the review.

In case the discussion comes to the result that a merge of the dictionaries is required we can quickly roll back.

[blag]

Please refer to the PR implementing this, and the discussion about an extra parameter here.

IIRC, the intent here was not to break database normalization and copy DatasetModel.extra for every DatasetEvent, it was to enable third party DatasetManagers (originally named DatasetEventManager in my PR linked above) to record additional data from external sources (including non-Airflow sources) with a minimum of fuss. The extra fields of DatasetModel and DatasetEvent are similarly named, but used for different and distinct purposes, and I did not intend for DatasetEvent.extra to be copied from DatasetModel.extra.

Given that, I would not expect DatasetEvent.extra to be used within Airflow, as if we wanted to store more information in the DatasetEvent table, we would just update the database schema. However, just because it is not used internally within Airflow does not mean may not be useful.

IMO #35297 is invalid, because tasks should be able to query the Airflow DB directly for the dataset extra field using the keys in triggering_dataset_events, and should absolutely not expect the dataset extra field to be copied to all dataset event extra fields.

[blag]

The docs are not optimally worded and definitely conflate datasets and dataset events, possibly leading to this confusion. See here.

Fetching information from a Triggering Dataset Event

A triggered DAG can fetch information from the Dataset that triggered it using the triggering_dataset_events template or parameter.

Do datasets trigger DAGs, or do dataset events trigger DAGs? The title of this section and its text seem to disagree. The answer is: dataset events trigger DAGs.

A one word change would vastly improve the precision of this section:

A triggered DAG can fetch information from the Dataset event that triggered it using the triggering_dataset_events template or parameter.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.