Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add supported hooks to OpenLineage docs #41958

Merged
merged 1 commit into from
Oct 22, 2024

Conversation

kacpermuda
Copy link
Contributor

@kacpermuda kacpermuda commented Sep 2, 2024

As Hook Level Lineage (AIP-62) is now live, we should update OpenLineage docs by adding supported hooks.

Also, here is a preview of the change:
Before:
image

After:
image

Side notes:

For now, we only check for classes for full calls like get_hook_lineage_collector().add_input_dataset(...) or get_hook_lineage_collector().add_output_dataset(...). Any other call will not be detected. Statically detecting these calls is challenging, so for now we should probably leave it like this, on a best effort basis with an important notice at the top.

Also, I don't think we can cross reference that with OL translators provided for schema in all providers and detect which hook has OL translator provided. We can assume that when adding hook level lineage, one will try to add ol translator as well. I'll try to work on some most popular hooks soon.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@kacpermuda kacpermuda force-pushed the ol-docs-add-hook-level-lineage branch from 2e76fd4 to 4a7db32 Compare September 3, 2024 10:22
@kacpermuda kacpermuda marked this pull request as ready for review September 3, 2024 10:25
@kacpermuda kacpermuda force-pushed the ol-docs-add-hook-level-lineage branch from 4a7db32 to 1ae98c6 Compare September 24, 2024 10:15
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

Signed-off-by: Kacper Muda <mudakacper@gmail.com>
@kacpermuda kacpermuda force-pushed the ol-docs-add-hook-level-lineage branch from 1ae98c6 to 1d1393d Compare October 22, 2024 11:00
@kacpermuda
Copy link
Contributor Author

Adjusted the doc to work with both new add_*_asset and old add_*_dataset methods, as different providers may use different ones. IMHO it's ready to be merged.

@mobuchowski mobuchowski merged commit 15b41b4 into apache:main Oct 22, 2024
42 checks passed
@kacpermuda kacpermuda deleted the ol-docs-add-hook-level-lineage branch October 22, 2024 18:21
harjeevanmaan pushed a commit to harjeevanmaan/airflow that referenced this pull request Oct 23, 2024
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
PaulKobow7536 pushed a commit to PaulKobow7536/airflow that referenced this pull request Oct 24, 2024
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants