Skip to content

Conversation

@kacpermuda
Copy link
Contributor

@kacpermuda kacpermuda commented Jan 13, 2026

Documentation improvements for OpenLineage provider

This PR improves the OpenLineage provider documentation with better organization and expanded content:

  • spark.rst: Added separate page for Spark integration, explaining that it's a separate entity from the Airflow provider and requires its own installation and configuration.

  • macros.rst: Expanded to be the central documentation for job hierarchy and macros, consolidating information about cross-job dependencies (TriggerDagRunOperator, API triggers, ExternalTaskSensor, Airflow Assets) and how OpenLineage handles job relationships in each scenario.

  • troubleshooting.rst: Added troubleshooting page with best practices and common errors, where users trying to debug OL problems can find helpful information.

  • supported_classes.rst: Added explanation of what "supported" means, clarifying that all operators emit basic lineage while "supported" operators provide additional operator-specific metadata.

  • guides/structure.rst: Added paragraph explaining the difference between the openlineage-python client package and the apache-airflow-providers-openlineage provider package.

  • guides/developer.rst: Added section about helper functions e.g. emit_openlineage_events_for_databricks_queries

  • provider.yml: Reordered options so that they are alphabetical.

  • guides/user.rst: Moved some basic stuff to structure.rst and dev stuff to developer.rst, nothing was left so this file was removed.

Was generative AI tooling used to co-author this PR?

  • Yes (please specify the tool below)

Used combination of different models to re-write parts of the docs and look for inconsistencies.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@kacpermuda kacpermuda force-pushed the feat-ol-troubleshooting-doc branch from bd74367 to 32da1cb Compare January 13, 2026 16:31
@kacpermuda kacpermuda marked this pull request as ready for review January 13, 2026 16:31
@kacpermuda kacpermuda force-pushed the feat-ol-troubleshooting-doc branch from 32da1cb to c425cf7 Compare January 14, 2026 19:20
@kacpermuda kacpermuda force-pushed the feat-ol-troubleshooting-doc branch from c425cf7 to b280d7c Compare January 15, 2026 10:05
@mobuchowski mobuchowski merged commit 8acd7c0 into apache:main Jan 19, 2026
88 checks passed
@kacpermuda kacpermuda deleted the feat-ol-troubleshooting-doc branch January 20, 2026 01:30
jason810496 pushed a commit to jason810496/airflow that referenced this pull request Jan 22, 2026
suii2210 pushed a commit to suii2210/airflow that referenced this pull request Jan 26, 2026
shreyas-dev pushed a commit to shreyas-dev/airflow that referenced this pull request Jan 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants