Fix OpenLineage DAG & Task facets field types #51165
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
OpenLineage facet for Task describes inlets & outlets as string, but they actually list of strings:
{ "run": { "facets": { "airflow": { "_producer": "https://github.com/apache/airflow/tree/providers-openlineage/1.11.0", "_schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet", "task": { "depends_on_past": false, "downstream_task_ids": "['add_period_in_hive']", "executor_config": {}, "ignore_first_depends_on_past": true, "inlets": [], "is_setup": false, "is_teardown": false, "mapped": false, "multiple_outputs": false, "operator_class": "PythonOperator", "operator_class_path": "***.operators.python.PythonOperator", "outlets": [], "owner": "***", "priority_weight": 1, "queue": "default", "retries": 2, "retry_exponential_backoff": false, "task_id": "add_params_in_xcom", "trigger_rule": "all_success", "upstream_task_ids": "['Sensors.Sensor__hdp2gp_oebsap_dm__oebs-dm-replication-to-greenplum', 'Sensors.Sensor__hdp2gp_oebsar_dm__oebs-dm-replication-to-greenplum', 'Sensors.Sensor__hdp2gp_oebsfa_dm__oebs-dm-replication-to-greenplum', 'Sensors.Sensor__postgres2gp_1c_sales_report__oebs-dm-replication-to-greenplum', 'Sensors.Sensor__hdp2gp_oebsgl_dm__oebs-dm-replication-to-greenplum', 'Sensors.Sensor__hdp2gp_oebspa_dm__oebs-dm-replication-to-greenplum', 'Sensors.Sensor__hdp2gp_oebsinv_dm__oebs-dm-replication-to-greenplum', 'Sensors.Sensor__hdp2gp_oebsxtr_dm__oebs-dm-replication-to-greenplum', 'Sensors.Sensor__hdp2gp_oebspay_dm__oebs-dm-replication-to-greenplum', 'Sensors.Sensor__hdp2gp_oebsce_dm__oebs-dm-replication-to-greenplum']", "wait_for_downstream": false, "wait_for_past_depends_before_skipping": false, "weight_rule": "<<non-serializable: _DownstreamPriorityWeightStrategy>>" } } } } }Same for DAG tags:
{ "run": { "facets": { "airflow": { "_producer": "https://github.com/apache/airflow/tree/providers-openlineage/1.11.0", "_schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet", "dag": { "dag_id": "control_dm__oebsstatus", "fileloc": "/data/airflow/dags/oebsstatus/master/control_dm.py", "owner": "airflow", "schedule_interval": "00 4 * * *", "tags": [ "oebsstatus", "master" ], "timetable": { "expression": "00 4 * * *", "timezone": "UTC" } } } } } }^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.