Remove old lineage stuff

[jason810496]

closes: #44983

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[jason810496]

When attempting to remove the lineage logic from the core module, I noticed that it causes failures in tests related to OpenLineage listener capturing hook-level lineage (#41482).

For example, removing apply_lineage and prepare_lineage from BaseOperator:

airflow/airflow/models/baseoperator.py

Lines 705 to 739 in 0efd9e6

	@prepare_lineage
	def pre_execute(self, context: Any):
	"""Execute right before self.execute() is called."""
	if self._pre_execute_hook is None:
	return
	ExecutionCallableRunner(
	self._pre_execute_hook,
	context_get_outlet_events(context),
	logger=self.log,
	).run(context)
	def execute(self, context: Context) -> Any:
	"""
	Derive when creating an operator.
	Context is the same dictionary used as when rendering jinja templates.
	Refer to get_template_context for more context.
	"""
	raise NotImplementedError()
	@apply_lineage
	def post_execute(self, context: Any, result: Any = None):
	"""
	Execute right after self.execute() is called.
	It is passed the execution context and any results returned by the operator.
	"""
	if self._post_execute_hook is None:
	return
	ExecutionCallableRunner(
	self._post_execute_hook,
	context_get_outlet_events(context),
	logger=self.log,
	).run(context, result)

Results in the following test failure:

FAILED providers/tests/openlineage/extractors/test_manager.py::test_extractor_manager_gets_data_from_pythonoperator - assert 0 == 1
 +  where 0 = len([])
 +    where [] = HookLineage(inputs=[], outputs=[]).outputs

airflow/providers/tests/openlineage/extractors/test_manager.py

Line 301 in 0efd9e6

def test_extractor_manager_gets_data_from_pythonoperator(session, dag_maker, hook_lineage_collector):

It seems OpenLineage is still coupled with the lineage module and might need to be moved to compact.lineage for now (or to the OpenLineage module in a future PR ).

After some experimentation, implementing an on_load callback in the OpenLineageProviderPlugin to monkey-patch the core module at runtime prevents the OpenLineage test failures, even with the lineage module removed from the core.

Based on https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html#interface

I’m not sure if this is a suitable long-term solution for maintaining OpenLineage compatibility while cleaning up the lineage module ?

cc @Lee-W @uranusjr

[jason810496]

The CI failure is caused by a flaky K8s test (#45145) and a breaking change in the compatibility tests for providers 2.9.3 and 2.10.4 .

Should we fix the tests to pass the compatibility checks, or is it acceptable to ignore the compatibility tests since all tests for 3.0 have passed ?

[Lee-W]

The CI failure is caused by a flaky K8s test (#45145) and a breaking change in the compatibility tests for providers 2.3.9 and 2.3.10.

Should we fix the tests to pass the compatibility checks, or is it acceptable to ignore the compatibility tests since all tests for 3.0 have passed ?

2.9.3 or 2.3.9. If it's 2.9.3, then we'll need to fix it as providers will still support airflow 2 for some time

[jason810496]

2.9.3 or 2.3.9. If it's 2.9.3, then we'll need to fix it as providers will still support airflow 2 for some time

Sorry for the typo, I mean 2.9.3 and 2.10.4 . Thanks for reply, if then I will fix the test.

[uranusjr]

We should not remove the entire lineage mechanism. The issue is only meant to remove the old lineage constructs in airflow.lineage.entities. The rest of the mechanism integrates modern Airflow assets with OpenLineage, and they should be kept.

[mobuchowski]

Just as TP said. Removing airflow/lineage/hook.py is wrong

[potiuk]

@jason810496 I rebased it -> we found and issue with @jscheffl with the new caching scheme - fixed in #45347 that would run "main" version of the tests.

[jason810496]

Fixed: I only moved airflow.lineage.entities to airflow.providers.common.compat.lineage.entities.
The CI failure is unrelated, as no relevant frontend tests failed.

One small question: Does this PR need to be backported to 2.10 ?

[Lee-W]

Hi @jason810496, I just found the PR we discussed yesterday. #44720 I think this one would be useful for this PR.

[uranusjr]

One small question: Does this PR need to be backported to 2.10?

No since you only moved the classes. The providers are always released from main, and it’s OK for those classes to be available in two places in 2.10.