openlineage: add support for system tests, add support for example_bigquery_queries system test #31717
Conversation
mobuchowski
requested review from
potiuk,
ashb,
jedcunningham,
kaxil and
XD-DENG
as code owners
boring-cyborg
bot
added
area:dev-tools
area:providers
area:system-tests
provider:google
mobuchowski
force-pushed
the
openlineage-system-tests
branch
3 times, most recently
from
e99a8fd
to
fb98053
Compare
| @@ -235,11 +236,21 @@ | |||
| trigger_rule=TriggerRule.ALL_DONE, | |||
| ) | |||
|
|
|||
| openlineage_test = OpenlineageTestOperator( | |||
There was a problem hiding this comment.
I am wondering if this is the right place to keep OpenLineage system tests or should we have it in its own home tests/system/providers/openlineage/...? WDYT?
Wondering if some config disables OpenLineage or uses a different lineage backend then would having this here affect the system test run?
There was a problem hiding this comment.
I believe it's advantageous to have OL system tests as an addition to regular ones, because they are just an addition of one operator, and actively use rest of the dag.
Those aren't only "OpenLineage system tests" in vacuum, but test OL integration of this particular provider.
Having separate dags for system tests would result in fragmentation - we'd have to maintain the same dag, or at least subset of it, with just the addition of a single OpenLineageTestOperator.
However, I believe option to disable OpenLineage and make this special operator always pass is a very valid need, and will add such option.
| @@ -235,11 +236,21 @@ | |||
| trigger_rule=TriggerRule.ALL_DONE, | |||
| ) | |||
|
|
|||
| openlineage_test = OpenlineageTestOperator( | |||
mobuchowski
force-pushed
the
openlineage-system-tests
branch
from
fb98053
to
dd5f3d1
Compare
mobuchowski
force-pushed
the
openlineage-system-tests
branch
4 times, most recently
from
61718b5
to
351086a
Compare
mobuchowski
force-pushed
the
openlineage-system-tests
branch
from
351086a
to
b61a13a
Compare
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
mobuchowski
force-pushed
the
openlineage-system-tests
branch
from
b61a13a
to
25a95e1
Compare
|
Will reopen later |
The idea for system tests is as follows:
VariableTransportthat will push OpenLineage events toVariablewith key<DAG_ID>.<TASK_ID>.event.<EVENT_TYPE>.OpenLineageTestOperatorwill compare theexpectedevents with the ones that are stored in Variables.This PR adds those components, as well as OL support for
BigQueryGetDataOperatorand OL support for system test utilizing it.PR is based on previous PR: #31293 - only second commit is relevant to this PR.
Closes: #29676