-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AIP-72: Task SDK support for on_task_instance_* listeners, make OpenLineage compatible #45294
Open
mobuchowski
wants to merge
4
commits into
apache:main
Choose a base branch
from
mobuchowski:tasksdk-call-listeners
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,821
−778
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:Executors-core
LocalExecutor & SequentialExecutor
area:providers
area:Scheduler
including HA (high availability) scheduler
area:task-sdk
provider:openlineage
AIP-53
labels
Dec 30, 2024
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
4 times, most recently
from
January 2, 2025 08:20
f07a78e
to
81b886b
Compare
@mobuchowski - can you please rebase that one -> we found and issue with @jscheffl with the new caching scheme - fixed in #45347 that would run "main" version of the tests. I am asking in all affected PRs to rebase. |
potiuk
force-pushed
the
tasksdk-call-listeners
branch
from
January 2, 2025 12:26
81b886b
to
15fd95a
Compare
Actually - I rebased it now. |
I will rebase very soon as I'm working on some of the test failures anyway 🙂 |
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
7 times, most recently
from
January 6, 2025 16:35
35a94e2
to
86297af
Compare
mobuchowski
requested review from
ephraimbuddy,
pierrejeambrun,
jedcunningham,
XD-DENG,
ashb,
o-nikolas and
hussein-awala
as code owners
January 6, 2025 16:52
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
from
January 6, 2025 17:04
86297af
to
6d88389
Compare
mobuchowski
changed the title
[draft] AIP-72: call on_task_instance_* listeners
AIP-72: call on_task_instance_* listeners
Jan 6, 2025
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
4 times, most recently
from
January 8, 2025 12:32
454af61
to
a7fb9e5
Compare
mobuchowski
changed the title
AIP-72: call on_task_instance_* listeners
AIP-72: Task SDK support for on_task_instance_* listeners, make OpenLineage compatible
Jan 8, 2025
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
from
January 8, 2025 14:05
a7fb9e5
to
0b6cd12
Compare
kaxil
reviewed
Jan 8, 2025
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
from
January 8, 2025 18:32
3375938
to
843f34e
Compare
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
from
January 9, 2025 16:35
843f34e
to
498d80c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area:Executors-core
LocalExecutor & SequentialExecutor
area:providers
area:Scheduler
including HA (high availability) scheduler
area:task-sdk
provider:openlineage
AIP-53
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With AIP-72, there is no access to the database session from the worker process, and the runtime objects have some differences to the db models. This PR contains three commits that deal with that situation:
on_task_instance_*
listeners interface to AIP-72: dropssession
argument and makestask_instance
argument an instance ofRuntimeTaskInstance
class, not database modelSome followup work:
Activity
to make logging better visible from UI, and distinct from task logsRuntimeTaskInstance
)closes #45423