Implemented streaming functionality with IterableOperator and DeferredIterable#42572

Closed

dabla wants to merge 167 commits intoapache:mainfrom

dabla:feature/streamed-operator

Commits on Sep 29, 2024

refactor: Implemented StreamedOperator which runs all mapped tasks within the same TaskInstance using a ThreadPoolExecutor on the same worker instance
davidblain-infrabel
committed
refactor: Changed return type of next_callable method
davidblain-infrabel
committed
refactor: Changed operator type to BaseOperator in OperatorMethodExecutor
davidblain-infrabel
committed

Commits on Oct 1, 2024

Commits on Oct 2, 2024

Commits on Oct 3, 2024

Commits on Oct 4, 2024

Commits on Oct 11, 2024

Commits on Oct 15, 2024

Merge branch 'main' into feature/streamed-operator
dabla
authored

Commits on Oct 18, 2024

Merge branch 'main' into feature/streamed-operator
dabla
authored

Commits on Oct 21, 2024

Merge branch 'main' into feature/streamed-operator
dabla
authored

Commits on Oct 23, 2024

Commits on Oct 25, 2024

Merge branch 'main' into feature/streamed-operator
dabla
authored

Commits on Nov 4, 2024

Merge branch 'main' into feature/streamed-operator
dabla
authored

Commits on Nov 21, 2024

Commits on Nov 22, 2024

refactor: Use ThreadPoolExecutor to execute tasks concurrently instead of purely relying on asyncio
davidblain-infrabel
committed

Commits on Nov 30, 2024

refactor: Refactored the StreamedOperator even more
davidblain-infrabel
committed

Commits on Dec 2, 2024

feature: Fix StreamedOperator so it can run triggers in async mode and normal
David Blain
committed
feature: Re-added semaphore for async execution
David Blain
committed

Commits on Dec 3, 2024

Commits on Dec 4, 2024

Commits on Dec 13, 2024

Merge branch 'main' into feature/streamed-operator
dabla
authored

Commits on Jan 10, 2025

Commits on Jan 28, 2025

Commits on Jan 29, 2025

Commits on Jan 30, 2025

Merge branch 'main' into feature/streamed-operator
dabla
authored

Commits on Feb 5, 2025

Commits on Feb 6, 2025

Merge branch 'main' into feature/streamed-operator
dabla
authored

Commits on Feb 10, 2025

Merge branch 'main' into feature/streamed-operator
dabla
authored

Commits on Feb 11, 2025

Commits on Feb 13, 2025

refactor: Updated IterableOperator which optimizes execution of deferrable tasks
David Blain
committed

Commits on Feb 18, 2025

Commits on Feb 19, 2025

Merge branch 'main' into feature/streamed-operator
dabla
authored

Commits on Feb 25, 2025

Commits on Feb 26, 2025

Commits on Mar 20, 2025

refactor: Inherit LoggingMixin for DeferrableIterable
David Blain
committed
refactor: Moved DeferrableIterable to dedicated iterable module under airflow models
David Blain
committed
refactor: Also use ThreadPool to execute task producer and consumer instead of Threads
David Blain
committed
refactor: Reformatted files
David Blain
committed

Commits on Mar 23, 2025

Commits on Mar 24, 2025

refactor: Improved IterableOperator and DeferredIterable
davidblain-infrabel
committed

Commits on Mar 26, 2025

refactor: Made next_kwargs optional for next_callable method in BaseOperator
davidblain-infrabel
committed

Commits on Mar 27, 2025

refactor: Improved lazy evaluation of tasks passed to the _run_tasks method of the IterableOperator
David Blain
committed
Merge branch 'main' into feature/streamed-operator
David Blain
committed

Commits on Mar 28, 2025

Commits on Apr 7, 2025

Commits on May 6, 2025

refactor: Refactored IterableOperator to be Airflow 3 compliant
davidblain-infrabel
committed

Commits on May 7, 2025

Commits on May 19, 2025

Merge branch 'main' into feature/streamed-operator
dabla
authored