Add ExecutableCoordinator for native self-contained Dag bundles#67161
Draft
jason810496 wants to merge 66 commits into
Draft
Add ExecutableCoordinator for native self-contained Dag bundles#67161jason810496 wants to merge 66 commits into
jason810496 wants to merge 66 commits into
Conversation
- Introduced the `apache-airflow-providers-languages-java` package with version 0.1.0. - Added Java-specific task coordinators and DAG file processors. - Created documentation including README, changelog, and installation instructions. - Implemented provider info retrieval and commit tracking. - Established testing framework with initial unit tests for Java provider components.
…ng for improved performance and reliability
…e related components
- Renamed all instances of "process coordinators" to "runtime coordinators" in the codebase. - Updated the ProvidersManager and ProvidersManagerTaskRuntime classes to handle runtime coordinators. - Modified the DagFileProcessorManager to collect file extensions from runtime coordinators. - Adjusted the Java provider to implement the new runtime coordinator structure. - Updated tests to reflect changes from process to runtime coordinators.
…_runtime_mapping] config
Tweak coordinator class names, attribute names, and method names to be shorter and avoid the term 'runtime'.
- Remove Java SDK setup in Dockerfile - add multi-language extras documentation - Update TaskInstanceDTO description, and adjust API version in generated files
…ocumentation and codebase
…etadata and source
…etection and update documentation accordingly
boring-cyborg
Bot
added
area:ConfigTemplates
area:DAG-processing
area:dev-tools
area:Executors-core
jason810496
added
go-sdk
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # same comm/logs server bootstrap and socket cleanup. | ||
| def _start_server() -> socket.socket: | ||
| server = socket.socket() | ||
| server.bind(("", 0)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
BaseCoordinatorinterface already supports non-Python SDKs (Java is the prior art),but there is no in-tree coordinator for native compiled SDKs (Go, Rust, C++, Zig, ...).
This adds one —
ExecutableCoordinator— together with a language-agnostic on-diskbundle format (
AFBNDL01) so any compiled SDK produces artifacts the TaskSDK can handle identically.What
and
airflow-metadata.yamlmanifest appended after end-of-file, located by a fixed32-byte trailer ending in the magic
AFBNDL01. The file stays directly runnable;detection is by trailer magic, not by filename or extension.
ExecutableCoordinator(subclass ofBaseCoordinator) that:BundleScanner, reading the trailer to extract theembedded source and metadata without unpacking;
sockets the child connects back to, reusing the
ActivitySubprocessplumbingintroduced for
JavaCoordinator;the worker's
$PATHdiffers from the build environment.Was generative AI tooling used to co-author this PR?