SPIKE: Investigate separation strategy for Triggerer and DAG Processor for server-client separation

[kaxil]

Investigate where Triggerer and DAG Processor should live in the AIP-72 server-client separation architecture.

Background

Triggerer and DAG Processor:

• Both are currently in airflow-core
• Both have dependencies on Task SDK execution-time components
• They create circular dependency issues between server and SDK

Open Questions

Where should Triggerer & DAG Processor live? Server or Client?

Option 1: Create airflow-core-execution provider

• Contains Task SDK Execution logic, worker, trigger, and DAG Processor
• Pros: Isolated execution concerns, clear separation
• Cons: Package explosion, coupling between apache-airflow-task-sdk and apache-airflow-core-execution

Option 2: Move Triggerer & DAG Processor to Task SDK

• Relocate both components to apache-airflow-task-sdk
• Pros: Execution components together, simpler dependency graph, enables future multi-language DAG authoring
• Cons: Task SDK becomes heavier, multi-language implementation complexity

Can we do them in separate phases i.e. not have explicit dependencies in metadata but import them dynamically if they are installed.

Multi-language SDK Implications:

• Future-proofing: Enables Go SDK and other languages to have native DAG definition & processing
• Language-specific optimizations: Each SDK could optimize for their ecosystem (Go's concurrency, etc.)
• Consistency challenges: Risk of different behaviors across language implementations
• Maintenance burden: Multiple Triggerer/DAG Processor implementations to maintain