SPIKE: Investigate what should be shared between server & client and determine if a common package is needed

[kaxil]

With server-client separation, we need to determine what should be shared vs duplicated vs kept separate.

Should the Providers with executors be separated into providers to that they can be included on Server side?

Categories of Potentially Shared Components

Core Interfaces and Protocols

• SecretsBackend
• AirflowConfigParser
• airflow.serialization
• Listeners
• Metrics
• Traces
• Airflow Local Settings [Ash is going to remove it] / Policy
• secrets_masker

Exception Hierarchies

• AirflowException: Base exception class
• AirflowSkipException: Task skipping exceptions
• AirflowSensorTimeout: Sensor-specific exceptions
• Common error types: Used by both server and SDK

Core Utilities

• Module loading: import_string, import_module_safely
• Trigger rules: TriggerRule enum and logic
• Date/time utilities: Timezone handling, date parsing
• String utilities: Template processing, formatting

Serialization Components

• Schema definitions: JSON schemas for DAG representation
• Schema versioning: Version management and compatibility
• Validation logic: Schema validation and error handling

Configuration Components

• Configuration interfaces: Protocol definitions for config access
• Default values: Shared configuration defaults
• Validation logic: Configuration validation rules
• Environment handling: Environment variable processing

Shared Package Strategy Options

Option 1: Single Shared Package (airflow-protocols / airflow-commons)

• Approach: All shared components in one package
• Pros: Simple dependency management, single source of truth
• Cons: Package may become large, mixed concerns
• Use case: Clean separation with minimal package proliferation

Option 2: No Shared Package (Duplication)

• Approach: Each package implements its own versions
• Pros: Complete independence, no circular dependencies
• Cons: Code duplication, maintenance overhead
• Use case: When separation is more important than efficiency

Key Questions to Answer

• Should the Providers with executors be separated into providers to that they can be included on Server side?
• How many packages is optimal? Trade-off between focus and complexity
• What naming convention is clearest? For users and developers
• How to handle backward compatibility? Migration path for existing users
• What about provider dependencies? Which packages should providers depend on?
• Testing strategy? How to test package combinations efficiently

Other open questions

• Where should Triggers be? They are inherited in providers for various deferrable operators.

Definition of Done

• All package structure options evaluated
• Dependency and testing implications understood
• Clear recommendation with migration path