AIP-72: Handling task instance state exceptions from task sdk and execution API

[amoghrajesh]

Body

Similar to #44137, there are various state exceptions raised while running ti.task.execute. This issue tracks handling them gracefully.

• Deferral Exception: TaskDeferred (PR: AIP 72: Handling "deferrable" tasks in execution_api and task SDK #44241 @amoghrajesh)
• Skip Exception: AirflowSkipException (PR AIP-72: Handling skipped tasks in task_sdk #44786 @amoghrajesh)
• Reschedule exception: AirflowRescheduleException (PR AIP-72: Handling "up_for_reschedule" task instance states #44907 @amoghrajesh)
• Fail Exception: AirflowFailException (PR AIP-72: Handling failed TI state for AirflowFailException & AirflowSensorTimeout #44954 @amoghrajesh)
• Sensor timeout exception: AirflowSensorTimeout (PR AIP-72: Handling failed TI state for AirflowFailException & AirflowSensorTimeout #44954 @amoghrajesh)
• Task terminated exception: AirflowTaskTerminated (PR AIP-72: Handling failed TI state for AirflowTaskTerminated #44977 @amoghrajesh)
• Task timeout (up_for_retry state) AirflowTaskTimeout (PR AIP-72: Handling execution_timeout for tasks in task SDK #45310 @amoghrajesh)
• Airflow Exception (up_for_retry state) (PR AIP-72: Handling AirflowException in task sdk #45308 @amoghrajesh)
• System Exit: SystemExit(PR: AIP-72: Handling SystemExit from task sdk #45282 @amoghrajesh)
• Base Exception: BaseException (PR AIP-72: Handling task retries in task SDK + execution API #45106 @amoghrajesh)

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[ashb]

System exit should be handled already I thought?

[amoghrajesh]

I do not think we do it:

    except SystemExit:
        ...
    except BaseException:
        # TODO: Handle TI handle failure
        raise