Fix Callback exception when a removed task is the last one in the task instance list

[joaopamaral]

When a task is removed and is returned in the last position of get_task_instances, the callback request will fail because it will try to access this task, but it doesn't exist anymore.

• Airflow 2.7.1:

ERROR - Error executing DagCallbackRequest callback for file: /dags/daily.py
Traceback (most recent call last):
  File "airflow/dag_processing/processor.py", line 711, in execute_callbacks
    self._execute_dag_callbacks(dagbag, request, session)
  File "airflow/utils/session.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "airflow/dag_processing/processor.py", line 745, in _execute_dag_callbacks
    dag.handle_callback(
  File "airflow/utils/session.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "airflow/models/dag.py", line 1422, in handle_callback
    ti.task = self.get_task(ti.task_id)
  File "airflow/models/dag.py", line 2496, in get_task
    raise TaskNotFound(f"Task {task_id} not found")
airflow.exceptions.TaskNotFound: Task removed_task not found

This exception was noticed in airflow 2.7.1 but the issue is still in the latest version

To fix it we are filtering out the removed tasks to avoid having a removed task in the last position again.

• After

>>> tis = dailydag.get_task_instances()
>>> dailydag.get_task(tis[-1].task_id)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "airflow/models/dag.py", line 2496, in get_task
    raise TaskNotFound(f"Task {task_id} not found")
airflow.exceptions.TaskNotFound: Task removed_task not found

• Before

>>> tis = [ti for ti in dailydag.get_task_instances() if ti.state != TaskInstanceState.REMOVED]
>>> dailydag.get_task(tis[-1].task_id)
<Task(EmptyOperator): existing_task>

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[hussein-awala]

Could you add a test for your change?

[eladkal]

Tests are failing
Can you look into it @joaopamaral ?