Amazon NeptuneStopDbClusterOperator bug #38120
Description
Apache Airflow Provider(s)
amazon
Versions of Apache Airflow Providers
latest; AWS System test stack pulls from main
Apache Airflow version
latest; AWS System test stack pulls from main
Operating System
linux
Deployment
Docker-Compose
Deployment details
No response
What happened
On occasion the stop_db_cluster() call will return a "backing-up" state which is not handled, causing an exception. Example log message:
ERROR [airflow.task] Task failed with exception
Traceback (most recent call last):
File "/opt/airflow/airflow/models/taskinstance.py", line 447, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
File "/opt/airflow/airflow/models/taskinstance.py", line 417, in _execute_callable
return execute_callable(context=context, **execute_callable_kwargs)
File "/opt/airflow/airflow/providers/amazon/aws/operators/neptune.py", line 186, in execute
resp = self.hook.conn.stop_db_cluster(DBClusterIdentifier=self.cluster_id)
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 553, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 1009, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidDBClusterStateFault: An error occurred (InvalidDBClusterStateFault) when calling the StopDBCluster operation: DbCluster env989356b0-cluster is in backing-up state but expected it to be one of available.
INFO [airflow.models.taskinstance] Marking task as UP_FOR_RETRY. dag_id=example_neptune, task_id=stop_task, execution_date=20210101T000000, start_date=20240313T100059, end_date=20240313T100100
What you think should happen instead
If that is a possible state, it needs to be handled gracefully
How to reproduce
The issue is intermittent so I'm not 100% sure. Force a cluster to start a backup then immediately try to delete it while it is in progress.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct