-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Amplify push hangs indefinitely after failed cloudformation updates #2305
Comments
Hey @d-huck, 👋 thanks for raising this! From the logs provided it appear this occurring when pushing a API resource. |
@ykethan Thank you for the response. I had already attempted removing
before it resumes and hangs indefinitely. Attempting to continue rollback in cloudformation results in the same failure. Diving down the stack in CloudFormation, I see this error message:
which contains many of the resolvers for the object type I initially attempted to remove and am now attempting to restore. However, I can't push anything until this rollback completes, which is seemingly a Catch-22 |
Creating fake resolvers may allow you to get out of the UPDATE_ROLLBACK_FAILED state. #2157 (comment) |
Thanks for the link. I followed #2157 and was able to get past that into
I've tried deleting the deployment-state from the deployment s3 bucket. But I'm not able to move forward with the last known good schema. |
Are there any additional error messages on the CloudFormation console? |
There are a few possible solutions in #92. |
@dpilch Thank you for the link. I had been up and down that thread and was hoping for solutions other than the ones that were proposed there. In the end, we ended up destroying the api and associated tables and rebuilt them fresh, which wasn't as catastrophic as it could have been considering we're at a very early stage. It seems like this problem is common among people who are making large, quick changes to their backend, so hopefully we won't be facing this in the future. I'll spare y'all the rant about this being unacceptable, because I assume y'all have read #92 in detail. I'll leave our solution for anyone who may find themselves on this page in the future. First, if you've tried the normal things, don't hold out for a solution, just follow #92. Here's our resolution steps:
Outside of the backup and restoration process, this whole process takes roughly 30 minutes. |
I am sharing this because a similar event has occurred. [ Problem ]
[ after problem ] [ Cause ]
As a result, there was a difference of more than 2 between the number of GSIs in the CloudFormation Stack and the number of GSIs in the CloudFormation Template that Amplify first attempts to deploy. [ Recovery ]
Amplify CLI version : 12.8.2 |
How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
21.2.0
Amplify CLI Version
12.10.1
What operating system are you using?
MacOS
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
The only manual changes are a few custom images for running lambdas online. Based on other issues similar to mine, I attempted to make dummy resolvers to clear the UPDATE_ROLLBACK_FAILED state to no avail. These dummy resolvers have been removed.
Describe the bug
When pushing updates from our dev environment to production, API building failed due to an object being removed from the API. The API push has an unfortunately large number of changes due to our frontend devlopment lagging far behind. The commands used for pushing to main were:
Which failed after roughly 30 minutes of the CLI doing its thing. The result is our cloudformation stack is in UPDATE_ROLLBACK_FAILED and cannot be cleared out of this state.
After attempting to rollback, I have reverted the schema to the last known working state and pushed, which results in the behavior of an indefinite hang of the CLI. The last known working state does not remove the object in question. To further debug this, I pulled the main environment down using
amplify pull
, added a comment line to trigger a rebuild, and experience the same behavior where the CLI hangs and does not move forward. Our production environment has been offline for 12 hours now, which is generally considered to be a bad thing.Expected behavior
Pushing changes from environment to another should work or at least leave things in a revertible state..
Reproduction steps
Not sure if this can be reproduced in an empty directory, I have never experienced this level of amplify failing before.
Project Identifier
a7f88f1c8eb39da02933e54e978f3c1e
Log output
Additional information
Before submitting, please confirm:
The text was updated successfully, but these errors were encountered: