SIGINT ("graceful shutdown") is unacceptably slow / doesn't always work

[jcmcken]

Terraform Version

Terraform v0.11.11

Terraform Configuration Files

Any AWS-based configuration will do. (Actually, any Terraform resource that reaches out to remote network endpoints would be fine).

Expected Behavior

Our environment requires talking through a corporate HTTP proxy to reach AWS API endpoints. If I fail to set this proxy and and issue a terraform apply, Terraform spins and spins, retrying the API calls, which is expected. However in this case I want to send a SIGINT via Ctrl + C to cancel the apply so I can set the proxy. I expect this graceful shutdown to be quick, since under no circumstance will the apply work, no matter how many times it retries.

Actual Behavior

When I Ctrl + C, however, Terraform claims that it's gracefully shutting down, but really doesn't seem to do anything. The 'apply' command continues for minutes and minutes retrying the resource (at some point I just give up and force quit, but maybe it times out after awhile).

Here's some output:

module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (9m30s elapsed)
^CInterrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
stopping operation...
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (9m40s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (9m50s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (10m0s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (10m10s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (10m20s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (10m30s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (10m40s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (10m50s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (11m0s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (11m10s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (11m20s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (11m30s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (11m40s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (11m50s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (12m0s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (12m10s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (12m20s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (12m30s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (12m40s elapsed)
module.postgresql-log-subscription.aws_cloudwatch_log_group.default: Still creating... (12m50s elapsed)

(This goes on for several more minutes before I Ctrl +C again and force quit)

Steps to Reproduce

Unsure of the exact steps, but the general steps are:

1. Configure your local machine to block outbound access to AWS.
2. Run a terraform apply using AWS resources.
3. Observe that Terraform retries the API calls over and over.
4. Hit Ctrl + C or otherwise send a SIGINT to the Terraform process.
5. Observe that Terraform doesn't actually gracefully shutdown.

Additional Context

The only non-standard setting is use of TF_PLUGIN_CACHE_DIR environment variable to set a global plugin cache in my home directory (so that each module doesn't need to re-download every module)

I don't expect this has any relevance to my problem.

[jbardin]

Hi @jcmcken,

Yes, the behavior here often seems to be ignoring the request to shut down. What is happening under the covers is that terraform has acknowledged the request, and will halt processing of all remaining resources, but it still needs to wait for pending requests.

Interrupting the actual resource operations is up to the individual providers, which means either the provider hasn't implemented the Stop request, or it can't interrupt the particular request.

While this will still be up to the providers in the future, I'll mark this as an sdk enhancement, since we will probably want to find ways to encourage and help ensure this is better implemented by providers in future versions.

[bflad]

The issue for implementing Stop in the Terraform AWS Provider can be found here (hashicorp/terraform-provider-aws#4177) although I think we were waiting for the Terraform Provider SDK improvements before doing a full implementation, at least looking at the first attempt (hashicorp/terraform-provider-aws#4179).

[jcmcken]

@jbardin @bflad Thanks for the explanations/references, that definitely makes more sense. Sounds like this is already on the radar

[jcmcken]

FYI, the terraform init command seems to have a similar problem. Sending SIGINT (e.g. when I forget to set the HTTP proxy, so there's no way it can connect) doesn't cancel the init. Instead I have to wait for an I/O timeout.

$ terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...


^C^C^C^C^C^C^C^C^C^C^C
Error installing provider "gitlab": Get https://releases.hashicorp.com/terraform-provider-gitlab/: dial tcp 151.101.201.183:443: i/o timeout.

Terraform analyses the configuration and state and automatically downloads
plugins for the providers used. However, when attempting to download this
plugin an unexpected error occured.

This may be caused if for some reason Terraform is unable to reach the
plugin repository. The repository may be unreachable if access is blocked
by a firewall.

If automatic installation is not possible or desirable in your environment,
you may alternatively manually install plugins by downloading a suitable
distribution package and placing the plugin's executable file in the
following directory:
    terraform.d/plugins/linux_amd64

[paultyng]

Should investigate this as part of #276

[radeksimko]

Hi everyone 👋
From my perspective this problem has 3 parts:

1. Cancellable terraform init - this is something to discuss in the context of Terraform core since provider discovery is done there, not in the SDK - I just opened Make terraform init cancellable terraform#23647
2. Allow retry functions to be cancellable, i.e. allow providers to pass context - this is tracked in Implement graceful shutdown for helper/resource waiter #73
3. Implementation of StopContext and Implement graceful shutdown for helper/resource waiter #73 (when it is exposed) by providers - this is up to each provider and needs to be coordinated individually. AWS has Integrate Provider Context For Cancelling Requests terraform-provider-aws#4177 and anyone is welcomed to open other issues in other providers.

With that in mind I'm going to close this issue and recommend subscribing to all or any of the above issues. If there's any other context I missed feel free to raise a separate issue for that context.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.