tfexec: Enable graceful (SIGINT-based) cancellation

[radeksimko]

Closes #389

Changelog draft

• Context cancellation on Unix systems will now send Terraform process SIGINT instead of killing it (which is otherwise default os/exec behaviour)
  • You can change the default 60s WaitDelay via SetWaitDelay(time.Duration)
• error type returned from individual commands now implements Unwrap making it possible to pass it into errors.As and access lower-level error such as exec.ExitError

[dbanck]

🚀

[wjf3121]

Hi @radeksimko thanks for your effort to address this issue. However, I think this PR won't work as expected:

writeOutput uses the context spawned from the command context, which means when the command is canceled, writeOutput will in turn be canceled leading to signal: broken pipe. This will cause the Terraform command to immediately exit without waiting for the graceful shutdown.

We found this issue when making our own implementation in #454. Hopefully it can be addressed in the main branch so we can switch to the official version again.

Another difference between this PR and #454 is that we pass Setpgid: true to unix cmd, which currently is only passed in the Linux implementation. I don't recall exactly why we do this but I remember there were issues when we needed to force kill the command when graceful shutdown timed out.

[radeksimko]

@wjf3121 I would be more than happy to address any outstanding issues if you can provide a way of reliably reproducing it, preferably in a form of a test.

That applies to both signal: broken pipe and Setpgid: true.

As far as I can tell, the linked PR #454 does not contain test for either of these issues so it is hard to understand the nature of those problems and confirm that the code actually solves them. Correct me if I'm wrong.

[dastrobu]

I am a bit late to the party, but we see the broken pipe issue as well in our project. I picked up the suggestion to create a test case for this scenario. You can find it here: #523.

It would be great to get feedback or suggestions on how to fix the code, as the PR currently just contains the test case. Feedback is welcome.

[busser]

PR #527 provides a test and a fix, if anyone wants to have a look :)