cloud: Allow Terraform Cloud/Enterprise to apply backpressure to intermediate state snapshots

[apparentlymart]

NOTE: The scope of this PR changed during discussion; see the commentary below to learn about the expanded scope.

The remaining text is the original PR writeup from before the scope increase.

---

We've seen some concern about the additional storage usage in Terraform Enterprise implied by creating intermediate state snapshots for particularly long apply phases that can arise when managing a large number of resource instances together in a single workspace.

This is an initial coarse approach to solving that concern, just restoring the original behavior when running inside Terraform Cloud or Enterprise for now and not creating intermediate snapshots at all.

This is here as a solution of last resort in case we do not find a better compromise before the v1.5.0 final release. Hopefully a future changeset will implement a more subtle take on this which still gets some of the benefits when running in a Terraform Enterprise environment but in a way that will hopefully be less concerning for Terraform Enterprise administrators, in which case the effects of this PR will not make it into any stable release.

This does not affect any other state storage implementation except the Terraform Cloud integration and the "remote" backend's state storage when running inside a TFC/TFE-driven remote execution environment.

[apparentlymart]

The two additional commits just added change the rule for the Terraform Cloud integration so that it will still persist state snapshots like any other state storage by default, but will allow the server to apply dynamic backpressure by setting a special HTTP header in its response to Create New State Version.

If the server requests an interval shorter than the default one (currently 20 seconds) then the default one will take priority. If the server requests a longer interval then that will take priority, saturating at 1 hour because a delay of longer than that isn't very reasonable and probably suggests a bug.

This doesn't change the rules for the "remote" backend because using that backend for Terraform Cloud is now deprecated anyway, and for newer versions of Terraform CLI the remote backend would never be used in a remote run in Terraform Cloud anyway. For older versions of Terraform CLI that did require the remote backend for Terraform Cloud, they would not yet have had the possibility of persisting intermediate snapshots.

[apparentlymart]

For the moment this is depending on an unreleased commit from github.com/hashicorp/go-tfe. We'll need to merge hashicorp/go-tfe#689 before moving forward with this, then make a new go-tfe release, and then finally change go.mod and go.sum in this PR to refer to the real release instead.

I'll leave this in draft status until we have a real go-tfe release to depend on, but otherwise I'd love for folks to take a look at this and flag any concerns so that I can respond to them concurrently with the review and release process on go-tfe.

I intend this to be complementary to rather than instead of other efforts to improve Terraform Cloud's handling of intermediate state snapshots. This change is focused on giving Terraform Cloud/Enterprise a way to protect themselves from a barrage of incoming requests, but doesn't do anything to address other concerns such as the unfortunate way Terraform Cloud presents multiple state versions per run in its UI today.

[brandonc]

@apparentlymart One thing that jumps out at me is that the platform assumes that the last created state version is the current workspace state version (and each get parsed for outputs and workspace resources, which become workspace artifacts)

It feels like we need the Persister API to give us the hint that the state is finalized or still pending.

[apparentlymart]

Hi @brandonc,

I'm not sure I see a situation where the server-side state storage should try to distinguish between an intermediate state snapshot and a final one.

What makes a final snapshot final is only that it was the most recent one. If Terraform CLI writes an intermediate snapshot and then crashes, that intermediate snapshot is the final snapshot for that run, and so I don't think it should be treated any differently than the final snapshot created after a run successfully completes. In either case the latest snapshot is the most accurate description we have of the infrastructure that exists, and so should always replace any snapshots created by earlier runs.

Does that make sense to you too, or am I missing something?

[brandonc]

@apparentlymart It does make sense, yes. My concern, though, is that we had aimed at not engaging certain Cloud services (state parsing mainly, which produce resource and output artifacts) until the last snapshot in the operation was completed. I suppose we could engage that stuff after x-terraform-snapshot-interval + timeout seconds after the last persistence... But it would be nice to be able to get a hint that the operation was completed so we could process the state sooner.

[apparentlymart]

Hmm, interesting!

I suppose I had assumed that the completion of the run in Terraform Cloud would be the trigger for that sort of thing.

By which I mean:

• Whenever the "apply" step for a run reaches a terminal status -- whether success or error -- trigger whatever analysis needs to happen against the most recently-written snapshot. Terraform Cloud owns that state machine, so it should not need Terraform CLI's help to detect that event.
• If a "Create State Version" requests arrives from outside of the context of a Terraform Cloud run, then we have no "framing context" for collecting multiple state snapshots together, so we'll need to assume all such snapshots are final and trigger the analysis immediately.

It seems to me that we need to make sure that we always eventually do the async analysis, even if Terraform CLI crashes in the middle of a run, because otherwise that would leave the Terraform Cloud metadata out of sync with the real latest state snapshot, right?

So what I described above is just an optimization to allow us to delay doing it when we're in control of the execution environment, because in that case we can be guaranteed that even if Terraform CLI or the Terraform Cloud Agent crashes the platform will still have an opportunity to run some final steps once the apply step is complete, and therefore we don't need to be so conservative in processing every intermediate snapshot just in case the CLI or agent crashes.

[sebasslash]

Correct, the tricky issue here is workspaces in local execution mode. As you stated, TFC has no "framing context" in this case so I was trying to figure out a way we could minimize the number of statefiles we'd need to store & process. It seems one potential way to do it is only process the latest state version (one created outside of a TFC run) when a workspace is unlocked. Although I feel like this might be "too late" and be a detriment to the local execution experience.

It would've really simplified things if the StateManager had context about the status of the current apply operation that was modifying state. That way we have a source of truth in order to tell TFC "hey we aren't going to create any more state versions"

[apparentlymart]

I agree that it would be better to avoid processing the intermediate snapshots when possible, but from what I can see it doesn't seem to be possible in all cases: if Terraform CLI crashes before it produces the final snapshot then we need to somehow retroactively decide that the most recent intermediate snapshot was the final one, because otherwise the system will become inconsistent.

I hadn't previously considered using the locking mechanism for the "run framing", but that does seem like an alternative worth investigating: even in local mode Terraform CLI will hold the workspace lock throughout the apply phase, so perhaps the rule would be that any new snapshots created while the lock is held don't trigger any analysis, but then once the lock is released we then immediately trigger the analysis of whatever is the latest snapshot at that point, if the latest snapshot wasn't already analyzed. Unless I'm missing something, that would achieve the needed behavior for both local and remote runs, because both of them involve the workspace lock.

(This does mean that if Terraform CLI crashes while holding the lock for a local operation then the workspace will stay locked and the analysis won't immediately run, but presumably it won't be too long before someone manually unlocks the workspace to continue working, and the analysis would begin at that point.)

[apparentlymart]

I've rebased this to include the merged version of the go-tfe change, but that isn't included in a go-tfe release yet so I cannot finalize this PR to refer to a released version.

[alisdair]

go-tfe v1.24.0 was just released.

[sebasslash]

🚀 I'll go ahead and add further refinements on toggling the setting in a new PR

[github-actions]

Reminder for the merging maintainer: if this is a user-visible change, please update the changelog on the appropriate release branch.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.