azurerm_virtual_network_peering are failing with error ReferencedResourceNotProvisioned

[culpinnis]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.6.5

AzureRM Provider Version

3.80.0

Affected Resource(s)/Data Source(s)

azurerm_virtual_network_peering

Terraform Configuration Files

We have developed a minimal example, but because the error seems to be a run condition in the resource manager API, we were not able to reproduce it yet (the error only happens rarely). 

https://futureconsultant-my.sharepoint.com/:u:/g/personal/culpinnis_thinkport_digital/EWHTm7M83_tOsPN8l7ZBWu4B15blBbFTt8pAY4qwRktd5g?e=1MWWbA

Debug Output/Panic Output

We can not yet provide debug output, because the error is thrown by a production system that usually does not output debug information: 

https://gist.github.com/culpinnis/ac99845aaab3e32f6a57cc4d4c100ada

Expected Behaviour

The vnet peerings should be established.

Actual Behaviour

The peerings are failing randomly with error network.VirtualNetworkPeeringsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ReferencedResourceNotProvisioned"

Steps to Reproduce

See readme file of attached code.

Important Factoids

One of the peerings is established between different subscriptions of the same organization.

References

The error was first mentioned in #2605 and fixed by #3392.

In 42721b5 the code was refactored. Afterwards the error started to appear again.
Maybe the condition in

terraform-provider-azurerm/internal/services/network/virtual_network_peering_resource.go

Lines 148 to 150 in 12770f6

	if resp := future.Response(); resp != nil && response.WasBadRequest(resp) && strings.Contains(err.Error(), "ReferencedResourceNotProvisioned") {
	// Resource is not yet ready, this may be the case if the Vnet was just created or another peering was just initiated.
	return future.Response(), "Pending", err

is not checked in all circumstances.