feat: consideration for retry v2 (#745)

* add 429 as an retryable status code

update

* update

* Update azapi_data_plane_resource.go

* update

* Update azapi_resource.go

* update

* Update azapi_data_plane_resource.go

* Update azapi_data_plane_resource.go

* Update azapi_data_plane_resource.go

* Update azapi_resource_action_ephemeral.go

* update

* Update azapi_resource.go

* Update azapi_data_plane_resource.go

* Update azapi_data_plane_resource.go

* Update common.go

* Update common.go

* feat: retryv2

* style: go lint

* test: remove old test

* feat: update default retry

* fix: incorrect doc defaults

* fix: make error messages optional

* docs: updates following review

* feat: add skip external request

* docs: remove duplicate sentence

* test: fix test and move skip package to correct location

* refactor: change name of provider retry attribute

* docs: update guide with new attr name

* Revert "test: fix test and move skip package to correct location"

This reverts commit 3be6445.

* Revert "feat: add skip external request"

This reverts commit b11892c.

* refactor: update retry based on feedback

* docs: make docs

* chore: lint

* feat: adopt agreed 2.3 retry

* test: update tests

* test: improve tests

* fix: make err message required

* vendor

* docs: docs

* chore: refactor client retry configure func due to linter

* fix: updates following PR review

---------

Co-authored-by: Harry Qu <harryqu@microsoft.com>

Author: matt-FFFFFF

docs/data-sources/resource.md

@@ -108,7 +108,7 @@ output "quarantine_policy" {
 	```
 
 To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
-- `retry` (Attributes) The retry block supports the following arguments: (see [below for nested schema](#nestedatt--retry))
+- `retry` (Attributes) The retry object supports the following attributes: (see [below for nested schema](#nestedatt--retry))
 - `timeouts` (Block, Optional) (see [below for nested schema](#nestedblock--timeouts))
 
 ### Read-Only
@@ -136,7 +136,7 @@ To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
 
 Required:
 
-- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the error is considered retryable.
+- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the request will be retried.
 
 Optional:
 

docs/data-sources/resource_action.md

@@ -87,7 +87,7 @@ data "azapi_resource_action" "example" {
 	```
 
 To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
-- `retry` (Attributes) The retry block supports the following arguments: (see [below for nested schema](#nestedatt--retry))
+- `retry` (Attributes) The retry object supports the following attributes: (see [below for nested schema](#nestedatt--retry))
 - `sensitive_response_export_values` (Dynamic) The attribute can accept either a list or a map.
 
 - **List**: A list of paths that need to be exported from the response body. Setting it to `["*"]` will export the full response body. Here's an example. If it sets to `["properties.loginServer", "properties.policies.quarantinePolicy.status"]`, it will set the following HCL object to the computed property sensitive_output.
@@ -154,7 +154,7 @@ To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
 
 Required:
 
-- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the error is considered retryable.
+- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the request will be retried.
 
 Optional:
 

docs/data-sources/resource_list.md

@@ -113,7 +113,7 @@ data "azapi_resource_list" "listSubnetsByVnet" {
 	```
 
 To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
-- `retry` (Attributes) The retry block supports the following arguments: (see [below for nested schema](#nestedatt--retry))
+- `retry` (Attributes) The retry object supports the following attributes: (see [below for nested schema](#nestedatt--retry))
 - `timeouts` (Block, Optional) (see [below for nested schema](#nestedblock--timeouts))
 
 ### Read-Only
@@ -138,7 +138,7 @@ To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
 
 Required:
 
-- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the error is considered retryable.
+- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the request will be retried.
 
 Optional:
 

docs/ephemeral-resources/resource_action.md

@@ -98,7 +98,7 @@ ephemeral "azapi_resource_action" "listKeys" {
 	```
 
 To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
-- `retry` (Attributes) The retry block supports the following arguments: (see [below for nested schema](#nestedatt--retry))
+- `retry` (Attributes) The retry object supports the following attributes: (see [below for nested schema](#nestedatt--retry))
 - `timeouts` (Block, Optional) (see [below for nested schema](#nestedblock--timeouts))
 
 ### Read-Only
@@ -123,7 +123,7 @@ To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
 
 Required:
 
-- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the error is considered retryable.
+- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the request will be retried.
 
 Optional:
 

docs/guides/feature_customized_retry.md

@@ -6,14 +6,64 @@ description: |-
 
 ---
 
-The AzAPI provider can digest the intermittent API errors and retry the requests based on the customized retry configuration. This feature is useful when you need to handle the API errors gracefully and improve the reliability of the Terraform deployments.
+## Why retry?
 
+Sometimes, when managing cloud infrastructure, requests to the cloud provider may fail due to transient issues such as network problems, timeouts, eventual consistency, or rate limiting. In these cases, it can be beneficial to retry the request a few times before giving up.
+
+The AzAPI provider can digest these intermittent API errors and retry the requests based on the customized retry configuration. This feature is useful when you need to handle the API errors gracefully and improve the reliability of the Terraform deployments.
 
 ## Prerequisites
 
-- [Terraform AzAPI provider](https://registry.terraform.io/providers/azure/azapi) version 2.1.0 or later
+- [Terraform AzAPI provider](https://registry.terraform.io/providers/azure/azapi) version 2.1.0 or later. Some features are only available from version 2.3.0.
+
+## Retry configuration
+
+There are two types of retry configurations available in the AzAPI provider:
+
+1. Provider retry configuration
+2. Resource-specific retry configuration
+
+### Provider retry configuration
+
+The provider retry configuration is a global configuration that applies to all resources managed by the provider. You can configure the provider retry behavior by setting the following provider values:
+
+- `maximum_busy_retry_attempts`
+
+This value controls the number of times the provider will retry a failed request. The default value is `3`.
+A retry will be triggered if the request fails with HTTP 408, 429, 500, 502, 503, or 504.
+
+In the case that the response header contains a `Retry-After` value, the provider will wait for the specified duration before retrying the request.
+
+### Resource-specific retry configuration
+
+In addition to the provider retry configuration, you can also configure the retry behavior for individual resources. This allows you to fine-tune the retry behavior for specific resources.
 
-## Customized Retry for Resource Creation
+The resource-specific retry comes after the provider retry, that is to say that the provider retry will be attempted first, and if it fails or exceeds the maximum attempts, the resource-specific retry will be attempted.
+Note that the resource-specific retry does not honour the `Retry-After` header and is exponential backoff based.
+
+Resource specific retry is configured using the `retry` attribute.
+
+If you configure a retry configuration, the maximum elapsed time for the retry will be set to the resource's timeout value for that operation (create, update, read, delete).
+
+With `azapi_resource` and `azapi_data_plane_resource`, the provider performs a read operation after the resource has been created so that we can store the read-only values.
+
+The schema of these retry attributes is as follows:
+
+- `error_message_regex` - A list of regular expressions to match against error messages. If any of the regular expressions match, the request will be retried.
+- `interval_seconds` - The initial number of seconds to wait before the 1st retry. The default value is `10`.
+- `max_interval_seconds` - The maximum number of seconds to wait before retrying a request. The default value is `180`.
+- `multiplier` - The multiplier to apply to the interval between retries. The default value is `1.5`.
+- `randomization_factor` - The randomization factor to apply to the interval between retries. The default value is `0.5`. The formula for the randomized interval is: `RetryInterval * (random value in range [1 - RandomizationFactor, 1 + RandomizationFactor])`. Set to zero `0.0` for no randomization.
+
+## Default resource-specific retry configuration
+
+If you do not configure any retry values, the provider will use the following:
+
+For the initial create/read/update/delete operation we will only retry on the provider's `maximum_busy_retry_attempts` value.
+
+For the read-after-create, the provider will retry on HTTP 404 and 403 status codes up to the operation timeout. This is logical as, if we have just successfully created the resource, we should not be getting a 404 or 403 on any subsequent GET request.
+
+## Example - Customized Retry for Resource Creation
 
 The virtual network link resource may not be available immediately after the virtual network is created. In this case, you can configure the customized retry configuration to handle the `ResourceNotFound` error and retry the request.
 
@@ -41,9 +91,9 @@ resource "azapi_resource" "privateDnsZoneLinkBlob" {
 }
 ```
 
-Above configuration is only used for demonstration purposes. From the `2.0.1` version, the AzAPI provider will automatically retry the GET requests when the `ResourceNotFound` error occurs after the resource creation. 
+Above configuration is only used for demonstration purposes. From the `2.0.1` version, the AzAPI provider will automatically retry the GET requests when the `ResourceNotFound` error occurs after the resource creation.
 
-## Customized Retry for Resource Deletion
+## Example - Customized Retry for Resource Deletion
 
 The private DNS zone may not be deleted immediately after the nested virtual network link is deleted. In this case, you can configure the customized retry configuration to handle the `CannotDeleteResource` error and retry the request.
 
@@ -62,4 +112,3 @@ resource "azapi_resource" "privateDnsZoneQueue" {
   }
 }
 ```
-

docs/index.md

@@ -72,6 +72,7 @@ provider "azapi" {
 - `enable_preflight` (Boolean) Enable Preflight Validation. The default is false. When set to true, the provider will use Preflight to do static validation before really deploying a new resource. When set to false, the provider will disable this validation. This can also be sourced from the `ARM_ENABLE_PREFLIGHT` Environment Variable.
 - `endpoint` (Attributes List) The Azure API Endpoint Configuration. (see [below for nested schema](#nestedatt--endpoint))
 - `environment` (String) The Cloud Environment which should be used. Possible values are `public`, `usgovernment` and `china`. Defaults to `public`. This can also be sourced from the `ARM_ENVIRONMENT` Environment Variable.
+- `maximum_busy_retry_attempts` (Number) The maximum number of retries to attempt if the Azure API returns an HTTP 408, 429, 500, 502, 503, or 504 response. The default is `3`. The resource-specific retry configuration may additionally be used to retry on other errors and conditions.
 - `oidc_azure_service_connection_id` (String) The Azure Pipelines Service Connection ID to use for authentication. This can also be sourced from the `ARM_ADO_PIPELINE_SERVICE_CONNECTION_ID` or `ARM_OIDC_AZURE_SERVICE_CONNECTION_ID` Environment Variables.
 - `oidc_request_token` (String) The bearer token for the request to the OIDC provider. This can also be sourced from the `ARM_OIDC_REQUEST_TOKEN` or `ACTIONS_ID_TOKEN_REQUEST_TOKEN` Environment Variables.
 - `oidc_request_url` (String) The URL for the OIDC provider from which to request an ID token. This can also be sourced from the `ARM_OIDC_REQUEST_URL` or `ACTIONS_ID_TOKEN_REQUEST_URL` Environment Variables.

docs/resources/data_plane_resource.md

@@ -137,7 +137,7 @@ resource "azapi_data_plane_resource" "example" {
 	```
 
 To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
-- `retry` (Attributes) The retry block supports the following arguments: (see [below for nested schema](#nestedatt--retry))
+- `retry` (Attributes) The retry object supports the following attributes: (see [below for nested schema](#nestedatt--retry))
 - `timeouts` (Block, Optional) (see [below for nested schema](#nestedblock--timeouts))
 - `update_headers` (Map of String) A mapping of headers to be sent with the update request.
 - `update_query_parameters` (Map of List of String) A mapping of query parameters to be sent with the update request.
@@ -164,7 +164,7 @@ To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
 
 Required:
 
-- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the error is considered retryable.
+- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the request will be retried.
 
 Optional:
 

docs/resources/resource.md

@@ -162,7 +162,7 @@ resource "azapi_resource" "example" {
 	```
 
 To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
-- `retry` (Attributes) The retry block supports the following arguments: (see [below for nested schema](#nestedatt--retry))
+- `retry` (Attributes) The retry object supports the following attributes: (see [below for nested schema](#nestedatt--retry))
 - `schema_validation_enabled` (Boolean) Whether enabled the validation on `type` and `body` with embedded schema. Defaults to `true`.
 - `tags` (Map of String) A mapping of tags which should be assigned to the Azure resource.
 - `timeouts` (Block, Optional) (see [below for nested schema](#nestedblock--timeouts))
@@ -208,7 +208,7 @@ Read-Only:
 
 Required:
 
-- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the error is considered retryable.
+- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the request will be retried.
 
 Optional:
 

docs/resources/resource_action.md

@@ -107,7 +107,7 @@ description: |-
 	```
 
 To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
-- `retry` (Attributes) The retry block supports the following arguments: (see [below for nested schema](#nestedatt--retry))
+- `retry` (Attributes) The retry object supports the following attributes: (see [below for nested schema](#nestedatt--retry))
 - `sensitive_response_export_values` (Dynamic) The attribute can accept either a list or a map.
 
 - **List**: A list of paths that need to be exported from the response body. Setting it to `["*"]` will export the full response body. Here's an example. If it sets to `["properties.loginServer", "properties.policies.quarantinePolicy.status"]`, it will set the following HCL object to the computed property output.
@@ -175,7 +175,7 @@ To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
 
 Required:
 
-- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the error is considered retryable.
+- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the request will be retried.
 
 Optional:
 

docs/resources/update_resource.md

@@ -140,7 +140,7 @@ This resource can manage a subset of any existing Azure resource manager resourc
 	```
 
 To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
-- `retry` (Attributes) The retry block supports the following arguments: (see [below for nested schema](#nestedatt--retry))
+- `retry` (Attributes) The retry object supports the following attributes: (see [below for nested schema](#nestedatt--retry))
 - `timeouts` (Block, Optional) (see [below for nested schema](#nestedblock--timeouts))
 - `update_headers` (Map of String) A mapping of headers to be sent with the update request.
 - `update_query_parameters` (Map of List of String) A mapping of query parameters to be sent with the update request.
@@ -167,7 +167,7 @@ To learn more about JMESPath, visit [JMESPath](https://jmespath.org/).
 
 Required:
 
-- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the error is considered retryable.
+- `error_message_regex` (List of String) A list of regular expressions to match against error messages. If any of the regular expressions match, the request will be retried.
 
 Optional: