Enabling data trace on aws_apigatewayv2_stage prevent endpoint to be reached with 429 too many request error

[rlapray]

Terraform CLI and Terraform AWS Provider Version

• Terraform v0.12.29
• provider.aws v3.2.0
• provider.template v2.1.2

Affected Resource(s)

• aws_apigatewayv2_stage => default_route_settings

Terraform Configuration Files

resource "aws_apigatewayv2_api" "default" {
  name          = "webscoket api name"
  protocol_type = "WEBSOCKET"
  route_selection_expression = "$request.body.action"

}


resource "aws_apigatewayv2_stage" "default" {
  api_id = aws_apigatewayv2_api.default.id
  name   = "ws"
  auto_deploy = true

  default_route_settings {
    data_trace_enabled       = true
    detailed_metrics_enabled = false
    logging_level            = "INFO"
  }
}

Expected Behavior

When not setting or setting to null throttling_burst_limit and throttling_rate_limit in default_route_settings, I'm expecting these properties to be null.
When theses properties are set to null, there is no throttling of any kind on my websocket api gateway.

Actual Behavior

throttling_burst_limit and throttling_rate_limit are set to 0 instead of null, and the throttling even prevent me to connect to my websocket endpoint, with a 429 (too many request) error.

Steps to Reproduce

1. I create the api gateway, a lambda and the stage without the default_route_settings. The planner produce the following plan :

# module.api_gateway_websocket.aws_apigatewayv2_stage.default will be created
  + resource "aws_apigatewayv2_stage" "default" {
      + api_id        = (known after apply)
      + arn           = (known after apply)
      + auto_deploy   = true
      + deployment_id = (known after apply)
      + execution_arn = (known after apply)
      + id            = (known after apply)
      + invoke_url    = (known after apply)
      + name          = "ws"
    }

2. I can connect to my websocket.
3. I add a default_route_settings to my stage:

resource "aws_apigatewayv2_stage" "default" {
  api_id = aws_apigatewayv2_api.default.id
  name   = "ws"
  auto_deploy = true

  default_route_settings {
    data_trace_enabled       = true
    detailed_metrics_enabled = false
    logging_level            = "INFO"
  }

  lifecycle {
    ignore_changes = [ deployment_id ]
  }
}

4. The planner produce this output :

# module.api_gateway_websocket.aws_apigatewayv2_stage.default will be updated in-place
  ~ resource "aws_apigatewayv2_stage" "default" {
        api_id          = "9297xlzr5h"
        arn             = "arn:aws:apigateway:eu-west-3::/apis/9297xlzr5h/stages/ws"
        auto_deploy     = true
        deployment_id   = "z1jxj8"
        execution_arn   = "arn:aws:execute-api:eu-west-3:665636607204:9297xlzr5h/ws"
        id              = "ws"
        invoke_url      = "wss://9297xlzr5h.execute-api.eu-west-3.amazonaws.com/ws"
        name            = "ws"
        stage_variables = {}

      ~ default_route_settings {
          ~ data_trace_enabled       = false -> true
            detailed_metrics_enabled = false
          ~ logging_level            = "OFF" -> "INFO"
            throttling_burst_limit   = 0
            throttling_rate_limit    = 0
        }
    }

5. If instead I specify a null value for throttling_burst_limit and throttling_rate_limit the planner output is exactly the same :

 # module.api_gateway_websocket.aws_apigatewayv2_stage.default will be updated in-place
  ~ resource "aws_apigatewayv2_stage" "default" {
        api_id          = "9297xlzr5h"
        arn             = "arn:aws:apigateway:eu-west-3::/apis/9297xlzr5h/stages/ws"
        auto_deploy     = true
        deployment_id   = "z1jxj8"
        execution_arn   = "arn:aws:execute-api:eu-west-3:665636607204:9297xlzr5h/ws"
        id              = "ws"
        invoke_url      = "wss://9297xlzr5h.execute-api.eu-west-3.amazonaws.com/ws"
        name            = "ws"
        stage_variables = {}

      ~ default_route_settings {
          ~ data_trace_enabled       = false -> true
            detailed_metrics_enabled = false
          ~ logging_level            = "OFF" -> "INFO"
            throttling_burst_limit   = 0
            throttling_rate_limit    = 0
        }
    }

6. I apply the change anyway
7. wscat -c wss://**********.execute-api.eu-west-3.amazonaws.com/ws => error: Unexpected server response: 429

Important Factoids

If I just want logs on my websocket api gateway, the mere fact of creating a default_route_settings and set data_trace_enabled to true in my stage prevent me to access my websocket api gateway with a 429 error.

[ewbankkit]

@rlapray Thanks for raising this issue.
Due to current limitations in the way that providers map to Terraform core it is not possible to distinguish between 0 values and null values in those throttling limit fields.
Are you aware of any scenarios where a practitioner would validly set a 0 value in those fields?
My proposed solution would ignore any 0 (or equivalently null) values and hence would not explicitly set to zero in the AWS API.

[ewbankkit]

Interestingly, once you select Enable throttling on an API stage in the console and save and next deselect that check box and save then the console attempt to send values of -1 for the throttling limits and you get a Invalid throttling setting error.
So there is no way to disable throttling right now 😄.

[rlapray]

@ewbankkit First thank you for your interest in this bug !

Are you aware of any scenarios where a practitioner would validly set a 0 value in those fields?
My proposed solution would ignore any 0 (or equivalently null) values and hence would not explicitly set to zero in the AWS API.

Yes, in fact i can think of one : as a practitioner i would want to stop any incoming request to my stage endpoints in a few seconds with a simple fix like throttling to 0 accepted requests per second. It's not conventional, but brutally effective and really quick to setup 😄

Interestingly, once you select Enable throttling on an API stage in the console and save and next deselect that check box and save then the console attempt to send values of -1 for the throttling limits and you get a Invalid throttling setting error.
So there is no way to disable throttling right now 😄.

Yes absolutely ! You have to destroy the stage and recreate it correctly, i didn't find any other way either !

As for a solution, and as you mentioned, -1 is not a valid value for any throttling scenario. Would it be consistent to use -1 to force a real null on these fields ?

[rlapray]

Hello again ! Does any prioritization had been done around this issue ? As stated before, setting a default route settings will axe the websocket silently.

[DeltaByte]

Couple of notes to add to this;

1. This isn't specific to websocket APIs, it happens to the HTTP ones too.

2. The same bug can be triggered if you manually change the throttling limits in the console from undefined. Since terraform will try to set the values back to null it ends up actually setting the value to zero, disabling your entire API 😅

  # aws_apigatewayv2_stage.v1 will be updated in-place
  ~ resource "aws_apigatewayv2_stage" "v1" {
       // omitted state

      ~ default_route_settings {
            data_trace_enabled       = false
            detailed_metrics_enabled = false
          - throttling_burst_limit   = 100 -> null
          - throttling_rate_limit    = 100 -> null
        }
    }

For what it's worth, the API Docs don't seem to actually mention how to handle this, apart from being signed int/floats.

[ewbankkit]

There is currently no way to disable throttling:

[spidermonk]

Is there any update on the status of this item? I've resorted to setting limits to the account max (5,000 & 10,000) to get around it for now.

[cojack]

Basically guys you need to validate your schema for null values there, somehow casting to int makes 0 from null

[dguo]

If the AWS API is a blocker, would it be reasonable to add a warning to the Terraform docs in the meantime?

[StackRef]

FWIW, I hit this issue as well and approached AWS support on the issue. Their response:

I tried creating an API and checking this behaviour from my end and initially the Default route throttling is not configured, but once we setup the same, then we can only enter a value which can be ''0'' or the required limit as per the availability. The value cannot be set back to ''null''.

To check and confirm this further, I contacted the internal escalations team and got to know that this is the expected behaviour as of now. I understand that the lack of feature to achieve this might be causing inconvenience at your end. Therefore, I have shared this as a feedback with our Internal Service Team and they will file a feature request for the same on your behalf. Unfortunately, we wouldn't be able to provide any ETA on the same, as to when this feature will be available.

More info: https://stackref.substack.com/p/psa-aws-api-gateway-429-errors-and?s=r

[globus68]

Hello!
I also stumbled upon this one. Additional warning: The issue with "0" throttling seems always to occur when applying changes to aws_apigateway2_stage after you one way or another have enabled detailed metrics, eg.:

1. Apply aws_apigateway2_stage without default_route_settings->detailed_metrics_enabled. Everything works as expected: API is available
2. Enable detailed metrics in AWS Console. Also everything works as expected after this
3. Do a small change in aws_apigateway2_stage, eg. change the name or add/change access_log_settings->format. Apply the script. Now, throttling is set to "0" and API is not reachable with 429.

Workaround to avoid unexpected unreachable API is to add add the default setting for throttling as mentioned by @spidermonk

[dekimsey]

Interestingly, the Lambda (functions?) UI shows a warning when throttling rates are set to 0 but API Gateway does not. I've made a request of our TAM to support that in addition to raising the issue discussed in here.

My ask was that:
The API Gateway API should allow resetting the throttle to null, referencing this ticket.
The API Gateway UI should alert an operator of this (value = 0) suspicious configuration as is done elsewhere.

[rdepping]

I stumbled across this after raising the comment at #30373 (comment)

On the TF side - it feels wrong that if you set a value like detailed_metrics_enabled all the other settings under default_route_settings are getting set for "free" to some default - including the rate limits going to 0 and blocking all traffic to the APIGW.

re:

terraform-provider-aws/internal/service/apigatewayv2/stage.go

Line 537 in 7a2014a

"data_trace_enabled": aws.BoolValue(routeSettings.DataTraceEnabled),