PSA: AWS API Gateway, 429 errors, and Terraform
AWS's API calls to API Gateway have a bit of a shortcoming if you change your mind about throttling protection.
We recently hit upon an unfortunate issue regarding the modification of an HTTP-based AWS API Gateway, one which resulted in 100% of API calls being rejected with 429 (“rate exceeded” or “too many requests”) errors.
Here’s the issue in a nutshell: if you set your API Gateway with throttling protection — burst limit, rate limit — and then think, “hey, we’re just in development now — let’s turn that off,” you’re out of luck.
It turns out there’s no way to turn it “off” — set to null
— once you’ve pulled that trigger. So now you think, “OK, can’t remove it altogether. So let’s make it zero and that should do it.” Does it turn throttling off? No — it turns it way on, as in rejecting all traffic!
Terraform tries — it really does! If you decide to remove it out of your Terraform configuration, it assumes to set it to null
, but AWS sees this and thinks, “oh you must mean 0
, not null
!” — and now your API is down.
# aws_apigatewayv2_stage.v1 will be updated in-place
~ resource "aws_apigatewayv2_stage" "v1" {
// omitted state
~ default_route_settings {
data_trace_enabled = false
detailed_metrics_enabled = false
- throttling_burst_limit = 100 -> null
- throttling_rate_limit = 100 -> null
}
}
I hit up AWS Support on this issue. Here’s their response:
I tried creating an API and checking this behaviour from my end and initially the Default route throttling is not configured, but once we setup the same, then we can only enter a value which can be ''0'' or the required limit as per the availability. The value cannot be set back to ''null''.
To check and confirm this further, I contacted the internal escalations team and got to know that this is the expected behaviour as of now. I understand that the lack of feature to achieve this might be causing inconvenience at your end. Therefore, I have shared this as a feedback with our Internal Service Team and they will file a feature request for the same on your behalf. Unfortunately, we wouldn't be able to provide any ETA on the same, as to when this feature will be available.
So to there you have it. While it’s not necessarily a shortcoming of Terraform or its AWS provider, there should be ways to mitigate this in another fashion, until AWS figures things out on the back end.
Good luck out there …
— K