Sporadic 401 errors when using terraform to update a zone with more than 800 records

olivier.schiavo · September 23, 2020, 11:56am

Hello,
We are using terraform to manage a new zone, everything went very well, but now we have 816 records created, and every time we want to update the state terraform spends several minutes checking the state, and then chokes on:

cloudflare_record.vnewgorapreprod_cname["*.xxxxx-xxxcom"]: Refreshing state... [id=2476fb86706e18d2a3d5dd5240ee36e9]

cloudflare_record.vnewgorapreprod_cname[“xxxx.xxxxcom”]: Refreshing state… [id=66e2dd28c9134e617a7101aa3d26cfa7]
cloudflare_record.vnewgorapreprod_cname["*.xxxxx.coxxxxm"]: Refreshing state… [id=9335d9be8d0fa9320c76897bb5ded
ef5]

Error: error from makeRequest: HTTP status 401: Authentication error

Expected result:
Terraform should be able to successfully validate its state then apply the changes.

Actual result:
Terraform stops after 401 errors when calling api cloudflare com

In the TF_LOG we have a lot of successful

2020-09-23T13:05:41.925+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: —[ REQUEST ]---------------------------------------
2020-09-23T13:05:41.925+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: GET /client/v4/zones/xxxxxxxxx/dns_records/xxxxxxxxxxxxxxxxxxHTTP/1.1
2020-09-23T13:05:41.925+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: Host: api.cloudflare.com
2020-09-23T13:05:41.925+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: User-Agent: HashiCorp Terraform/0.12.26 (+https://www.terraform.io) Terraform Plugin SDK/1.15.0 terraform-provider-cloudflare/2.10.0
2020-09-23T13:05:41.925+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: Authorization: Bearer xxxxxxxxxxxxxx
2020-09-23T13:05:41.925+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: Content-Type: application/json
2020-09-23T13:05:41.925+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: Accept-Encoding: gzip
2020-09-23T13:05:41.925+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0:
2020-09-23T13:05:41.925+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0:
2020-09-23T13:05:41.925+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: -----------------------------------------------------
2020-09-23T13:05:42.030+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: 2020/09/23 13:05:42 [DEBUG] Cloudflare API Response Details:
2020-09-23T13:05:42.030+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: —[ RESPONSE ]--------------------------------------
2020-09-23T13:05:42.030+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: HTTP/1.1 200 OK
…

But then we have:

2020-09-23T13:05:42.175+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: —[ REQUEST ]---------------------------------------
2020-09-23T13:05:42.175+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: GET /client/v4/zones/xxxxx/dns_records/xxxxx HTTP/1.1
2020-09-23T13:05:42.175+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: Host: api.cloudflare.com
2020-09-23T13:05:42.175+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: User-Agent: HashiCorp Terraform/0.12.26 (+https://www.terraform.io) Terraform Plugin SDK/1.15.0 terraform-provider-cloudflare/2.10.0
2020-09-23T13:05:42.175+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: Authorization: Bearer xxxxxxx
2020-09-23T13:05:42.175+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: Content-Type: application/json
2020-09-23T13:05:42.175+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: Accept-Encoding: gzip
2020-09-23T13:05:42.175+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0:
2020-09-23T13:05:42.175+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0:
2020-09-23T13:05:42.175+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: -----------------------------------------------------
2020-09-23T13:05:42.424+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: 2020/09/23 13:05:42 [DEBUG] Cloudflare API Response Details:
2020-09-23T13:05:42.424+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: —[ RESPONSE ]--------------------------------------
2020-09-23T13:05:42.424+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: HTTP/1.1 401 Unauthorized
2020-09-23T13:05:42.424+0200 [DEBUG] plugin.terraform-provider-cloudflare_v2.10.0: Connection: close

When I do the failed request with cURL it works perfectly. So it is not a problem with the token. When I launch terraform several times it fails everytime but not on the same api requests.
When I configure CLOUDFLARE_RETRIES=10 the 401 errors are not retried.

So now I am a bit stuck. We can no more manage the zone with terraform. Is there a way to make terraform retry these errors ?

olivier.schiavo · September 23, 2020, 4:58pm

update: I was able to hack my way by editing cloudflare.go line 231:

                 if respErr != nil || resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500 || resp.StatusCode == http.StatusUnauthorized

the || resp.StatusCode == http.StatusUnauthorized does the trick but it does not look like an elegant solution, and I suspect that a PR on this will be rejected.

If someone from Cloudflare has a suggestion which would allow a much, much, lower rate of 401 error with valid api keys, it would be very welcome.