429 Status codes using crawlers

Website, Application, Performance Security
1

What is the name of the domain?

docenter.nl

What is the error number?

429

What is the error message?

HTTP Error 429: Too Many Requests

What is the issue you’re encountering

My crawlers are not correctly crawling all of my pages

What steps have you taken to resolve the issue?

I have tried to the disable the proxy, I tried lowering the tick rate and I tried allowlisting the IP addresses in Cloudflare

Was the site working with SSL prior to adding it to Cloudflare?

Yes

What is the current SSL/TLS setting?

Flexible

What are the steps to reproduce the issue?

Use Cloudflare

2

Hi there,

Check your origin logs, and likely you’ll see the 429s being issued there due to rate limiting.
Please allow Cloudflare IP addresses unmitigated access to your origin.

Create some type of allowlist at the origin and add Cloudflare IPs:

Take care.

3

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.