Getting on the verified bot list

General
1

I work for a company which operates software that crawls clients’ websites for accessibility issues, broken links, etc. We’ve had reports from some clients that we are not able to crawl Cloudflare sites and would therefore like to be added to your list of verified bots.

However, we can see in your documentation that we are only allowed to request one page per second. That’s fine for us but I want to ask if the one second limit also applies to any assets on the page? So, if there for example are 200 assets on a page we would need 200 seconds to crawl it properly.

2

You can submit your request here:

I believe that you would need 200 seconds to crawl, as it is 1 request per second. So if you are making 200 separate calls, then they would count as separate requests.

3

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.