Stop Cloudflare from unauthorized crawling of my website

General
1

I have never used Cloudflare, but I am getting lots of hits from Cloudflare ips that are chewing up my resources and transferring my content in ways that I have not agreed to.

I have blocked a number of ip ranges, but this really isn’t the answer - Cloudflare should NOT be caching my content, neither should they be crawling my website.

How can I make this stop?

2

It sounds like user are using warp (like a VPN) to connect to your site. The only thing you can do is block the IPs.

3

I appreciate the reply - and I have blocked the ip ranges listed in the authoritative list. My issue is that companies in the networking and CDN spaces add new ip ranges regularly. Which means I have a regular todo - to check to see if the ranges have changed, or if a new range has been added.

I am certain that cloudflare gets regular orders from courts of competent jurisdiction to remove content and to ensure that the content isn’t re-introduced by a new customer - that is, they are required to never retrieve, store, or forward content from certain sites. So I am am equally certain that there is a mechanism to ‘blocklist’ certain sources.

I wish I could get added to that list without having to hire an attorney to accomplish that.

That said, with sufficient information I am able to automate the process –

whois -h whois.radb.net -- '-i origin AS32934' | grep ^route | sed -e 's/route.*: //' -e 's/ \+//'
yields a list of all of the ip ranges owned by facebook.

Can I get the requisite information for cloudflare, so that I may automate this? I haven’t been able to find it.

Thank you, again.

4

I think there is a misunderstanding. Cloudflare isn’t accessing your site as a CDN but as a VPN. It would be similar to asking companies such as NordVPN, PIA, ExpressVPN or TorGaurd to block connections to your site.
If you want a full list of Cloudflare’s IPs it is available here IP Ranges or with their API https://api.cloudflare.com/client/v4/ips

5

I hadn’t known to look for the api. This is analogous to the facebook solution and can be easily automated.

This is my solution:

curl 'htps://api.cloudflare.com/client/v4/ips' 2>/dev/null| sed -E -e ‘s/{“result”:{“ipv._cidrs”:[([^]]+)],“ipv._cidrs”:[([^]]+)./\1,\2,/;s/“([^”])",/\1\n/g’

Which will yield the current list of cloudflare ip ranges. Works for me…

Thank you!

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.