Cloudflare bot Crawl-delay setting, and timeout setting


#1

These 2 settings belong in the CloudFlare interface. They are unrelated of course.

1.) I use CloudFlare to protect my site against heavy bot traffic. But sometimes CloudFlare’s bot spiders my sites so aggressively it severely increases the server load (causing the problem I’m using CloudFlare to prevent!). I really wish there were a setting in my CloudFlare control panel so I could tell CloudFlare to spider slow/medium/fast. Sorta like a Crawl-delay setting. Or if CloudFlare prefers to do it differently, another approach could be to somehow let CloudFlare determine my server load so it knows how hard or softly it can crawl my site.

2.) Another setting that I wish I had in my CloudFlare interface was a timeout setting. Currently any requests time out at 100 seconds or something. But every now and then I run large commands in my server’s File Manager (etc.) that need more than 100 seconds to execute. I (the server administrator) would love to have the ability to run these commands again… which I lose when I come to CloudFlare (unless I turn CloudFlare off temporarily, which is a PIB). I know what you’re thinking… I should do it in a cronjob, but life doesn’t always work like that! :slight_smile: Anyway, a timeout setting would be nice, at least for my IP. Another approach could just be to give us a whitelist where IP’s on this list don’t get the timeout until something ridiculous like 15 minutes.

Dan

Dan