I am running few local (on same dedicated server as website) python scripts that scrape my website. It worked flawlessly until I added CloudFlare protection and now it stops this python script from scraping site using requests. This is how I scrape the website:
class AppURLopener(urllib.request.FancyURLopener): version = "Mozilla/5.0" opener = AppURLopener() ...... with opener.open(url) as response: data = response.read() text = data.decode('utf-8') bs = BeautifulSoup(text, features="html.parser")
How can I configure CloudFlare in such way that it allows parsing from this Python script without harming my website’s security? We’ve been hit with a pretty bad botnet level7 attack so it’s important for me that I don’t lose any security that CloudFlare provides. Bare in mind that this script is only run locally from the same dedicated server where the website is hosted and it scrapes only that website.
I am using CloudFlare’s free plan.
Thanks in advance!