Custom bot getting 403 from cloudflare

Hello,

I’m the founder of a software which is officially going public this week that browses websites to look for specific content. It which relies on a bot identified by a fixed user agent created for this purpose. The bot identifies and respects robots.txt. CloudFlare is blocking some of my requests and is returning a 403 Forbidden.

I’m currently capping the number of requests per second to 4 for the entire agent, which browses ~3 websites at once, so on average I’m rarely browsing more than 1 page per second per website.

I also think this is partially IP related, as I can see lower rejections when running it from my laptop when compared to running it in AWS. If I remove the user agent, I don’t get rejected requests at all - which is weird since using a user agent is somewhat a need to be compliant with the “bot rules”.

Does anyone have any idea on how I could proceed? Is there a way to contact Cloudflare to whitelist a specific user agent?

Best Regards,
Diogo {redacted}

It would help to see which setting is blocking your bot. That would show up in the site owner’s Firewall Events Activity Log.

You’d have to apply to get your bot on the Known Bots list:

1 Like

Hey!

Thanks for the quick reply!

That would be hard for me to find out :sweat_smile: though I might reach out to them eventually.

Thank you! That was exactly what I was looking for.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.