How to automatedly request RSS feeds protected by Cloudflare?

I run a small website (http://ooh.directory) that lists blogs and, so that I can link to those blogs’ most recent posts, I have code that regularly requests their RSS/Atom feeds.

This works fine except for sites whose feeds are protected by Cloudflare, including all of those hosted by TypePad.com (as of about six months ago), for which I get 403 Forbidden errors.

I’ve read about the Verified Bots Policy and applied months ago, but I don’t make the minimum amount of traffic (1000 requests per day).

I don’t know what else I can do to automatedly fetch these feeds. I don’t spam them – at most I request any individual feed once per hour. I can view them if I request the URLs in a browser.

Some example feeds that generate 403s and have server: cloudflare in their headers:

They may not be aware that they have a firewall setting that prevents their RSS feeds from being reachable. You’d have to reach out to them to let them know.

Thanks, but I created a TypePad blog and can’t see any settings related to this.

1 Like

You can go to Security → Events to find the reason why your request was blocked.

How to lift the restriction depends on what product imposed it.

Three of those hostnames point straight at Typepad, so it wouldn’t be in the end user’s firewall events log. The only one with a chance of seeing these would be PogoWasRight. But given the fact that the others are blocked, it sure sounds like it’s due to Typepad’s firewall settings.

2 Likes