Access by RSS Readers

I develop Unread, an RSS reader for iOS. I am writing because it looks like Cloudflare blocks my crawlers from retrieving feeds from some websites. For example, there is a feed at:

https://www.addictivetips.com/feed/

Below is the curl output I see when retrieving that feed.

It looks like Cloudflare blocks the request to ensure that it being made by a human, but RSS feeds are intended for programmatic access. If Cloudflare cannot distinguish feeds from pages intended to be read by humans, is there a way my servers could be added to some kind of allow list?

Thank you.

John


$ curl -i https://www.addictivetips.com/feed/
HTTP/2 403 
content-type: text/html;charset=utf-8
content-length: 537
server: CloudFront
date: Mon, 09 May 2022 12:54:29 GMT
x-datadome: protected
accept-ch: Sec-CH-UA,Sec-CH-UA-Mobile,Sec-CH-UA-Platform,Sec-CH-UA-Arch,Sec-CH-UA-Full-Version-List,Sec-CH-UA-Model,Sec-CH-Device-Memory, Sec-CH-UA,Sec-CH-UA-Mobile,Sec-CH-UA-Platform,Sec-CH-UA-Arch,Sec-CH-UA-Full-Version-List,Sec-CH-UA-Model,Sec-CH-Device-Memory
charset: utf-8
cache-control: max-age=0, private, no-cache, no-store, must-revalidate
pragma: no-cache
x-datadome-cid: AHrlqAAAAAMAxEisPUTgO4UASXcsgA==
set-cookie: datadome=.2MwtB0W2URyxGDsjwqT7ZiizbArGCWvtWMIHZmlMaei2F-rN3ZF~ekz-I2Xg~k.FG01q.6XNeX6vDKtEqLAMfIE.qp2wJi83EHI8.pegLmI_KqhiX~K6R1cuKq_.X7_; Max-Age=31536000; Domain=.addictivetips.com; Path=/; Secure; SameSite=Lax
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 80246b01173b7304bbc7804bfe173f32.cloudfront.net (CloudFront)
x-amz-cf-pop: BOS50-C2
x-amz-cf-id: u-V386Zcxb-UUvk84_eGMCqNH6mdlePiIbkQiAeIf6amajh6xw4bqA==

<html><head><title>addictivetips.com</title><style>#cmsg{animation: A 1.5s;}@keyframes A{0%{opacity:0;}99%{opacity:0;}100%{opacity:1;}}</style></head><body style="margin:0"><p id="cmsg">Please enable JS and disable any ad blocker</p><script>var dd={'cid':'AHrlqAAAAAMAxEisPUTgO4UASXcsgA==','hsh':'213572971491A8E4BBCAD9259C278F','t':'bv','s':35922,'e':'0cc04a75710bd2d25abc8163e8af27f7b4ca7ab144463740d56b92fad89df8dc','host':'geo.captcha-delivery.com'}</script><script src="https://ct.captcha-delivery.com/c.js"></script></body></html>

This isn’t Cloudflare, this is AWS CloudFront.

1 Like

Oh! Sorry. Thanks!