steps to reproduce:
I start Tor Browser (under linux).
Then I start Mozilla Thunderbird that is set to use socks5 proxy on port 9150 for me (torified Thunderbird).
I try subscribe to an RSS feed that is located on a site that uses Cloudflare, such as this
Thunderbird says “this feed is not autorised”. This is true for any RSS feed on any site that uses Cloudflare (including my own).
If I change Thunderbird settings to “no proxy”, I can subscribe to the feed. So the problem is not in Thunderbird setup, it is somehow related to how Cloudflare handles Tor traffic.
It is the same problem with torified curl
curl https://navalny.com/blog/post.rss --socks5-hostname 127.0.0.1:9150 -v -O
< HTTP/2 403
< date: Fri, 20 May 2022 09:27:53 GMT
< content-type: text/html; charset=UTF-8
< cf-chl-bypass: 1
You’ll have to contact the site owner as it is their security settings, Cloudflare can’t help you with this issue or overrule their security settings.
Which exactly security settings should be applied? My own site has the same problem, I want to try it on my own site first.
The problem seems to be intermittent. Once in, say, 50 requests it works as expected.
This happened just now - both curl and thunderbird work. But then again it returns 403.
On my own site I can look in the activity logs.
It logs this request:
IP address 2a03:4000:6:36b7::4
Rule ID badscore
Action taken Managed Challenge
this is what curl shows as 403 and Thunderbird as Not authorised
There should be more in the activity log but at a glance, the IP’s reputation is bad and therefore the configured Security Level is blocking it.
The IP logged is a Tor network exit node. This is expected.
The problem is when I paste exactly the same URL in Tor Browser,
it always returns the correct page, there is no challenge no matter what the exit node could be.
It is possible to copy all the headers Tor Browser sends with this request. Adding a parameter
to route this request through Tor we get the following
curl 'https://navalny.com/blog/post.rss' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1' -H 'TE: trailers' --socks5-hostname 127.0.0.1:9150 -v -O
But it still returns 403 (Cloudflare challenge) although it is exactly the same request that Tor Browser sends.
It seems to me that Tor Browser has some Cloudflare integration that allows its users to avoid the challenge. To confirm this I tried to search Tor Browser code to find any references to Cloudflare. Although some were found it seems to be there is no integration.
That leaves me puzzled.
Cloudflare’s integration with the Tor network is documented here: https://blog.cloudflare.com/cloudflare-onion-service/
Request inspector in Tor Browser shows a subtle difference between the command for curl it generates and what it actually sends. This is the difference:
Accept-Encoding: gzip, deflate, br
So adding this header to curl command line
-H 'Accept-Encoding: gzip, deflate, br'
this makes up the following command
curl 'https://navalny.com/blog/post.rss' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1' -H 'TE: trailers' -H 'Accept-Encoding: gzip, deflate, br' --socks5-hostname 127.0.0.1:9150 -v -O
which works as expected!
“br” parameter in Accept-Encoding seems to make the difference and curl utility seems to have all the necessary support for this scenario. So, the problem with curl is solved, but with Thunderbird it persists. However, we can notify its developers what the solution could be.
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.