Service can't get my RSS feeds unless Cloudflare DNS Proxy is off

[user71413]

I use a service called micro.blog that periodically grabs the RSS feed from my website and posts the contents to the service’s timeline. For about a month or so, the service has been displaying an error of “invalid RSS feed”. I’ve been working with the folks at micro.blog to figure out what is causing this.

micro.blog says that when their bot goes to my site to get my RSS feed, instead of the RSS feed, they get an HTML page that says “Please wait while your request is being verified”.

Doing some searching online, I’ve seen folks having a similar issue but it seems to have something to do with Wordpress and Imunify360 not playing nicely with Cloudflare. I’m not running Wordpress, however, my webhost does use Imunify360. I’ve been told by my webhost that I can disable some of the aspects of the web application firewall by disabling mod_security, so I did a few tests to see if I could figure this out. Here’s what I tried:

1. Added micro.blog’s IP to Cloudflare’s Security > WAF > Tools and set to Allow
2. Disabled mod_security on my webhost, cleared Cloudflare cache
3. Disabled Cloudflare DNS proxy on my A and CNAME records

Only #3 disabling the Cloudflare DNS proxy resolves this issue. micro.blog is able to grab my RSS feeds without issue if the proxy is off. Once I turn it back on, the issue returns. Does anyone have any suggestions on next steps to try?

[cbrandt]

Hi,

@user71413

something to do with Wordpress

Just a coincidence that past reports came from WordPress users, but the Imunify360 issue is unrelated to the CMS platform.

@user71413

my webhost does use Imunify360

Whoever told you this probably don’t know. Imunify360 is normally adopted by web hosting services for the protection of their whole infrastructure, not to protect individual accounts, and for that reason it cannot be disabled in shared hosting environment. Many hosts don’t even train their support team about it.

@user71413

Only #3 disabling the Cloudflare DNS proxy resolves this issue

The issue is a compound of Cloudflare caching with origin server firewall. An user triggers the webhost firewall, which sends an HTML (in this case XML) response with 200 status code (cacheable by Cloudflare if you use APO, Cache Rules, or Page Rule with Cache Everything), and the cached response is then returned to all subsequent visitors.

To break out of this cycle, you can create a Cloudflare Redirect Rule to redirect any request containing wsidchk= in its query string, then block the target URL. This will make the request that triggered Imunify360 to be blocked, and subsequent requests will get the proper response.

[user71413]

@cbrandt

To break out of this cycle, you can create a Cloudflare Redirect Rule to redirect any request containing wsidchk= in its query string, then block the target URL. This will make the request that triggered Imunify360 to be blocked, and subsequent requests will get the proper response.

Thank you for that insight on how to break out of this. Looking at the the Redirect Rules page, I see that I can create a Redirect Rule that matches wsidchk= in the URI Path, but how do I block the target URL?

[cbrandt]

@user71413

matches wsidchk= in the URI Path

You want to redirect based on URI Query, which is where this identifier parameter shows up.

When incoming requests match
URI Query contains "wsidchk="

Redirect to: http://www.example.com/blocked-by-origin

@user71413

how do I block the target

Create a WAF Custom Rule. If you followed the above example, then:

When incoming requests match...
URI Path equals "/blocked-by-origin"

The path /blocked-by-origin is just an example, it could be any path that doesn’t exist in your origin, as long as you block with WAF the same path you redirect with the Redirect Rule.

[user71413]

@cbrandt

as long as you block with WAF the same path you redirect with the Redirect Rule

Thanks for all of the help and tips! I set this up, but no joy. micro.blog still seems to be getting blocked from getting to my rss feeds. I’m not using any Page Rules, just the default “standard” caching level. I also don’t see any Security events in Cloudflare. I wonder if there is something else going on.

[cbrandt]

Check your Security > Events for log entries related to the RSS path. They should tell you if any Cloudflare service is blocking the requests.

[user71413]

I’ve checked but haven’t seen anything in Security > Events related to the RSS feed. I tried looking, but can’t see find any sort of log from Imunify360 in my webhost cPanel.

[cbrandt]

Did you purge the cache for the RSS path after you created the Redirect and WAF rules?

[user71413]

Yes, I purged cache of both RSS paths.

Just to make sure I set this up correctly, could you please take a look?

The top is the Redirect rule, the bottom is the WAF rule:

[cbrandt]

Yes, the rules are correct. Are the requests from the RSS monitor still getting that error message you mentioned in your first post?

[user71413]

Thanks for checking my set up!

I’ll need to check with the folks at micro.blog and ask if they’re still seeing the same issue. I can’t see the issue on my end. Going to my RSS feed via a web browser seems to work and my RSS feed reader doesn’t have any issues either.

[user71413]

micro.blog folks said they’re still seeing the “Please wait while your request is being verified” HTML page. They were kind ti send the full response with JavaScript bits that they think that Cloudflare is adding.

<!doctype html>
<html>
<head>
<meta charset="utf-8">
<meta name="robots" content="noindex, nofollow">
<title>One moment, please...</title>
<style>
body {
    background: #F6F7F8;
    color: #303131;
    font-family: sans-serif;
    margin-top: 45vh;
    text-align: center;
}
</style>
</head>
<body>
<h1>Please wait while your request is being verified...</h1>
<form id="wsidchk-form" style="display:none;" action="/z0f76a1d14fd21a8fb5fd0d03e0fdc3d3cedae52f" method="get">
<input type="hidden" id="wsidchk" name="wsidchk"/>
</form>
<script>
(function(){
    var west=+((+!+[])+(+!+[]+!![]+!![]+!![]+[])+(+!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+!+[]+!![]+!![]+!![]+!![]+!![]+[])+(+!+[]+!![]+!![])+(+!+[]+!![]+!![]+!![]+!![]+[])+(+!+[]+!![]+!![]+!![]+!![]+!![])+(+!+[]+!![]+!![]+[])),
        east=+((+!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+!+[])+(+!+[]+[])+(+!+[]+!![]+!![]+!![])+(+!+[]+[])+(+!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])),
        x=function(){try{return !!window.addEventListener;}catch(e){return !!0;} },
        y=function(y,z){x() ? document.addEventListener("DOMContentLoaded",y,z) : document.attachEvent("onreadystatechange",y);};
    y(function(){
        document.getElementById('wsidchk').value = west + east;
        document.getElementById('wsidchk-form').submit();
    }, false);
})();
</script>
</body>
</html>

[cbrandt]

What’s the feed URL, do you mind to share?

[user71413]

Sure! There are two:

https://ohmypizza.com/micro/atom.xml
https://ohmypizza.com/mb-feed.xml

[cbrandt]

OK, I see it now.

Since this is an XML content, the redirect and subsequent block are not happening, as the JavaScript redirect would only occur if the request that triggered the Imunify360 were responded to the browser as HTML. But I can see that the page is not being cached:

Request triggering the Imunify360 response

Subsequent request from an IP that doesn't trigger the i360

That tells me that the request from your monitoring service is what is triggering the Imunify360 response, not another bot request that was being cached as I had suspected.

You should bring this info to your host and ask them to disable Imunify360 for XML pages, as it doesn’t make sense to have a JavaScript-based challenge for a content (XML) that will never run it. The firewall is intercepting a request for the XML file, inserting HTML in its place, but it’s not properly replacing the Content-Type header, so the browser treats it as an XML document and can’t even interpret it (at least that’s what seems from Opera’s rendering).

Content-Type

Also, you should talk to your RSS monitor service, so they understand it’s their IP being rejected by Imunify360, and they need to investigate it and perhaps reach out to Imunify360 developers to try to figure out what’s the issue.

[user71413]

Thanks for doing the sleuthing on this! I’ll send this to the folks at micro.blog and see what they say before I contact my hosting provider. I’ll post when I hear back.

[user71413]

My hosting provider got back to me and this is what they say:

Regretfully, at this point there is nothing else we’re able to do. As you have confirmed that the issue only arises when using Cloudflare’s proxy service, it is exceedingly unlikely that anything on our server is causing the issue. Our recommendation would be to disable Cloudflare proxying or press Cloudflare’s support to further investigate the issue as it is originating with their service.

Haven’t heard back from the micro.blog folks yet.