Google AdSense permamently loosing ads.txt file

About a month ago, I implemented Cloudflare, and around the same time, the issue with losing the ads.txt file began. Google simply can’t reach it. In the Adsense account, I click the ‘check’ button next to the site where Google lost the ads.txt, and the file is there. However, after some time, it gets lost again.

Here is my WAF rules:

  1. Known Bots, action “Skip”
( or (http.user_agent contains "Google-Site-Verification") or (http.user_agent contains "AdsBot-Google") or (http.user_agent contains "Google Favicon") or (http.user_agent contains "Google-AMPHTML") or (http.user_agent contains "Google-Ads") or (http.user_agent contains "Google-Adwords") or (http.user_agent contains "GoogleAdSenseInfeed") or (http.user_agent contains "googleweblight") or (http.user_agent contains "Mediapartners") or (http.user_agent contains "")
  1. Direct traffic, action “Managed Challenge”
(http.referer eq "")
  1. Old HTTP protocols, action “Managed Challenge”
(http.request.version in {"HTTP/1.0" "HTTP/1.1" "HTTP/1.2"})

While your skip rule does not specifically exlude that particular file, I would assume it’s the direct traffic rule. I don’t think anyone/many will send a referrer when requesting that file. You probably best include it in your skip rule.

Just for the sake of it, there is no HTTP 1.2 (no idea why Cloudflare lists that). You can safely challenge 1.0, but I’d be careful with 1.1. Going by that, 2.0 is also an “old” protocol.

Do you suggest to remove this rule?

Which rule? The one about HTTP?
I would probably limit it to 1.0. Even that might fix your issue with the file, but I’d adjust the referrer as well.

In short, I’d drop the last two rules and possibly add the file to your skip rule. However, it may be also important to know why you configured those at all.

But the first rule explicitly states to skip all known bots and not apply the remaining WAF rules in the future. Therefore, if it is a known bot, then the Redirect and Old Protocol rules will not apply to it.

That does not necessarily mean it applies to this request. Did you take a look at the firewall event log?

Anyhow, why do you have the other rules?

I looked at this log, but I couldn’t find any entries from /ads.txt due to the fact that I have a free plan that does not provide the ability to search by URL.

The main reason why I use Cloudflare and other rules is that I want to protect my website from AdSense Click Bombing and from bots on the site.

That is not correct, you can search by the path -

But I am not sure these rules will actually do much in this context. You only have referred visitors? Even if so, there’s no guarantee they send a referrer. And yes, HTTP 1.1 is an older version, but so is 2.0. Both are still being used however.

1 Like

Yeah! You’are right! I don’t know how I didn’t notice that filter :slight_smile:

I have traffic from search engines. But most bots come with direct requests.

Now, I added new rule in the WAF. I made /ads.txt and /robots.txt accessible for all bots.

(http.request.uri.path eq "/ads.txt") or (http.request.uri.path eq "/robots.txt")

My WAF rules now:

Thank you very much for help!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.