Ads.txt 403 Forbidden Error - cf-mitigated: challenge

Website, Application, Performance
1

I work at an ad network company - and many of our publishers use WordPress and Cloudflare. They need to have their ads.txt files available to demand partners to crawl programmatically with tools like curl. Often this will fail using curl but not a normal web browser so the publisher thinks all is well but partners like Google and Appnexus are not able to crawl and injest this important file. When using curl on a site like this we get:

HTTP/1.1 403 Forbidden
Date: Wed, 10 May 2023 17:50:21 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: close
Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Resource-Policy: same-origin
Permissions-Policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
Referrer-Policy: same-origin
X-Frame-Options: SAMEORIGIN
cf-mitigated: challenge
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=t3R4vOZNP6I42V6Z0sx4Za0pPh2%2F94bmTCTLdhatr%2B6L99iWlnif9LFgoKPS2kRl7twWSyZKko%2Bsmo9MVV%2B1XOzfzSomZ4rKcY1CTZDlPIObQXnrjtmVmFbwC1hH59Dp124xNcrvt2CVf6jn"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7c54062ab86e31df-LAX
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
yada yada
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>

What can we tell users to do to keep this text file from being part of Cloudflare’s javascript challenge?

3

Hi,

Cloudflare challenges can have several sources. They should first check their WAF/Firewall logs to understand which of Cloudflare tools are performing the challenge. It could be:

  1. Bot Fight Mode, in which case their only option is to turn it off;
  2. Super Bot Fight Mode, Browser Integrity Check, another Cloudflare security feature or some of their own Custom Rules (aka Firewall Rules), which can be bypassed if they so whish
    a) within their own Custom Rules with a condition that these rules do not apply to URI Path “ads.txt”; or
    b) with an extra Custom Rule with the Bypass or Skip action for that condition
  3. WAF Managed Rules, for which they need to create a WAF Exception.

There might be other possibilities, and they can always :search: the documentation for solutions or open a topic here at the community.

1 Like
4

Thank you so much for your detailed reply. Much appreciated!

1 Like
5

No problem. Please notice I’ve edited my reply to replace: WAF Exemption Exception

1 Like
6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.