What is the name of the domain?
What is the error number?
403 (usually), 415
What is the error message?
Can’t say with precision (see below)
What is the issue you’re encountering
I’m using Cloudflare in front of an API-providing Web-based service, and have done so for many, many years. Requests and replies are almost always in text/plain, possibly JSON, and even sometimes XML, depending on the service. This has worked reliably for years, but recents increases in CF’s protection measures have made all requests coming from Amazon AWS IP range to be protected via JavaScript-based CAPTCHA — which obviously an API client cannot produce. The rules I wrote to attempt to exclude my endpoint from CAPTCHAs seem to have no effect whatsoever. The vague references I read about “API Shield” and similar technologies are appropriate for Pro/Enterprise users, but not for Basic users. What is the workaround, sort of removing CF proxying for that specific domain name?
What steps have you taken to resolve the issue?
- Added a Configuration Rule to turn off Browser Integrity Protection for that specific hostname
- Added a Page Rule to turn off Browser Integrity Protection for all URLs starting with that hostname
- On the API side and the client side, enforced text/plain as the MIME type for encoding and accepting, in the hope that CF’s systems understand that these requests are NOT to be subject to ‘Browser Integrity Protection’ since they obviously do NOT come from a browser!
I tried also to add a rule to exclude text/plain requests from being subject to BIP altogether (since I have several other APIs running under the same hostname, all of which presumably now being filtered out by BIP), but to no avail: without passing BIP at least once, the system will continue to assume it’s a request from a ‘suspicious’ network.
The only ‘workaround’, of course, is to disable CF proxying for that hostname.
Note: Currently this address is being actively used by API consumers; as such, since I don’t know any alternative, I have removed the address from CF proxying. I will shortly provide an alternative endpoint for testing purposes only, in the case that it might be useful.
Some references:
- Does Cloudflare protect API endpoints
- which points to this Cloudflare blog: Introducing API Shield
Note that API Shield (if it still exists) is a product which requires WAF to be active (i.e., not available for Basic users)
What are the steps to reproduce the issue?
You will need a client making an API request from a large pool of IP addresses which are often rotated/recycled among several different IP addresses — such as, in fact, anything on a cloud service. I have tested it with Amazon AWS, but hosting the client on any cloud should, in theory, exhibit the same issue.
Note that several tests I made on, say, bare metal servers with a fixed IP address and which have never been subject to ‘Browser Integrity Protection’, show that in such cases the API works exactly like intended (and like it did work for years).
The exact error is hard to figure out, since, in my case, clients are only able to retrieve the first 1000 bytes or so, and this hardly captures the whole BIP page. You can still see the ‘Just a moment…’ title on the header, though. But getting HTML when you asked for text/plain — and expected text/plain as a response — is a good sign that BIP is filtering the request and treating it as coming from a web browser on a ‘suspect’ ever-changing range of IP addresses (as, inevitably, almost all Cloud-base infrastructures will be these days, as well as most residential ones).
