Cloudflare-Smart-Transit/1.0 User-Agent requests


#1

For the last month, I’ve noticed some requests with this format in the log of my web server behind CloudFlare:

162.158.41.29 - - [06/May/2017:21:40:28 +0000] “GET / HTTP/1.1” 301 0 “” “Mozilla/5.0 (compatible; Cloudflare-Smart-Transit/1.0; +https://www.cloudflare.com/” 0/159085
162.158.41.29 - - [06/May/2017:21:40:29 +0000] “GET / HTTP/1.1” 301 0 “” “Mozilla/5.0 (compatible; Cloudflare-Smart-Transit/1.0; +https://www.cloudflare.com/” 0/168656
162.158.41.29 - - [06/May/2017:21:40:30 +0000] “GET / HTTP/1.1” 301 0 “” “Mozilla/5.0 (compatible; Cloudflare-Smart-Transit/1.0; +https://www.cloudflare.com/” 0/205860

There are always 3 at a time, with an interval of between 20 and 50 minutes between every 3 requests, and they all come from the CloudFlare IP range.
The only HTTP headers provided are:
User-Agent: Mozilla/5.0 (compatible; Cloudflare-Smart-Transit/1.0; +https://www.cloudflare.com/
Host:
Accept-Encoding: gzip

The only place I’ve found that User-Agent menctioned is in this CS stats site.
They are answered with a 301 redirect because I redirect every request that does not have my site as the Host header.

Should I be concerned about these requests? Am I breaking any CloudFlare functionality redirecting them?


#2

Typo: the Host header is my origin server IP, which I prefer to not disclose.

Clarification: the 301 redirects to the correct domain.


#3

Hi
I’m not a Cloudflare support member, just a user like you.
It seems you are redirecting a Cloudflare’s crawler and blocking whatever the purpose is.


#4

That user-agent, specifically when matching this exact string:
Mozilla/5.0 (compatible; Cloudflare-Smart-Transit/1.0; +https://www.cloudflare.com/

When originating from one of Cloudflare IPs:

Is part of a new tool/product that monitors network health (between Cloudflare PoPs, and also between Cloudflare and origins). This data is then used by to dynamically update the routes that traffic takes which provides both a smart transit capability as well as tiered caching.

There is no downside to this operating (the product announcement will be in the near future and the improvements as a result of this are really impressive) but you may wish to have your origin web server return a static file for this user agent if not doing so is resulting in additional computation by your web app (i.e. if the network monitor hits a page that will do database lookups, since the tool only cares about measuring the network rather than the page speed, perhaps you want to avoid those DB queries).

David


Cloudflare Sends More Than 1200 Bot Traffic Per Day
DNS Analytics extended to Page Speed Analytics?
#5

Great! Thanks for the clear explanation.