if we go to https://radar.cloudflare.com/traffic/verified-bots and search for bot names containing “crawl”
we get googlebot and bingbot in the results. why ?
isnt this search meant to be for user agent names ?
if yes, then “googlebot” does not contain “crawl”
I am concerned because sometimes i block (lower(http.user_agent) contains “crawl”). need to make sure that it wont affect google or bing
You’re searching across all of the fields. If you search “monitoring” or “feed”, you get results as well from that category. You can search “Google” to find all of the google bots/search by the owner field.
You can find the user agents of google/bing on their respective pages: Google Crawler (User Agent) Overview | Google Search Central | Documentation | Google for Developers, and confirm that they do not contain the word crawl.
That’s an interesting rule, probably not too effective since actual bad bots wouldn’t be nice enough to identify themselves like that, but you could just exclude verified bots (not cf.client.bot and http.user_agent contains "crawl")
I already use not cf.client.bot but want to filter down the list
So, do you mean we cant search for bots names containing for example “crawl” in this link
https://radar.cloudflare.com/traffic/verified-bots
You can. I’m just saying the actual search is checking if the search query is contained within any of the visible fields. Your “crawl” search is matching the values in the Category field, hence why they’re being shown
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.