I am trying to only allow googlebot and bingbot onto my RSS feeds for selected reasons. However, when attempting this I am having many conflicting issues ranging from my whole site being blocked to google feedburner not being able to access my rss urls so I lose instant news indexing.
I have tried so many different ways of creating expressions, Nothing seem to work.
For example, This is just one of many expression combos I have tried, But instead it blocks access to my whole site even though its not even speficied to do so
(http.request.uri.path eq “/feed” and http.user_agent ne “APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)”) or (http.user_agent ne “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”) or (http.user_agent ne “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36”) or (http.user_agent ne “Googlebot/2.1 (+http://www.google.com/bot.html)”) or (http.user_agent ne “FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)”) or (http.user_agent ne “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”) or (http.user_agent ne “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36”) or (ip.geoip.asnum ne 15169) or (not http.user_agent contains “bingbot”) or (http.user_agent ne “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36”) or (http.user_agent ne “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)”)
THEN BLOCK <blocks my whole site to the world) Only specified the /feed url
Ive also tried swapping this around putting the url last
My guess is the brackets are messed up. It looks like you meant to keep adding user agents but instead it become (http.request.uri.path eq “/feed” and http.user_agent ne “APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)”) or (http.user_agent ne “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”) which mean that the first user agent is allowed to visit the /feed URL but all other user agents are blocked. Try something like
(http.request.uri.path eq “/feed” and http.user_agent ne “APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)” or http.user_agent ne “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” or http.user_agent ne “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36” or http.user_agent ne “Googlebot/2.1 (+http://www.google.com/bot.html)” or http.user_agent ne “FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)” or http.user_agent ne “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” or http.user_agent ne “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36” or ip.geoip.asnum ne 15169 or not http.user_agent contains “bingbot” or http.user_agent ne “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36) or http.user_agent ne “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)”)