One of my users wants to use a 3rd party tool to crawl our website (for SEO analysis, etc). However they are requesting to have it whitelisted. I believe they want to whitelist the user agent. My question is, is it safe to whitelist based on user agents?
Makes me nervous, user agents are really not unique correct? Dont we all have user agents? out of the millions/billions of people online, im sure many have the same.
User Agent isn’t a 100% safe method. While browsers do share user agents, the 3rd party tool should have their own user agent, so it will be safer, but anyone can spoof a user agent. If the 3rd part tool offers the their IPs as well, then you can whitelist based on user agent plus IP.