My site’s internal search is powered by Swiftype (an Elastic Search product) but when I recently enabled my Bot settings to “Manage a Challenge” of “Likely Automated” traffic, it blocked the index. After some troubleshooting I discovered that it was returning a 403 error.
Is there a way to allow a certain user agent so I can keep the “Managed Challenge” setting on “Likely Automated” traffic, while allowing this User-Agent. The user agent is known. It is:
Mozilla/5.0 (compatible; Swiftbot/1.0; UID/5c40f8bc8db2314527c1b35f; +http://swiftype.com/swiftbot)