I’m trying to grab data from the Cloudflare api, however my trawler obeys
robots.txt rules. The current
robots.txt for api.Cloudflare .com returns a 522 error, which causes my trawler to default to not allowed.
Can this robots.txt be fixed or a 404 returned if there is none for the api as well as checks to ensure this error is flagged by Cloudflare in the future?