I have setup a free Cloudflare account in the summer for my site and everything I thought was working great.
I have since found my site has disappeared from Bing. I logged into Bing Webmaster Tools and found that Bingbot could not fetch any pages, robots.txt or sitemap files from my site. Other bots like GoogleBot can.
I did contact Bing Support and this is what they said:
During my investigation I was able to identify that we are in fact having issues reaching your site URL with our crawlers.
When checked the crawl status of the your site URL with help of our internal tool, receiving crawl error as " OtherFailure" response, instead should receive 200 response.
I can simulate BingBot using their user agent and have access to the website so it appears Cloudflare is blocking BingBot by IP. I am not sure of this as I see no notifications in the Cloudflare Dashboard about blocking BingBot.
Anyways I currently have paused Cloudflare from the Overview tab (lower right corner), and Bing is able to crawl my site.
What do I need to do to get Cloudflare to allow Bingbot to access my site?