Pinterest crawlers are being blocked

Cloudflare is blocking Pinterest bots, and I have no idea why.

I have been using Cloudflare for a few months, and Pinterest could fetch data without any problems, but since the last few days, the bots are unable to access my site (they can when I deactivate Cloudflare). I haven’t change anything to Cloudflare settings that could have caused this.

I’ve tried adding firewall exceptions with Pinterest IP range and user agents but it didn’t change anything. I’ve also tried setting the security level to “essentially off”, same, nothing changed.

What could be causing this?

We’ve noticed some bots being blocked that weren’t blocked before. For some of us, it’s due to Bot Fight Mode -> Tools.

I’ve checked it, it’s turned off. I’ve added a “known bots” firewall exception, set on allow. I can see the Pinterest bots from the log. I’ve also checked my access log, and it shows Pinterest IP. But somewhat, Pinterest is unable to fetch any metadata or images.

Could there be something letting the bots access the site, but blocking them from retrieving any data?

Hi Julie,

This sentence caught my attention:

Please keep in mind the scope of the Allow action is limited to Firewall Rules; matching requests are not exempt from action by other Cloudflare Firewall products, such as IP Access Rules, WAF, BIC, Hotlink protection, etc.

Try changing to Bypass and choose at least hotlink protection and browser integrity check and see if that solves the problem.

1 Like

Hi,

Thank you for your help. It’s not working, I’ve even tried to bypass everything.

I’m still getting “couldn’t fetch the image” when trying to pin an image from my website, and a 500 internal error when using Pinterest’s URL debugger tool.

I don’t understand how, all of the sudden, it stopped working. I have another website on Cloudflare, and this one is working just fine…

Hi,

Just to update this thread as i’m certainly it might be useful.
Cloudflare’s security features were not blocking the crawler, and we managed to demonstrate that via ticket.

Picking up the last public feedback it’s mentioned all good in another website being proxied via Cloudflare, same crawler so why do we have a 500 internal error?

Well, the impacted website was using a worker, “sg-worker”, after disabling it everything went back to normal.

I’ve now searched in the community and found another customer reporting similar problem:

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.