LinkedIn Post Inspector Error: “We encountered a server error while trying to inspect the URL.”
What is the issue you’re encountering
LinkedIn is not able to crawl pages and create previews if Cloudflare protection is enabled.
What steps have you taken to resolve the issue?
I tried disabling Cloudflare to confirm it is related to Cloudflare (confirmed);
I tried allowing Verified Bots (which includes LinkedInBot) on a custom firewall rule (didn’t work);
I tried seeing the Firewall Events and looking for anything with LinkedInBot and didn’t find any being blocked (could LinkedIn be using a non-standard user agent when scraping?);
Tested the page using Meta’s Sharing Debugger tool and it is working there.
What are the steps to reproduce the issue?
Put a site behind Cloudflare and try to scrape any page using LinkedIn Post Inspector (Post Inspector), it should fail.
try scraping the same page using Meta’s Sharing Debugger (Sharing Debugger - Meta for Developers), it should work.
Could you create a Custom WAF rule using instructions from below article as follows from below expression?
Expression:
(http.user_agent contains "LinkedInBot" and ip.src.asnum in {40793 132406 132466 13443 137709 14413 197613 20049 202745 20366}) or (http.user_agent contains "Twitterbot" and ip.src.asnum in {13414 35995 54888 63179}) or (http.user_agent contains "Pinterestbot" and ip.src.asnum in {14618}) or (http.user_agent contains "Disqus" and ip.src.asnum in {14618})
Add the action “Skip” with WAF components to skip “All remaining custom rules” and make sure it’s the 1st rule from above on the list.
You should be able to see the challenged or blocked event under the Security tab → Events at Cloudflare dashboard for your zone and know exactly which security option was triggered. Could be Managed Rules my best guess, otherwise Bot Fight Mode or Browser Integrity Check.
Once you find them, click on a particular one to find more details about it (user-agent, IP, HTTP version …). If yes, could you share some details which service was triggered that blocked you?
Out there definitely there are the “fake ones” using the same user-agent string.
Furthermore, for the official one, I doubt. It should be coming from Microsoft’s AS8075 or the above ones which I’ve listed in the WAF expression rule.
Hey, @fritex! Thanks for replying!
Unfortunately, this didn’t work for me. I created the rule you mentioned (I had a similar rule before, but it only checked the User-Agent), and the issue persists. This time, I couldn’t even see any events from the LinkedInBot in the WAF Events page.
Disabling Cloudflare makes LinkedIn work normally again. Losing my mind over this.
I’m attaching redacted .HAR files from a private tab loading the Post Inspector if you’re interested in trying to find something I’m missing.
