I hope somebody will be able to provide some guidance.
We are a HubSpot customer who themselves use Cloudflare. We host a majority of our website ourselves through our own servers and our blog is hosted via the HubSpot platform.
Due to a number of reasons, we decided to utilise a reverse proxy so that visitors reach the HubSpot hosted blog content via our main www domain and not a secondary domain.
However, when Googlebot goes via the reverse proxy to index the content of our blog, it is forbidden due to the security features of Cloudflare.
I am making the assumption that as this is going via the reverse proxy, the remote IP is that of our server and not the Googlebot servers and therefore triggering this security feature.
Is there a header I can pass via the reverse proxy that would allow this to work and Googlebot to correctly index this content via the main www domain as we wish? The proxy adds the typical X-Forwarded-For header but this doesn’t appear to change anything.
Could somebody please provide some guidance as to how this setup could work?
Have you tried allowlisting the IP address of the proxy? Though, of course this would allowlist it for all requests, not only those from Google.
Thanks for your speedy response.
Re. Whitelisting - I presume this is a feature within Cloudflare? We are not the Cloudflare customer direct. HubSpot is. I could ask them about this but we host our servers on AWS, which could theoretically change IP on occasions. I could of course look to get this as a fixed IP. I’d also need to ask HubSpot about this option.
Is there not a header of some kind that I can pass for this scenario?
Yes, but you do need control over the zone of course.
In this case you would need to contact them.
No what you are basically asking is a magic header which instructs Cloudflare to bypass security settings and that would be somewhat counterproductive to its primary purpose
Can you rewrite the Google user agent to something of your own? Cloudflare might not get that suspicious in that case - no guarantees of course
That’s great info - thanks Sandro.
Re. User Agent - I already did that and it did work ;-). It simply had the adverse effect that our analytics within the HubSpot product showed these hits.
So, just to conclude, can I interpret that the Cloudflare security setup will only take the IP remote address for it’s security assessment?
To my best knowledge, yes. You could contact support for clarification but I’d be surprised if they took anything else - not specifically configured as trusted - into account, as that would open the possibility for tampered requests.
Related to the above, as it is a header I’ve seen discussed, what is the relevance of the “CF-Connecting-IP” header. I’ve seen comments talk about overriding X-Forwarded-For with this header so I am wondering whether this could have an impact on my issue above?
It is a header, indicating the actual client’s address, Cloudflare sends along with the request TO the origin.
ok - understood. So this is not relevant in my case.
No, that header will hit the origin but will only contain your proxy’s address.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.