Hi, I have two origin servers behind a load balancer and I would like to direct search engine crawlers to only one specific server.
The reason for this is that we are migrating on old site to a new tech stack and we want to retain the seo by keeping the search engine crawlers going to the old site until the new site is at parity with the same content.
I know that AWS can do this, but is there a mechanism in CloudFlare to do this ?
Via the default load balancer you can’t, but you can use a simple worker (just a few lines on JS) to do just that, even without the need to use a load balancer per se.
Would that not be a redirect ? I don’t want to use redirects.
I think you answered my question anyway, no Cloudflare can’t direct traffic based on user agent.
Thanks
It’s not a redirect, the Worker can do a request to whatever (with some limits) origin it wants. I can’t find the example on the documentation anymore, but you can most certainly do that.
It can (and there is a quick solution), just not natively in the load balancer product.
I’d be interested to see an example of that @Matteo I’ll take another look in the docs to see if I can see what you are describing.
Cheers
I found this in the Blog, there used to be something similar in the Developer Docs, but still can’t find it.
Hi @Matteo, is it this ? Fetch HTML · Cloudflare Workers docs
I could see how that snippet could be adapted and combined with inspecting the headers. It would be a lot more more than just letting the LB do it but it may be a solution in our instance.
Cheers
No, it was basically the same as was in that blog post above, I don’t think it’s in the templates anymore.