Using cloudflare to load balance wordpress between to hosts

[matthew.giannelis]

Okay.

So I found a way to use Cloudflare to load balance two wordpress news websites.

What I did

Replaced real details with below

Site A www.site. com IP 0.0.0.1 - Australia
Site B balance.site. com IP 0.0.0.2 - Singapore

www.site. com is my origial welll establised wordpress site

I got myself a second hosting account with a different webhost (both are cpanel)

I created a sub-domain on my second server and called this balance.site. com

I then used Cloudflare to create an “A” record and pointed this to balance.site. com

I then migrated a copy of my site www.site. com over to the second server balance.site. com

Now my site is working on both servers. But I made sure to NOINDEX balance.site. com to avoid duplicate content issues in the search engines (At this stage is basically a staged site) running on balance.site. com hosted on different server elsewhere. (Stratigeic geo location for my purposes)

I then created a Cloudflare loadbalancer with IP and using HOSTHEADER names

Boom! Loadbalancing working great!

For site sync I used a "paid plugin* that has some good options that syncs the two sites, posts, plugins, databases all of it. I found something called wpsynchro. com

THE PROBLEM!

As I had to set the second site on balance.site. com to NOINDEX to avoid duplicate content issues, if the seach engines were to read the robots.txt file from site B balance.site. com/robots.txt this will cause my site to get deranked from Google.

So close to such a simple loadblancing solution!

So. My questions.

While using Cloudflare load balancing (dont care about failover) does anyone think or know if there is a way to make sure that only the robots.txt is being read from site “A” www.site. com at all times?

Of course, with this sync plugin I found there is actually a way to tell it not to push/pull a certain file. So thats good and avoids site B balance.site. com from being sent the robots.txt file from site A causing site B to end up with the indexable intructions from site "A"s robots.txt thus coming back to a duplicate content issue in search engines again

So.

Happy to hear from anyone who might be able to tell me if there is a way to ask cloudfare only to read the robots.txt file from site A at times only?

Thanks

[simon]

I’m not sure I follow how Google is going to crawl your second server here - so you probably don’t need to change the robots.txt on server B - what you need to do is secure your servers from direct access and ensure all public facing DNS records are orange-clouded. Or make sure all direct requests are 301 redirected to your orange-clouded hostname.

How is Google crawling server B?

[matthew.giannelis]

Hello

Thanks for your question and looking into this. Your question raises another question for me actually.

If server A was to go down. Then wouldent the files from site B get crawled where I have the robots.txt file set to noindex.

The whole reason I had to set server B to noindex was to avoid duplciate content issues,.

In case there is some confusion. if you to visit www.site .com and balance.site .com these both load the same site

This is why I had to NOINDEX balance.site .com

So if something were to happen www.site .com then search engines would be reading the robots.txt from balance.site .com which would make my site get deindex from search engines

Hope that make sense. Look forward to comments

[simon]

You don’t need to create a balance.site.com hostname - you could just add the IP address.

Either way - just 301 redirect all traffic to balance.site.com (or the IP address) to your actual, proper hostname e.g. www.site.com. That way nobody can crawl the wrong hostname.

[matthew.giannelis]

Hi there

Thanks for the info

What do you mean by I don’t need to create a balance.site.com hostname? That does not make sense to me in the way I have set things up.

This balance,site. com is basically just an exact copy of my website running on a different domain.(sub-domain with an A record pointing to the different serverr IP) So I need the hostname otherwise I would not have been able to install wordpress on the second cpanel server.

So within the load balancer in C/F load balacing simply wont work by just adding the server B IP This is why I used the HOSTHEADER name in the load balancer.

So anyway, I reallly apprciate your answer however this has left me confused.

Create the 301. Where?

If I am 301 all traffic from sever B to server A, does that not defeat the whole point of the load balancer

Really confused. Could you provide more detail on thiis

301 redirect? Where do I add the redirect?

[simon]

A simpler way to set this up is:

Host A - IP 1.2.3.4 - configure the server to be answer requests for www.site.com
Host B - IP 4.5.6.7 - configure the server to be answer requests for www.site.com

e.g. both servers can return content for www.site.com

Then you create your load balancer on Cloudflare for www.site.com and put your two origin IPs 1.2.3.4 / 4.5.6.7 in a pool or pools. Then there’s no “balance.site.com”.

If you can’t do this, you need to configure balance.site.com to redirect all traffic to www. That will force any visitor (including google) who somehow hits this hostname/IP directly to go back to www.site.com which will hit your load balancer and whatever server is healthy.